Job scheduling configuration information generation method and device

Abstract

The present application is about a method and device for generating job scheduling configuration information. The specific scheme is: obtaining the data demand information to be processed; sorting the data demand information to obtain the physical models of the source table and the target table, the data mapping information between the source table and the target table, the source system information and the target table system information in the data demand information; generating the target SQL statement based on the statement characteristics of the corresponding database according to the physical models of the source table and the target table, and the data mapping information between the source table and the target table; determining the job for executing the target SQL statement, and determining the upstream and downstream jobs that have a dependency relationship with the job according to the source table, the target table, the source system information and the target table system information; generating the job scheduling configuration information according to the upstream and downstream jobs, the business, and the job dependency relationship. The present application is of vital importance to improving development and work efficiency.

Description

Technical Field

[0001] The present application relates to the field of computer technology, in particular to database technology, machine learning technology, natural language processing technology, and more particularly to a method, device, electronic device, and storage medium for generating job scheduling configuration information. Background Art

[0002] With the development of the internet and technology, the importance of data has gradually been discovered. Data has permeated everyone's lives. Currently, almost every enterprise-level project requires data processing and replication. Data connections and dependencies exist between projects and within projects themselves, and downstream businesses require support from upstream data. Currently, data requirements between systems are communicated to business personnel before the specific requirements are explained to data personnel. This information transfer can lead to information bias. Furthermore, data is interpreted differently across systems, and processing logic is complex. Understanding the requirements takes a considerable amount of time for data personnel, resulting in high communication costs and low project development efficiency. Summary of the Invention

[0003] This application provides a method, device, electronic device, and storage medium for generating job scheduling configuration information to at least address the issues in related technologies such as the long time required to understand data requirements and high communication costs, which lead to low project development efficiency. The technical solutions of this application are as follows:

[0004] According to a first aspect of an embodiment of the present application, a method for generating job scheduling configuration information is provided, comprising:

[0005] Obtaining data requirement information to be processed;

[0006] Sorting the data requirement information to obtain physical models of the source table and the target table, data mapping information between the source table and the target table, source system information, and target table system information;

[0007] generating a target SQL statement based on the physical models of the source table and the target table, data mapping information between the source table and the target table, and statement characteristics of the corresponding database; wherein the target SQL statement is used to map the data in the source table to corresponding fields in the target table;

[0008] Determine a job for executing a target SQL statement, and determine upstream and downstream jobs that have a dependency relationship with the job based on the source table, the target table, the source system information, and the target table system information;

[0009] Job scheduling configuration information is generated according to the upstream and downstream jobs, the services, and job dependencies.

[0010] According to a second aspect of an embodiment of the present application, there is provided a device for generating job scheduling configuration information, comprising:

[0011] A first acquisition module is used to obtain data requirement information to be processed;

[0012] A second acquisition module is configured to sort out the data requirement information to obtain the physical models of the source table and the target table, data mapping information between the source table and the target table, source system information, and target table system information.

[0013] a first generation module, configured to generate a target SQL statement based on the physical models of the source table and the target table, data mapping information between the source table and the target table, and statement characteristics of the corresponding database; wherein the target SQL statement is used to map the data in the source table to corresponding fields in the target table;

[0014] a determination module, configured to determine a job for executing a target SQL statement, and to determine upstream and downstream jobs having a dependency relationship with the job based on the source table, the target table, the source system information, and the target table system information;

[0015] The second generating module is used to generate job scheduling configuration information according to the upstream and downstream jobs, the services, and job dependencies.

[0016] According to a third aspect of the embodiments of the present application, there is provided an electronic device, including:

[0017] a processor, and a memory communicatively connected to the processor;

[0018] The memory stores computer-executable instructions;

[0019] The processor executes the computer-executable instructions stored in the memory to implement the method as described in the first aspect.

[0020] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect above.

[0021] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, comprising a computer program, wherein the computer program implements the steps of the method described in the first aspect when executed by a processor.

[0022] The technical solutions provided by the embodiments of this application offer at least the following beneficial effects: SQL statements can be generated based on information such as the source table, target table, service mode, and processing rules contained in data demand information, and job scheduling configuration information can be generated based on the job's source table and system information. This intelligent implementation of the entire process, which permeates the development process for data developers, simplifies their work and streamlines the development process. This is crucial for improving development and work efficiency in response to the increasing demand for data.

[0023] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present application.

[0025] Figure 1 A flowchart of a method for generating job scheduling configuration information provided in an embodiment of the present application.

[0026] Figure 2 A flowchart of another method for generating job scheduling configuration information provided in an embodiment of the present application.

[0027] Figure 3 A flowchart of another method for generating job scheduling configuration information provided in an embodiment of the present application.

[0028] Figure 4 A flowchart for automatically verifying errors and optimisation points in data requirement information provided in an embodiment of the present application.

[0029] Figure 5 A block diagram of a job scheduling configuration information generation device provided in an embodiment of the present application.

[0030] Figure 6 A block diagram of another device for generating job scheduling configuration information provided in an embodiment of the present application.

[0031] Figure 7 A block diagram of another device for generating job scheduling configuration information provided in an embodiment of the present application.

[0032] Figure 8 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0033] In order to enable ordinary people in the art to better understand the technical solutions in the embodiments of the present application, a clear and complete description is given.

[0034] It should be noted that the terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0035] Figure 1 This is a flow chart of a method for generating job scheduling configuration information provided by an embodiment of the present application. Figure 1 As shown, the method for generating job scheduling configuration information may include but is not limited to the following steps.

[0036] In step 101, data demand information to be processed is obtained.

[0037] In one possible implementation, the data requirement information to be processed can be read from a data requirement document. The content of the data requirement document may include information such as the source system, source table, source table fields, target table, target table fields, service method, and processing rules required for data processing, which has been confirmed in advance with business personnel. The service method can be understood as the method used to map source table data to target table fields, such as which service method is used to map source table data to target table fields. The processing rule refers to the rule corresponding to the service method used, such as using processing rule a corresponding to service method 1 to map source table data to target table fields.

[0038] In step 102, the data requirement information is sorted to obtain the physical models of the source table and the target table, the data mapping information between the source table and the target table, the source system information, and the target table system information.

[0039] In one possible implementation, the data demand information can be combined with a text library and a machine learning model to perform text fuzzy matching and natural language processing to obtain the physical models of the source table and the target table, the data mapping information between the source table and the target table, the source system information, and the target table system information in the data demand information.

[0040] Optionally, the data demand information is fuzzy matched based on a preset text library, and the data demand information is natural language processed in combination with a preset machine learning model, so that the physical models of the source table and the target table, the data mapping information between the source table and the target table, the source system information and the target table system information in the data demand information can be obtained. Among them, the machine learning model can be a model obtained in advance based on a machine learning task. For example, the machine learning task can be association rule learning, or it can be other machine learning, such as deep learning, etc., using deep neural networks to make the model more complex, so that the model has a deeper understanding of the data, so that the model has the ability to sort out the data demand information. This application does not specifically limit the specific structure and function of the machine learning model, and will not elaborate on it.

[0041] In an embodiment of the present application, the physical model of the source table may include but is not limited to the source table and source table fields, and the physical model of the target table may include but is not limited to the target table and target table fields; the data mapping information may include but is not limited to the service mode and the processing rules corresponding to the service mode.

[0042] In step 103 , a target SQL statement is generated according to the physical models of the source table and the target table, the data mapping information between the source table and the target table, and the statement characteristics of the corresponding database.

[0043] In an embodiment of the present application, the target SQL statement may be used to map data in a source table to corresponding fields in a target table.

[0044] In one possible implementation, based on the physical models of the source table and the target table, as well as the data mapping information between the source table and the target table, the service mode and processing rules used when mapping each field in the source table to the corresponding field in the target table are determined; the fields in the source table are spliced ​​into SQL statements based on the statement characteristics of the corresponding database using the corresponding service mode and processing rules, and the spliced ​​SQL statements are assigned to the fields in the target table that correspond to the fields in the source table, and the SQL statements of all fields in the target table are spliced ​​to obtain the target SQL statements.

[0045] Optionally, the service mode and processing rules used when mapping each field in the source table to the corresponding field in the target table can be determined based on the physical models of the source table and the target table, as well as the data mapping information between the source table and the target table. Based on the source table, source table fields, target table, target table fields, and the service mode and processing rules used when mapping each field in the source table to the corresponding field in the target table, the target SQL statement can be spliced ​​and generated through the statement characteristics of each database.

[0046] For example, you can combine the source table's fields into a SQL statement using the corresponding service methods and processing rules (also known as data caliber processing logic). If the SQL statement is complex, you can convert it into a subquery and assign the combined result to the corresponding field in the target table. After all the required fields for the target table are organized, they are combined into a complete SQL statement, which becomes the target SQL statement.

[0047] In step 104, a job for executing the target SQL statement is determined, and upstream and downstream jobs having dependency relationships with the job are determined based on the source table, the target table, the source system information, and the target table system information.

[0048] In one possible implementation, a job can be created to execute the target SQL statement. For example, all jobs processing the target table can be organized, with each target table requiring a single job. Jobs of the same type can be placed in the same job flow for execution; optionally, a job flow can contain only one job. Based on the source table information, the source table's job flow name, condition name, dependent component information, and pre-dependent business date offset rules can be automatically queried to identify upstream and downstream jobs that have dependencies on the job executing the target SQL statement.

[0049] For example, information such as the job and job flow name can be generated based on the processed table and system information, and then the job can be searched in the job library based on the dependent table information as the upstream job of this job. There can be multiple upstream jobs.

[0050] In step 105, job scheduling configuration information is generated based on upstream and downstream jobs, services, and job dependencies.

[0051] Optionally, the job flow name of the target table can be generated based on the job execution frequency, function abbreviation information and time zone information, and the job name of the job can be generated. The job flow name, condition name, dependent component information and pre-dependent business date offset rule of the source table can be obtained based on the source table. The job flow name of the target table, the job name of the job, the job flow name of the source table, condition name, dependent component information and pre-dependent business date offset rule can be filled into the preset job scheduling configuration template to obtain the job scheduling configuration information. In a possible implementation, such as Figure 2 As shown, the method for generating the job scheduling configuration information may include but is not limited to the following steps.

[0052] In step 201, a job flow name of a target table is generated according to the execution frequency, function abbreviation information and time zone information of the job.

[0053] In step 202, a job name of the job is generated according to the name of the target table and the execution frequency.

[0054] In step 203, the job flow name, condition name, dependent component information, and preceding dependent business date offset rule of the source table are obtained according to the source table.

[0055] In step 204, the job flow name of the target table, the job name of the job, the job flow name of the source table, the condition name, the dependent component information and the preceding dependent business date offset rule are filled into the preset job scheduling configuration template to obtain the job scheduling configuration information.

[0056] Optionally, the job flow names can be deduplicated and automatically filled into a job scheduling configuration template. The job scheduling configuration template can be presented as a sheet page, with the job information filled into the corresponding position of the sheet page, and other information (such as the condition name, dependent component information, and the preceding dependent business date offset rule) filled into the corresponding position of the sheet page to obtain the job scheduling configuration information. Optionally, the job scheduling configuration information can be directly imported into the scheduling system to implement the scheduling configuration.

[0057] By implementing the embodiments of this application, SQL statements can be generated based on the source table, target table, service mode, and processing rules in the data demand information, and job scheduling configuration information can be generated based on the source table and system information of the job. This intelligent implementation throughout the entire process, which runs through the development process of data developers, can simplify the work of data developers and streamline the development process. With the increasing demand for data, it is crucial to improving development and work efficiency.

[0058] Figure 3 This is a flow chart of another method for generating job scheduling configuration information provided by an embodiment of the present application. Figure 3 As shown, the method for generating job scheduling configuration information may include but is not limited to the following steps.

[0059] In step 301, data demand information to be processed is obtained.

[0060] In the embodiments of the present application, step 301 can be implemented in any of the ways in the embodiments of the present application. The embodiments of the present application do not limit this and will not be described in detail.

[0061] In step 302, the data requirement information is sorted to obtain the physical models of the source table and the target table, the data mapping information between the source table and the target table, the source system information, and the target table system information.

[0062] In the embodiment of the present application, step 302 can be implemented in any of the ways in the embodiments of the present application. The embodiment of the present application does not limit this and will not be described in detail.

[0063] In step 303, a target SQL statement is generated based on the physical models of the source table and the target table, as well as the data mapping information between the source table and the target table, and based on the statement characteristics of the corresponding database; wherein the target SQL statement is used to map the data in the source table to the corresponding fields of the target table.

[0064] In the embodiment of the present application, step 303 can be implemented in any of the ways in the embodiments of the present application. The embodiment of the present application does not limit this and will not be described in detail.

[0065] In step 304, the target SQL statement is syntax-checked.

[0066] In one implementation, a preset SQL statement checker may be used to perform syntax check on the target SQL statement.

[0067] In step 305, it is determined that a syntax error exists in the target SQL statement, and syntax adjustment information of the target SQL statement is obtained.

[0068] Optionally, if it is determined that the target SQL statement contains a syntax error, syntax adjustment information for the target SQL statement may be provided based on a preset SQL statement checker. Alternatively, if it is determined that the target SQL statement contains a syntax error, the syntax error in the target SQL statement may be provided to a relevant person, who may provide syntax adjustment information for the target SQL statement, thereby obtaining the syntax adjustment information for the target SQL statement.

[0069] In step 306, the target SQL statement is automatically adjusted based on the grammar adjustment information, or the grammar adjustment information is provided to a target object, and the target SQL statement is adjusted based on the target object's operation on the adjustment information. The target object can be a data developer or other person, and this is not specifically limited.

[0070] Optionally, after adjusting the target SQL statement, step 307 may be executed.

[0071] In step 307 , a job for executing the target SQL statement is determined, and upstream and downstream jobs having dependency relationships with the job are determined based on the source table, the target table, the source system information, and the target table system information.

[0072] In the embodiment of the present application, step 307 can be implemented in any of the ways in the embodiments of the present application. The embodiment of the present application does not limit this and will not be described in detail.

[0073] In step 308, job scheduling configuration information is generated based on upstream and downstream jobs, services, and job dependencies.

[0074] In the embodiment of the present application, step 308 can be implemented in any of the ways in the embodiments of the present application. The embodiment of the present application does not limit this and will not be described in detail.

[0075] By implementing the embodiments of the present application, syntax judgment of SQL statements can be supported, thereby reducing the time data developers spend writing SQL statements. When SQL statements need to be adjusted, the function of modifying SQL statements is provided, which can greatly simplify the work of data developers and improve fault tolerance.

[0076] Optionally, in some embodiments of the present application, after combing the data requirement information to obtain the physical models of the source table and the target table, the data mapping information between the source table and the target table, the source system information and the target table system information in the data requirement information, the information obtained after the combing operation can be automatically verified. In one implementation, if Figure 4 As shown, the method of automatically checking errors and optimization points in the data requirement information may include the following steps:

[0077] In step 401 , the physical models of the source table and the target table, the data mapping information between the source table and the target table, the source system information, and the target table system information are verified.

[0078] Optionally, the physical models of the source and target tables, the data mapping information between the source and target tables, the source system information, and the target table system information can be verified to verify whether this information is normal. In one implementation, it can be determined whether the physical models of the source and target tables, the data mapping information between the source and target tables, the source system information, and the target table system information are empty; the target table fields can be obtained from the target table's physical model, and it can be determined whether the target table fields are compatible with the type of processing of the source table fields; it can be determined whether the source and target table fields exist; it can be determined whether the service method in the data mapping information complies with the specification, and it can be determined whether the processing rules in the data mapping information comply with SQL semantic rules.

[0079] For example, it can determine whether the source system, source table, field, processing rules, and target table-related information are empty; determine whether the target table field is compatible with the type of source table field processing; determine whether the source table field and target table field exist; determine whether the service method complies with the specifications; determine whether the data caliber processing rules comply with SQL semantic rules, etc.

[0080] In step 402, the verification result is determined, and corresponding modification and / or optimization suggestion information is generated according to the verification result.

[0081] For example, processing rules can be optimized to simplify cumbersome processing rules.

[0082] In step 403 , the physical models of the source table and the target table, the data mapping information between the source table and the target table, the source system information, and the target table system information are modified accordingly according to the modification and / or optimization suggestion information.

[0083] It can be seen that this application supports automatic verification of errors and optimisation points in data requirements, improves the quality of data requirement documents from the source, further reduces the time for subsequent development rework, and thus further improves work efficiency.

[0084] Figure 5 This is a block diagram of a device for generating job scheduling configuration information provided by an embodiment of the present application. Figure 5 As shown, the job scheduling configuration information generating device may include: a first acquiring module 501 , a second acquiring module 502 , a first generating module 503 , a determining module 504 and a second generating module 505 .

[0085] Specifically, the first acquisition module 501 is used to acquire data demand information to be processed.

[0086] The second acquisition module 502 is used to sort out the data requirement information to obtain the physical models of the source table and the target table, the data mapping information between the source table and the target table, the source system information and the target table system information.

[0087] In one possible implementation, the second acquisition module 502 is specifically used to: perform text fuzzy matching and natural language processing on the data demand information in combination with a text library and a machine learning model to obtain the physical models of the source table and the target table in the data demand information, the data mapping information between the source table and the target table, the source system information and the target table system information; wherein, the physical model of the source table includes the source table and the source table fields, and the physical model of the target table includes the target table and the target table fields; the data mapping information includes the service mode and the processing rules corresponding to the service mode.

[0088] The first generation module 503 is used to generate a target SQL statement based on the physical models of the source table and the target table, as well as the data mapping information between the source table and the target table, based on the statement characteristics of the corresponding database; wherein the target SQL statement is used to map the data in the source table to the corresponding fields of the target table.

[0089] In one possible implementation, the first generation module 503 is specifically used to: determine the service mode and processing rules used when mapping each field in the source table to the corresponding field in the target table based on the physical models of the source table and the target table, as well as the data mapping information between the source table and the target table; splice the fields in the source table into SQL statements based on the statement characteristics of the corresponding database through the corresponding service mode and processing rules, and assign the spliced ​​SQL statements to the fields in the target table corresponding to the fields in the source table, and splice the SQL statements of all fields in the target table to obtain the target SQL statements.

[0090] The determination module 504 is used to determine a job for executing a target SQL statement, and determine upstream and downstream jobs that have dependency relationships with the job based on the source table, the target table, the source system information, and the target table system information.

[0091] The second generation module 505 is used to generate job scheduling configuration information based on upstream and downstream jobs, businesses, and job dependencies. In one possible implementation, the second generation module 505 is specifically used to: generate a job flow name for the target table based on the job's execution frequency, function abbreviation information, and time zone information; generate a job name for the job based on the name and execution frequency of the target table; obtain the source table's job flow name, condition name, dependent component information, and pre-dependent business date offset rule from the source table; and enter the target table's job flow name, job name, source table's job flow name, condition name, dependent component information, and pre-dependent business date offset rule into a preset job scheduling configuration template to obtain the job scheduling configuration information.

[0092] Optionally, in some embodiments of the present application, such as Figure 6 As shown, the apparatus may further include: a grammar checking module 606, a third acquisition module 607, and an adjustment module 608. The grammar checking module 606 is used to perform grammar checking on the target SQL statement; the third acquisition module 607 determines whether there is a grammar error in the target SQL statement and obtains grammar adjustment information of the target SQL statement; the adjustment module 608 is used to automatically adjust the target SQL statement according to the grammar adjustment information, or provide the grammar adjustment information to the target object and adjust the target SQL statement based on the target object's operation on the adjustment information. Figure 6 601-605 and Figure 5 501-505 have the same function and structure.

[0093] Optionally, in some embodiments of the present application, such as Figure 7As shown, the apparatus may further include: a verification module 709, a third generation module 710, and a modification module 711. The verification module 709 is configured to verify the physical models of the source table and the target table, the data mapping information between the source table and the target table, the source system information, and the target table system information; the third generation module 710 is configured to determine the verification result and generate corresponding modification and / or optimization suggestion information based on the verification result; and the modification module 711 is configured to modify the physical models of the source table and the target table, the data mapping information between the source table and the target table, the source system information, and the target table system information based on the modification and / or optimization suggestion information.

[0094] In one possible implementation, the verification module 709 is specifically used to: determine whether the physical models of the source table and the target table, the data mapping information between the source table and the target table, the source system information, and the target table system information are empty; obtain the target table fields from the physical model of the target table, and determine whether the target table fields are compatible with the type of processing of the source table fields; determine whether the source table fields and the target table fields exist; determine whether the service mode in the data mapping information complies with the specification, and determine whether the processing rules in the data mapping information comply with SQL semantic rules. Figure 7 701-708 and Figure 6 601-608 have the same function and structure.

[0095] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0096] By implementing the embodiments of this application, SQL statements can be generated based on the source table, target table, service mode, and processing rules in the data demand information, and job scheduling configuration information can be generated based on the source table and system information of the job. This intelligent implementation throughout the entire process, which runs through the development process of data developers, can simplify the work of data developers and streamline the development process. With the increasing demand for data, it is crucial to improving development and work efficiency. Figure 8 FIG. 2 is a block diagram of an electronic device 200 according to an exemplary embodiment. Figure 8 As shown, the electronic device may be a server. The electronic device 200 may include:

[0097] The memory 210 and the processor 220, a bus 230 connecting different components (including the memory 210 and the processor 220), the memory 210 stores instructions executable by the processor 220; wherein, the processor 220 is configured to execute the instructions to implement the business recommendation method or the behavior data incremental prediction model generation method described in the embodiment of the present application.

[0098] Bus 230 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0099] The electronic device 200 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by the electronic device 200, including volatile and non-volatile media, removable and non-removable media. The memory 210 can also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 240 and / or cache memory 250. The electronic device 200 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 260 can be used to read and write non-removable, non-volatile magnetic media ( Figure 8 Not shown, often called a "hard drive"). Although Figure 8 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 230 via one or more data medium interfaces. Memory 210 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present application.

[0100] A program / utility 280 having a set (at least one) of program modules 270 may be stored, for example, in memory 210. Such program modules 270 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 270 generally implement the functions and / or methods of the embodiments described herein.

[0101] The electronic device 200 can also communicate with one or more external devices 290 (e.g., a keyboard, a pointing device, a display 291, etc.), one or more devices that enable a user to interact with the electronic device 200, and / or any device that enables the electronic device 200 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication can occur via an input / output (I / O) interface 292. Furthermore, the electronic device 200 can also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 293. As shown, the network adapter 293 communicates with other modules of the electronic device 200 via a bus 230. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 200, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0102] The processor 220 executes various functional applications and data processing by running programs stored in the memory 210 .

[0103] It should be noted that the implementation process and technical principles of the electronic device of this embodiment can be found in the aforementioned explanation of the method for generating job scheduling configuration information in the embodiment of the present application, and will not be repeated here.

[0104] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium.

[0105] When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is enabled to execute the method for generating job scheduling configuration information as described above.

[0106] To implement the above embodiments, the present application further provides a computer program product, which includes a computer program that, when executed by a processor of an electronic device, implements the above-mentioned method for generating job scheduling configuration information.

[0107] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0108] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for generating job scheduling configuration information, characterized in that: include: Obtaining data requirement information to be processed; Sorting the data requirement information to obtain physical models of the source table and the target table, data mapping information between the source table and the target table, source system information, and target table system information; generating a target SQL statement based on the physical models of the source table and the target table, data mapping information between the source table and the target table, and statement characteristics of the corresponding database; wherein the target SQL statement is used to map the data in the source table to corresponding fields in the target table; Determine a job for executing a target SQL statement, and determine upstream and downstream jobs that have a dependency relationship with the job based on the source table, the target table, the source system information, and the target table system information; Generate job scheduling configuration information based on the upstream and downstream jobs, services, and job dependencies; The generating of job scheduling configuration information according to the upstream and downstream jobs, services, and job dependencies includes: generating a job flow name of the target table according to the execution frequency, function abbreviation information and time zone information of the job; Generate a job name for the job according to the name of the target table and the execution frequency; Acquire the job flow name, condition name, dependent component information, and pre-dependent business date offset rule of the source table according to the source table; Fill the job flow name of the target table, the job name of the job, the job flow name of the source table, the condition name, the dependent component information and the pre-dependent business date offset rule into the preset job scheduling configuration template to obtain the job scheduling configuration information.

2. The method according to claim 1, wherein The combing operation on the data requirement information to obtain the physical models of the source table and the target table, the data mapping information between the source table and the target table, the source system information, and the target table system information in the data requirement information includes: Performing text fuzzy matching and natural language processing on the data requirement information in combination with a text library and a machine learning model to obtain the physical models of the source table and the target table, data mapping information between the source table and the target table, source system information, and target table system information in the data requirement information; The physical model of the source table includes the source table and source table fields, and the physical model of the target table includes the target table and target table fields; the data mapping information includes a service mode and processing rules corresponding to the service mode.

3. The method according to claim 1, wherein Generating a target SQL statement based on the physical models of the source table and the target table, data mapping information between the source table and the target table, and statement characteristics of a corresponding database includes: Determining, based on the respective physical models of the source table and the target table, and data mapping information between the source table and the target table, a service mode and processing rules used when mapping each field in the source table to a corresponding field in the target table; The fields in the source table are spliced ​​into SQL statements through corresponding service methods and processing rules based on the statement characteristics of the corresponding database, and the spliced ​​SQL statements are assigned to the fields in the target table corresponding to the fields in the source table. The SQL statements of all fields in the target table are spliced ​​to obtain the target SQL statement.

4. The method according to any one of claims 1 to 3, characterized in that Also includes: Performing syntax check on the target SQL statement; Determining that a syntax error exists in the target SQL statement, and obtaining syntax adjustment information of the target SQL statement; The target SQL statement is automatically adjusted according to the grammar adjustment information, or the grammar adjustment information is provided to a target object, and the target SQL statement is adjusted based on the target object's operation on the adjustment information.

5. The method according to claim 1, wherein After combing the data requirement information to obtain the physical models of the source table and the target table, data mapping information between the source table and the target table, source system information, and target table system information in the data requirement information, the method further includes: Verifying the physical models of the source table and the target table, data mapping information between the source table and the target table, source system information, and target table system information; Determine the verification results and generate corresponding modification and / or optimization suggestion information based on the verification results; According to the modification and / or optimization suggestion information, corresponding modifications are made to the physical models of the source table and the target table, the data mapping information between the source table and the target table, the source system information, and the target table system information.

6. The method according to claim 5, wherein The verifying of the physical models of the source table and the target table, the data mapping information between the source table and the target table, the source system information, and the target table system information includes: Determine whether the physical models of the source table and the target table, the data mapping information between the source table and the target table, the source system information, and the target table system information are empty; Obtaining a target table field from the physical model of the target table, and determining whether the target table field is compatible with the type of source table field processing; Determine whether the source table field and the target table field exist; It is determined whether the service mode in the data mapping information complies with the specification, and whether the processing rules in the data mapping information comply with the SQL semantic rules.

7. A job scheduling configuration information generating device, characterized in that: include: A first acquisition module is used to obtain data requirement information to be processed; A second acquisition module is configured to sort out the data requirement information to obtain the physical models of the source table and the target table, data mapping information between the source table and the target table, source system information, and target table system information. a first generation module, configured to generate a target SQL statement based on the physical models of the source table and the target table, data mapping information between the source table and the target table, and statement characteristics of the corresponding database; wherein the target SQL statement is used to map the data in the source table to corresponding fields in the target table; a determination module, configured to determine a job for executing a target SQL statement, and to determine upstream and downstream jobs having a dependency relationship with the job based on the source table, the target table, the source system information, and the target table system information; A second generation module is used to generate job scheduling configuration information based on the upstream and downstream jobs, services, and job dependencies; The second generation module is specifically configured to: Generate a job flow name of the target table according to the execution frequency, function abbreviation information and time zone information of the job; Generate a job name for the job according to the name of the target table and the execution frequency; Acquire the job flow name, condition name, dependent component information, and pre-dependent business date offset rule of the source table according to the source table; Fill the job flow name of the target table, the job name of the job, the job flow name of the source table, the condition name, the dependent component information and the pre-dependent business date offset rule into the preset job scheduling configuration template to obtain the job scheduling configuration information.

8. The device according to claim 7, wherein The second acquisition module is specifically used for: Performing text fuzzy matching and natural language processing on the data requirement information in combination with a text library and a machine learning model to obtain the physical models of the source table and the target table, data mapping information between the source table and the target table, source system information, and target table system information in the data requirement information; The physical model of the source table includes the source table and source table fields, and the physical model of the target table includes the target table and target table fields; the data mapping information includes a service mode and processing rules corresponding to the service mode.

9. The device according to claim 7, wherein The first generating module is specifically configured to: Determining, based on the respective physical models of the source table and the target table, and data mapping information between the source table and the target table, a service mode and processing rules used when mapping each field in the source table to a corresponding field in the target table; The fields in the source table are spliced ​​into SQL statements through corresponding service methods and processing rules based on the statement characteristics of the corresponding database, and the spliced ​​SQL statements are assigned to the fields in the target table corresponding to the fields in the source table. The SQL statements of all fields in the target table are spliced ​​to obtain the target SQL statement.

10. The device according to any one of claims 7 to 9, characterized in that Also includes: A syntax checking module, used for performing syntax checking on the target SQL statement; A third acquisition module determines whether the target SQL statement has a syntax error and acquires syntax adjustment information of the target SQL statement; An adjustment module is configured to automatically adjust the target SQL statement according to the grammar adjustment information, or to provide the grammar adjustment information to a target object and adjust the target SQL statement based on an operation of the target object on the adjustment information.

11. The device according to claim 7, wherein Also includes: A verification module, configured to verify the physical models of the source table and the target table, data mapping information between the source table and the target table, source system information, and target table system information; A third generating module is used to determine the verification result and generate corresponding modification and / or optimization suggestion information according to the verification result; The modification module is used to modify the physical models of the source table and the target table, the data mapping information between the source table and the target table, the source system information and the target table system information accordingly according to the modification and / or optimization suggestion information.

12. The device according to claim 11, wherein The verification module is specifically used for: Determine whether the physical models of the source table and the target table, the data mapping information between the source table and the target table, the source system information, and the target table system information are empty; Obtaining a target table field from the physical model of the target table, and determining whether the target table field is compatible with the type of source table field processing; Determine whether the source table field and the target table field exist; It is determined whether the service mode in the data mapping information complies with the specification, and whether the processing rules in the data mapping information comply with the SQL semantic rules.

13. An electronic device, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

15. A computer program product comprising a computer program, wherein The computer program implements the steps of the method according to any one of claims 1 to 6 when executed by a processor.

Images