Batch scheduling processing method and device, processor and electronic equipment
By generating and adjusting batch scheduling panoramas to update batch job dependencies in large systems, the problem of inefficient batch scheduling version update is solved and a more efficient processing process is achieved.
Patent Information
- Application Number
- CN202411832149.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-12
- Publication Date
- 2025-04-18
AI Technical Summary
In large systems, during the batch scheduling version update process, developers need to manually edit the SQL statements of the database, resulting in high workload and low efficiency.
By obtaining the original execution statement of the database, a batch scheduling panorama is generated, the panorama is adjusted to update the dependencies, and finally the adjusted panorama is converted into a target execution statement and published to the production environment.
It reduces the workload of manually modifying SQL statements, improves the processing efficiency of batch scheduling, and solves the problem of low efficiency.
Smart Images
Figure CN119848067A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of financial technology, and more specifically, to a batch scheduling processing method and device, a processor, and an electronic device. Background Art
[0002] In order to improve the batch development of platform applications, many applications use a distributed batch job scheduling framework to implement batch job operation and scheduling processing of each application.
[0003] In the related technology, developers mainly manually edit Excel template files of batch job scheduling and definition information, and use scripts through the continuous delivery pipeline to import the Excel files into the distributed batch console of the corresponding release.
[0004] However, when using distributed batches for R&D and operation and maintenance, since large systems (for example, large banking systems) are divided into multiple references, and the batch job schedules of these applications in different environments such as R&D, testing, and production are also different. In actual application scenarios, batch job schedules are usually added or modified according to business needs, but for the above-mentioned large systems, if the method of manually editing batch job schedule information in related technologies is still used to update the batch schedule version, it will take a lot of time to identify the SQL statements of the database and edit or modify multiple SQL statements in the database data, which increases the workload of developers and consumes a lot of time, resulting in technical problems of low efficiency in the batch schedule version update process.
[0005] Currently, no effective solution has been proposed to address the problem of low efficiency in batch scheduling processing in the above-mentioned related technologies. Summary of the invention
[0006] The main purpose of the present application is to provide a batch scheduling processing method and device, a processor and an electronic device to solve the problem of low efficiency in the batch scheduling processing process in the related art.
[0007] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a batch scheduling processing device is provided. The device includes: a first acquisition unit, which is used to acquire the original execution statement of the database, wherein the original database data includes the job information of the batch scheduling task; a first processing unit, which is used to generate a first batch scheduling panorama based on the original execution statement, wherein the first batch scheduling panorama displays the initial dependency relationship between batch jobs in the target application in the form of a visual diagram; an adjustment unit, which is used to adjust the first batch scheduling panorama to obtain an adjusted second batch scheduling panorama, wherein the second batch scheduling panorama displays the updated dependency relationship between batch jobs in a visual form; a conversion unit, which is used to convert the second batch scheduling panorama into a target execution statement of the database, and publish the target execution statement to the target production environment.
[0008] Optionally, in the batch scheduling processing device provided in the embodiment of the present application, the above-mentioned first processing unit includes: a first query module, which is used to use the target cursor object to query the batch job definition information of the target application from the original execution statement; a first processing module, which is used to use the target cursor object to traverse a set of data tables in the test environment database to obtain column data in the query data set; a second processing module, which is used to generate a first array consisting of row data and column data with each batch job definition information as row data; a second query module, which is used to query the first array according to the job grouping information to obtain a set of batch job data information with initial dependency relationships; a third processing module, which is used to generate a first batch scheduling panorama based on a set of batch job information with initial dependency relationships.
[0009] Optionally, in the batch scheduling processing device provided in an embodiment of the present application, the above-mentioned first query module includes: a first creation sub-module, used to use the test environment database connection information to create a connection between the target standard library and the database; a second creation sub-module, used to create a target cursor object, and obtain batch job definition information of the target application by cyclically using the target cursor object.
[0010] Optionally, in the batch scheduling processing device provided in the embodiment of the present application, the above-mentioned third processing module includes: a first processing sub-module, used to determine each batch job as a node object; a second processing sub-module, used to determine each dependency in the initial dependency as a connection line; a third processing sub-module, used to generate a first set of graph data based on the node objects and the connections, wherein pointers are used as pointing information between two node objects; and a fourth processing sub-module, used to draw a first batch scheduling panoramic map based on the first set of graph data.
[0011] Optionally, in the batch scheduling processing device provided in the embodiment of the present application, the above-mentioned conversion unit includes: an extraction module, which is used to extract a second set of graph data from the second batch scheduling panorama; a first parsing module, which is used to use the N-ary tree structure in the target standard library to parse the second set of graph data to obtain element information of each node object, wherein a batch job is regarded as a node object, and the element information of each node object represents the data information defined by a batch job, and N is a positive integer greater than or equal to 2; a fourth processing module, which is used to store the element information of each node object as a row of data to obtain a second array composed of element data of multiple node objects; a fifth processing module, which is used to fill in the preset program statement structure in the form of byte stream data based on the data information of the second array, obtain the target execution statement, and publish the target execution statement to the target production environment.
[0012] Optionally, in the batch scheduling processing device provided in the embodiment of the present application, the above-mentioned first parsing module includes: a fourth processing sub-module, used to determine the file object corresponding to the second batch scheduling panorama as a target object tree, wherein the target object tree includes multiple node objects, one node object includes multiple child nodes, and one child node represents a worksheet in the file object; a fifth processing sub-module, used to obtain element information of each node object by traversing and parsing the child nodes and pointer information of each node object in the multiple node objects, wherein the pointer information represents the dependency relationship between each node object and other node objects.
[0013] Optionally, in the batch scheduling processing device provided in the embodiment of the present application, the above-mentioned first parsing module includes: a first acquisition sub-module, used to sequentially acquire each node object from multiple node objects as a current node object; a fifth processing sub-module, used to obtain the attribute information of the current batch job corresponding to the current node object by traversing the current set of child nodes and current pointer information of the current node object, wherein the attribute information includes the job name, execution time and execution mode; a sixth processing sub-module, used to store the attribute information of the current batch job as a key-value pair, and determine the element information found based on the key-value pair as the element information of the current node object.
[0014] Optionally, in the batch scheduling processing device provided in an embodiment of the present application, the above-mentioned adjustment unit includes: an adjustment module, which is used to adjust the dependencies and job metadata of at least part of the batch jobs in the first batch scheduling panorama based on target business needs to obtain a second batch scheduling panorama.
[0015] Optionally, the batch scheduling processing device provided in the embodiment of the present application also includes: a conversion unit, which is used to convert the target execution statement into a third set of graph data based on the target standard library after publishing the target execution statement to the target production environment; and a second processing unit, which is used to generate a third batch scheduling panorama based on the third set of graph data.
[0016] Through the above-mentioned embodiments provided by the present application, the original execution statement of the database is converted into the first batch scheduling panorama using a tool program, and then the first batch scheduling panorama is adjusted according to business needs, and the adjusted second batch scheduling panorama is converted into the target execution statement of the database, thereby publishing the target execution statement to the target production environment. In other words, the technical solution of the present application reduces the workload of manually directly modifying the original execution statement by more conveniently modifying and adjusting the dependency relationship and job definition information between each batch job in the batch scheduling panorama, improves the processing efficiency of batch scheduling, and solves the technical problem of low processing efficiency of batch scheduling in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0018] Figure 1 is a flow chart of a batch scheduling processing method provided according to an embodiment of the present application;
[0019] Figure 2 It is a schematic diagram of mutual conversion between a batch scheduling panorama and database data provided according to an embodiment of the present application;
[0020] Figure 3 is an overall schematic diagram of a batch scheduling processing method provided according to an embodiment of the present application;
[0021] Figure 4 is a two-dimensional array generated by the traversed data provided in an embodiment of the present application;
[0022] Figure 5 It is a panoramic view grouped by job provided according to an embodiment of the present application;
[0023] Figure 6 It is an overall schematic diagram of a batch scheduling panorama provided according to an embodiment of the present application;
[0024] Figure 7 It is a tree structure of a batch scheduling panorama provided according to an embodiment of the present application;
[0025] Figure 8It is the node object element information in the key-value pair format provided according to the embodiment of the present application;
[0026] Fig. 9 It is job definition data in a two-dimensional array format obtained by traversing multiple node objects according to an embodiment of the present application;
[0027] Fig.10 This is a specific example of an Excel version update file provided according to an embodiment of the present application;
[0028] Fig.11 It is a specific example of a standard SQL statement provided according to an embodiment of the present application;
[0029] Fig.12 is a schematic diagram of a batch scheduling processing device provided according to an embodiment of the present application;
[0030] Fig.13 This is a structural frame diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION
[0031] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0032] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0034] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. For example, an interface is set up between this system and relevant users or institutions to provide users with corresponding operation entrances for users to choose to agree or reject the automated decision results; if the user chooses to reject, the expert decision process will be entered.
[0035] Example 1
[0036] According to an embodiment of the present application, a batch scheduling processing method embodiment is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0037] The present invention is described below in conjunction with preferred implementation steps. Figure 1 is a flow chart of a batch scheduling processing method provided in an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:
[0038] Step S102, obtaining an original execution statement from a database, wherein the original execution statement includes job information of a batch scheduling task;
[0039] Step S104, generating a first batch scheduling panorama based on the original execution statement, wherein the first batch scheduling panorama displays the initial dependency relationship between batch jobs in the target application in the form of a visual diagram;
[0040] Step S106, adjusting the first batch scheduling panorama to obtain an adjusted second batch scheduling panorama, wherein the second batch scheduling panorama displays the updated dependency relationship between the batch jobs in a visual form;
[0041] Step S108, converting the second batch scheduling panorama into a target execution statement of a database, and publishing the target execution statement to a target production environment.
[0042] In an embodiment of the present application, distributed batch job scheduling information can be but is not limited to two data storage forms, one is the data in the database, including the test environment database and the production environment database; the other is the scheduling panorama, including the full progress map of the test environment and the panoramic map of the production environment.
[0043] Among them, distributed batch can be but not limited to the abbreviation of distributed batch job scheduling framework, which is currently widely used in batch job operation and scheduling processing of various applications. Distributed batch has the ability to schedule multiple executors to collaboratively complete batch job operation, as well as corresponding management and monitoring.
[0044] Batch jobs can be, but are not limited to, programs that process data or files in batches and are unique within the distributed batch job scheduling framework.
[0045] Scheduling can be, but is not limited to, the topology of the dependencies and scheduling logic of distributed batch jobs. In an application, the execution time, sequence, and interdependencies of all batch jobs are determined when they are published and stored in the batch job definition table for the distributed batch scheduling framework to identify and execute the job.
[0046] The scheduling panorama (also known as the batch scheduling panorama) intuitively displays the dependencies between batch jobs in the form of a visual diagram. The panorama contains all the defined batch jobs and job metadata in an application. Developers can manually adjust the dependencies and job metadata of each batch job in the panorama.
[0047] The embodiments of the present application can achieve the following Figure 2 The closed-loop data conversion link shown is automatically converted from the distributed batch scheduling panorama to the file format required for the release version to the production environment. That is, the mutual conversion between the database data and the batch scheduling panorama can be realized, thereby avoiding manual editing and publishing by developers, greatly reducing the workload of the release version and the problem of manual filling errors, and can also quickly and accurately release the new version of the batch job schedule. The following will be described in detail in conjunction with a specific embodiment of the process of using the tool program to convert the batch scheduling panorama and database data.
[0048] The above-mentioned tool programs may include but are not limited to Python scripts and standard libraries, wherein the standard library in Python provides many modules for processing files, system calls, network communications, etc.
[0049] In order to understand the above batch scheduling method more clearly, Figure 3 The data processing shown explains this.
[0050] It is easy to understand that for the original database, some enterprises may have a batch scheduling panorama corresponding to the original execution statement in the original database, but some enterprises may not have a batch scheduling panorama corresponding to the original execution statement in the original database. The following introduces how to implement the release of batch scheduling versions in two cases.
[0051] (I) There is no batch scheduling panorama corresponding to the original execution statement
[0052] S11, obtain SQL data from the test environment database;
[0053] The test environment is suitable for developers and testers to conduct unit testing, integration testing, chaos testing and other testing environments, which require frequent updates and releases of batch job information. Developers can adjust the job definition in the panorama, automatically update the data in the test environment database, and complete the rapid release of the test environment.
[0054] After the test is completed, the job definition information that needs to be released can be exported and released to the production environment through the continuous delivery pipeline.
[0055] The SQL (Structured Query Language) data obtained may be, but is not limited to, the original execution statements in the database. If the SQL data is directly modified manually to update and publish batch job information, it will take a lot of time to modify the SQL statements in multiple places and publish the modified SQL data to the production environment.
[0056] S12, using a tool program to convert the SQL data into initial graph data;
[0057] S13, generating an initial batch scheduling panorama based on the graph data;
[0058] The initial batch scheduling panorama intuitively displays the definition information of each existing batch job and the dependencies between each batch job.
[0059] S14, adjusting the initial batch scheduling panorama according to business requirements to obtain an adjusted batch scheduling panorama;
[0060] S15, using a tool program, converting the adjusted batch scheduling panorama into adjusted graph data;
[0061] S16, converting the adjusted graph data into adjusted SQL data;
[0062] At this point, the data in the test environment database has been updated and adjusted.
[0063] S17, extract the adjusted SQL data from the test environment database, and publish the adjusted SQL data to the production environment database through the continuous delivery pipeline.
[0064] In order to facilitate viewing by users in the production environment, the same tool program can also be used to convert the adjusted database data in the production environment (ie, the adjusted SQL data) into graph data, thereby obtaining a panoramic view of the batch scheduling of the production environment.
[0065] The production environment is the environment that will be used for external services after the system is launched. Once released, it cannot be changed at will. After the job definition information of the test environment is released and imported into the production environment database, the new version can be released and a panoramic view of the production environment is generated. The panoramic view of the production environment can only be used to view the latest version of the batch job definition information.
[0066] (II) There is a batch scheduling panorama corresponding to the original execution statement
[0067] S21, directly modifying the batch scheduling panorama corresponding to the original execution statement according to business requirements to obtain a modified batch scheduling panorama;
[0068] The modification content of the batch scheduling panorama corresponding to the original execution statement includes but is not limited to modifying the dependency relationship between each batch job and the job metadata of at least part of the batch jobs.
[0069] S22, reading the modified batch scheduling panorama in the test environment and extracting the map data;
[0070] The extracted graph data is stored in memory for program processing.
[0071] S23, converting the extracted graph data into database data;
[0072] That is, the graph data is converted into updated SQL data using Python scripts and standard libraries, and the updated SQL data is synchronously updated to the test environment database.
[0073] S24, converting and extracting updated SQL data from the test environment database;
[0074] S25, publish the updated SQL data to the production environment.
[0075] It should be noted that the SQL data converted and extracted from the test environment database can be transferred to Figure 3 The "database conversion to panorama" processing module shown in the figure can also flow to the "continuous delivery" module to release the version to the production environment.
[0076] besides, Figure 3 The processing steps of the "production environment batch scheduling module" in the second part area are as follows:
[0077] Step 1: The "continuous delivery pipeline module" delivers versions in the form of SQL data and directly updates the production environment database;
[0078] Steps: The same process as converting the "test environment database" to the "test environment scheduling panorama" is used to convert the SQL data of the "production environment database" to the "production environment scheduling panorama" for the operation and maintenance personnel to view.
[0079] In the production environment, the panorama is edited and converted into SQL data to update the production environment database.
[0080] It can be seen that the technical solution of this application mainly involves Figure 3 The data processing program shown is also the batch scheduling panorama and SQL file generation program. The job definition and inter-job dependencies can be obtained by parsing and modifying the updated distributed batch scheduling diagram, and converted into the SQL file and Excel template file required for the release version through Python scripts and standard libraries, so that application developers can complete the release version by maintaining the scheduling panorama, reducing the workload.
[0081] At the same time, by parsing the distributed batch job definition table in the test environment database, the job definition and the dependency relationship between jobs can be obtained, converted into a directed graph structure, and a panoramic view of the distributed batch scheduling can be drawn through Python scripts and standard libraries, allowing developers to easily view the overall batch scheduling of the application.
[0082] As an optional example, the above method generates a first batch scheduling panorama based on the original execution statement, including:
[0083] Using the target cursor object, query the batch job definition information of the target application from the original execution statement;
[0084] Using the target cursor object, traverse a set of data tables in the test environment database to obtain the column data in the query data set;
[0085] Taking each batch job definition information as row data, generating a first array consisting of row data and column data;
[0086] According to the job grouping information, query the first array to obtain a set of batch job data information with an initial dependency relationship;
[0087] Based on a set of batch job information with initial dependencies, a first batch scheduling panorama is generated.
[0088] The specific steps are as follows:
[0089] S31, before converting the database data into a batch scheduling panorama using a Python script and a standard library, it is necessary to pre-set metadata variable information as shown in Table 1 below for drawing a distributed batch scheduling panorama;
[0090] Table 1
[0091]
[0092] S32, at this time, the test environment database is used as the data source. Because the database data itself is structured data, the tool program uses the Python standard library pymsql to easily obtain the batch job data in the database. The connect method under pymsql usually uses the MySQL_URL variable to establish a connection with the MySQL database and create a new cursor object "cursor". Before the database connection is closed, the cursor object can be used cyclically to obtain batch job definition data.
[0093] S33, obtaining batch job definition information of the target application by cyclically using the cursor object;
[0094] Optionally, query the batch job definition information of the target application from the original execution statement in the following way:
[0095] Use the test environment database connection information to create a connection between the target standard library and the database;
[0096] Create a target cursor object and obtain the batch job definition information of the target application by looping through the target cursor object.
[0097] Specifically, a simple SQL query statement is composed using the APP_NAME variable, and all batch job definition information of the application is queried from the database using the fetchall method of the cursor object "cursor". The returned data is all rows of the query result set, where one row of data is a tuple encapsulated from each batch job definition information.
[0098] By adopting the above metadata variables, such as FILE_PATH, GRAPH_PATH, APP_NAME, etc., which can be specified in other ways, it can adapt to different usage scenarios and needs. This configurable design enables the technical solution to flexibly respond to various environments and demand changes, and is particularly suitable for projects that need to quickly deploy and adjust scheduling information in different environments, which can significantly improve the efficiency and flexibility of scheduling management.
[0099] S34, in one data connection, use cursor to sequentially query the data tables in the test environment database such as the batch job definition information job_def table, the batch job dependency relationship job_depend table, the job group definition information set_def table, etc. as shown in Table 2 below;
[0100] Table 2
[0101]
[0102]
[0103] S35, traverse the queried data and store them into a two-dimensional array in the form of rows and columns according to the query result set;
[0104] For example, a two-dimensional array stored in the form of rows and columns according to the query result set is recorded as the first array.
[0105] Among them, the above steps S31 to S34 will execute the data information of the 7 tables shown in Table 2, and in step S35, 7 two-dimensional arrays will be generated. The table data will be stored in the memory through the 7 two-dimensional arrays. For details, please refer to Figure 4 shown.
[0106] After a group of batch job information with initial dependency is obtained according to the above steps S33 to S35, a first batch panoramic image is generated based on the batch job information, specifically including:
[0107] Identify each batch job as a node object;
[0108] Determine each dependency in the initial dependencies as a line;
[0109] Based on the node objects and the links, a first set of graph data is generated, wherein a pointer is used as pointing information between two node objects;
[0110] Based on the first set of graph data, a first batch scheduling panorama is drawn.
[0111] S36, searching according to the job group information GROUP_NAME, and filtering out a group of batch job information with logical dependencies;
[0112] Each batch job is considered as a "node" node object, and the data information of the batch job is the key-value pair data of the "node" node object. Each dependency is considered as a "line" connection, and a directed graph structure is generated in the memory. Pointers are used as pointing information between two "node" objects.
[0113] S37, draw the first batch scheduling panorama.
[0114] Specifically, a single batch job information is drawn into a rectangular area, and the GRAPH_ATTRIBUTES variable shown in Table 1 sets different attributes for the rectangular areas of different batch jobs, such as color, font, size, etc.
[0115] The data information in the rectangular area includes the job name, job description information, job submission type, job execution time, etc. The batch job dependency is represented by arrows between the rectangular areas, from the previous job to the next job, and the arrow direction can also be flipped through the GRAPH_ATTRIBUTES variable configuration.
[0116] In addition to the method of generating the first batch scheduling panorama in the above steps S31 to S37, in order to provide the user with more intuitive visual information, the tool program can also
[0117] The colors in the GRAPH_ATTRIBUTES variable correspond to the job execution type. The rectangular area marking the batch job gives a panoramic view of the batch schedule. Figure 1 Some business information.
[0118] For example, red represents daily cutting operations, green represents manual operations, yellow represents open operations, and gray represents ordinary batch operations.
[0119] Repeat the above steps S31 to S37 to traverse and obtain the database data information of all job groups GROUP_NAME, and draw Figure 5 A panoramic view of multiple sheet pages grouped by job in the dashed box.
[0120] according to Figure 5 The drawing method shown in the figure can finally draw a picture like Figure 6 The complete panorama shown in the figure contains the batch job information and dependencies of all job groups under the target application. The panorama generated and transformed by the test environment database of jobs belonging to different GROUP_NAME job groups is provided to developers for job dependency checking to assist in version release, and the panorama generated and transformed by the production environment database is provided to operation and maintenance personnel for verification and inspection with the test environment panorama.
[0121] In combination with the description in the above embodiments, it can be seen that the standard library of the Python language and the implementation drawing technology used in the technical solution of the present application can convert database data into a panoramic view of batch job scheduling, thereby intuitively displaying the dependencies and job information between each batch job to the developer, thereby improving the developer's processing efficiency for batch scheduling.
[0122] As an optional example, the above-mentioned conversion of the second batch scheduling panorama into a target execution statement of the database and publishing the target execution statement to the target production environment includes:
[0123] Extracting a second set of image data from the second batch scheduling panorama;
[0124] The second set of graph data is parsed using the N-ary tree structure in the target standard library to obtain element information of each node object, wherein a batch job is regarded as a node object, and the element information of each node object represents data information defined by a batch job, and N is a positive integer greater than or equal to 2;
[0125] The element information of each node object is stored as a row of data to obtain a second array consisting of the element data of multiple node objects;
[0126] Based on the data information of the second array, the data is filled into the preset program statement structure in the form of byte stream to obtain the target execution statement, and the target execution statement is published to the target production environment.
[0127] Before converting the second batch scheduling panorama into a target execution statement, it is necessary to first set metadata variable information as shown in the following Table 3, and the tool program parses and generates an SQL data file according to the metadata.
[0128] Table 3
[0129]
[0130]
[0131] Then, take the second batch scheduling panorama in the test environment as the data source and convert it into the target execution statement in the database by performing the following steps:
[0132] S41, using the Python standard library xml.etree.ElementTree, parsing the structure of the second batch of panoramic images to obtain structured data;
[0133] Among them, the batch scheduling panorama itself is a structured file with a logical relationship of batch jobs. Therefore, the Python standard library can be directly used to parse the structured file.
[0134] like Figure 7 As shown in the figure, the ElementTree type is a flexible N-ary tree structure that can be used to store structured data in memory. The interaction between files and programs is completed on the ElementTree tree structure, and single elements and sub-elements are completed on Element.
[0135] S42, define the entire file object as "Tree", regard the "Tree" object as a tree (ElementTree), and a child object (Element) represents a leaf node;
[0136] Specifically, the element information of each node object is obtained in the following way:
[0137] Determine the file object corresponding to the second batch scheduling panorama as a target object tree, wherein the target object tree includes a plurality of node objects, a node object includes a plurality of child nodes, and a child node represents a worksheet in the file object;
[0138] By traversing and parsing the child nodes and pointer information of each node object in multiple node objects, the element information of each node object is obtained, wherein the pointer information represents the dependency relationship between each node object and other node objects.
[0139] In this embodiment, the Element.iter attribute can be used, but is not limited to, to traverse the sheet pages set by the variable GRAPH_SHEETS in a panorama file (eg, the second batch scheduling panorama file), ie, the child nodes of the object "Tree".
[0140] Among them, a group of sheet pages is defined as a "pages" object. The first-level object "pages" parsed by the program is a list of key-value pairs of "page" objects. A "page" represents all the data information of a sheet page.
[0141] By converting the N-ary tree structure into SQL data in the above manner and updating it to the test environment database, it can be ensured that the scheduling information in the test environment is completely consistent with the updated panorama, so that sufficient testing and verification can be performed before the official release, reducing the risk of failure in the production environment.
[0142] As an optional implementation method, the above method obtains element information of each node object by traversing and parsing the child nodes and pointer information of each node object in multiple node objects, including:
[0143] Obtain each node object from multiple node objects in turn as the current node object;
[0144] By traversing the current group of child nodes and current pointer information of the current node object, the attribute information of the current batch job corresponding to the current node object is obtained, wherein the attribute information includes the job name, execution time and execution mode;
[0145] The attribute information of a current batch job is stored as a key-value pair, and the element information found based on the key-value pair is determined as the element information of a current node object.
[0146] Specifically, this can be achieved by following the steps below:
[0147] S43, the Element.find attribute finds the object according to the name of the "page" object, and then uses the Element.iter attribute to traverse and parse all child nodes and pointer information in a "page", and store them as a list of key-value pairs of "node" and "line" objects;
[0148] The node object represents all the data information of a batch job, and the line object represents the dependency between two node objects, which is stored in the memory in the form of a pointer.
[0149] S44, parse the node object and traverse all element information in a node object;
[0150] The metadata list includes all the data information defined by the distributed batch job, such as the job name, group, execution time, execution mode, number of shards, execution batch, etc., and stores this data information in the format of key-value pairs. Figure 8 shown.
[0151] S45, storing the job definition information data in an array;
[0152] Among them, one node is stored as a row of data, and the result of traversing multiple nodes is a two-dimensional array. Fig. 9 The two-dimensional array shown.
[0153] S46, writing the data into the output target SQL file in sequence in the form of byte stream according to the data information of the two-dimensional array.
[0154] The SQL file does not need to parse the file structure. In the process of writing the data information of the two-dimensional array into the target SQL file in the form of byte stream data, the data information is actually filled into the preset SQL structure, thereby generating Fig.11 The standard SQL statement shown can be used to directly execute updates in the test environment database.
[0155] After the second batch scheduling panorama is converted into a target execution statement of the database according to the above steps S41 to S46, the target execution statement is published to the production environment database.
[0156] It should be noted that the Excel template file needs to parse the file structure. The Excel template file is also parsed using the Python standard library xml.etree.ElementTree to obtain a tree structure.
[0157] According to the data information of the two-dimensional array, update the value of each "node" object node, and update the data information of all nodes from the child node to the root node. The final output is a preset Excel template file structure, which generates an Excel version update file that complies with the distributed batch console import rules. It can be used for continuous delivery pipelines to import console update version information in the production environment. For details, please refer to Fig.10 The Excel template file shown.
[0158] Among them, in the above step S46, the target SQL file is sequentially written into the output in the form of byte stream data according to the data information of the two-dimensional array, and the process of parsing and updating the Excel template file to obtain the Excel version update file that complies with the distributed batch console import rules are two parallel processing steps. The program can be run once to generate an SQL file for testing data updates and an Excel template file for publishing a new version.
[0159] In this embodiment, the batch job information obtained from the visual batch scheduling panorama or the test environment database is converted into the Excel template file format required by the continuous delivery pipeline through the template generation module for scheduling and publishing in the production environment.
[0160] This method is suitable for application scenarios of batch job scheduling management in large distributed systems, especially in industries such as finance and e-commerce that need to process large amounts of data, and can significantly improve the efficiency and accuracy of job scheduling.
[0161] In the above process of converting the batch scheduling panorama into SQL data, the flexibility of the data table structure is adopted to allow processing and conversion of various types of batch job definition data and dependency data, rather than being limited to specific batch job definition tables and dependency tables. This flexible design enables the technical solution of this application to be applicable to different types of jobs and dependencies, especially in scenarios where job types and dependencies are complex and changeable, and can provide more comprehensive and accurate scheduling information processing capabilities.
[0162] As an optional example, the first batch scheduling panorama is adjusted to obtain the adjusted second batch scheduling panorama, including:
[0163] Based on target business requirements, the dependency relationship and job metadata of at least part of the batch jobs in the first batch scheduling panorama are adjusted to obtain a second batch scheduling panorama.
[0164] As business requirements in target applications are constantly changing, in order to improve the adjustment of batch scheduling, developers can directly obtain the data information of each batch job and the dependencies of each batch job in the batch scheduling panorama.
[0165] Then, the batch scheduling panorama is modified or adjusted so that the modified batch scheduling panorama can meet the requirements of the target business needs.
[0166] By adopting the modification method in the above manner, the developer's ability to understand the dependencies between various batch jobs is improved, and the operational process of version update is simplified.
[0167] As an optional implementation manner, after publishing the target execution statement to the target production environment, the method further includes:
[0168] Based on the target standard library, convert the target execution statement into a third set of graph data;
[0169] Based on the third set of image data, a third batch scheduling panorama is generated.
[0170] By publishing the target execution statement to the target production environment and using the tool program in the above embodiment to convert the target execution statement into a third batch scheduling panorama, the dependency relationship between each batch job after the updated version is intuitively displayed to the user, thereby improving the efficiency of the user in executing batch jobs in the target application.
[0171] In combination with the above embodiments, it can be seen that the technical solution of the present application provides a system that can assist in publishing distributed batch scheduling versions. It uses the standard library and graphics drawing technology of the Python language to implement a tool that can convert database data and batch job scheduling panoramas into each other, solving the problem of manual editing of Excel template files being prone to errors, low efficiency, and high version publishing costs.
[0172] Therefore, the technical solution of this application has at least the following beneficial effects:
[0173] (1) Developers only need to maintain a batch scheduling panorama to complete the database data update and output the Excel template file required for version release. The cost of version release is greatly reduced and the efficiency is greatly improved;
[0174] (2) The Excel template file for the continuous delivery pipeline version is automatically generated by the program, which is much more accurate than the original manual editing, avoiding many low-level errors that affect the version release progress;
[0175] (3) The real-time batch scheduling panorama can be viewed at any time in multiple R&D, testing, and production environments. Developers, testers, and operation and maintenance personnel can easily view the information they are concerned about.
[0176] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0177] Example 2
[0178] The present application also provides a batch scheduling processing device. It should be noted that the batch scheduling processing device of the present application can be used to execute the batch scheduling processing method provided by the present application. The batch scheduling processing device provided by the present application is introduced below.
[0179] Fig.12 is a schematic diagram of a batch scheduling processing device according to an embodiment of the present application. Fig.12 As shown, the device includes: a first acquisition unit 1202, used to acquire the original execution statement of the database, wherein the original database data includes the job information of the batch scheduling task; a first processing unit 1204, used to generate a first batch scheduling panorama based on the original execution statement, wherein the first batch scheduling panorama displays the initial dependency relationship between batch jobs in the target application in the form of a visual diagram; an adjustment unit 1206, used to adjust the first batch scheduling panorama to obtain an adjusted second batch scheduling panorama, wherein the second batch scheduling panorama displays the updated dependency relationship between batch jobs in a visual form; a conversion unit 1208, used to convert the second batch scheduling panorama into a target execution statement of the database, and publish the target execution statement to the target production environment.
[0180] Optionally, in the batch scheduling processing device provided in the embodiment of the present application, the first processing unit 1204 includes: a first query module, used to use the target cursor object to query the batch job definition information of the target application from the original execution statement; a first processing module, used to use the target cursor object to traverse a set of data tables in the test environment database to obtain column data in the query data set; a second processing module, used to generate a first array consisting of row data and column data with each batch job definition information as row data; a second query module, used to query the first array according to the job grouping information to obtain a set of batch job data information with initial dependency relationships; a third processing module, used to generate a first batch scheduling panorama based on a set of batch job information with initial dependency relationships.
[0181] Optionally, in the batch scheduling processing device provided in the embodiment of the present application, the first query module includes: a first creation sub-module, used to use the test environment database connection information to create a connection between the target standard library and the database; a second creation sub-module, used to create a target cursor object, and obtain the batch job definition information of the target application by cyclically using the target cursor object.
[0182] Optionally, in the batch scheduling processing device provided in the embodiment of the present application, the third processing module includes: a first processing sub-module, used to determine each batch job as a node object; a second processing sub-module, used to determine each dependency in the initial dependency as a connection line; a third processing sub-module, used to generate a first set of graph data based on the node objects and the connections, wherein pointers are used as pointing information between two node objects; and a fourth processing sub-module, used to draw a first batch scheduling panoramic map based on the first set of graph data.
[0183] Optionally, in the batch scheduling processing device provided in the embodiment of the present application, the above-mentioned conversion unit 1208 includes: an extraction module, which is used to extract a second set of graph data from the second batch scheduling panorama; a first parsing module, which is used to use the N-ary tree structure in the target standard library to parse the second set of graph data to obtain element information of each node object, wherein a batch job is regarded as a node object, and the element information of each node object represents the data information defined by a batch job, and N is a positive integer greater than or equal to 2; a fourth processing module, which is used to store the element information of each node object as a row of data to obtain a second array composed of element data of multiple node objects; a fifth processing module, which is used to fill in the preset program statement structure in the form of byte stream data based on the data information of the second array, obtain the target execution statement, and publish the target execution statement to the target production environment.
[0184] Optionally, in the batch scheduling processing device provided in the embodiment of the present application, the above-mentioned first parsing module includes: a fourth processing sub-module, used to determine the file object corresponding to the second batch scheduling panorama as a target object tree, wherein the target object tree includes multiple node objects, one node object includes multiple child nodes, and one child node represents a worksheet in the file object; a fifth processing sub-module, used to obtain element information of each node object by traversing and parsing the child nodes and pointer information of each node object in the multiple node objects, wherein the pointer information represents the dependency relationship between each node object and other node objects.
[0185] Optionally, in the batch scheduling processing device provided in the embodiment of the present application, the above-mentioned first parsing module includes: a first acquisition sub-module, used to sequentially acquire each node object from multiple node objects as a current node object; a fifth processing sub-module, used to obtain the attribute information of the current batch job corresponding to the current node object by traversing the current set of child nodes and current pointer information of the current node object, wherein the attribute information includes the job name, execution time and execution mode; a sixth processing sub-module, used to store the attribute information of the current batch job as a key-value pair, and determine the element information found based on the key-value pair as the element information of the current node object.
[0186] Optionally, in the batch scheduling processing device provided in an embodiment of the present application, the above-mentioned adjustment unit 1206 includes: an adjustment module, used to adjust the dependencies and job metadata of at least part of the batch jobs in the first batch scheduling panorama based on target business needs to obtain a second batch scheduling panorama.
[0187] Optionally, the batch scheduling processing device provided in the embodiment of the present application also includes: a conversion unit, which is used to convert the target execution statement into a third set of graph data based on the target standard library after publishing the target execution statement to the target production environment; and a second processing unit, which is used to generate a third batch scheduling panorama based on the third set of graph data.
[0188] By using the batch scheduling processing device provided by the embodiment of the present application to utilize a tool program, the original execution statement of the database is converted into a first batch scheduling panorama, and then the first batch scheduling panorama is adjusted according to business needs, and the adjusted second batch scheduling panorama is converted into the target execution statement of the database, thereby publishing the target execution statement to the target production environment. In other words, the technical solution of the present application reduces the workload of manually directly modifying the original execution statement by more conveniently modifying and adjusting the dependency relationship and job definition information between each batch job in the batch scheduling panorama, improves the processing efficiency of batch scheduling, and solves the technical problem of low processing efficiency of batch scheduling in related technologies.
[0189] Example 3
[0190] An embodiment of the present application may provide an electronic device, Fig.13 is a structural block diagram of an electronic device according to an embodiment of the present application. Fig.13 As shown, the electronic device may include: one or more ( Fig.13 (only one is shown) processor 1302, memory 1304, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.
[0191] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0192] The processor may call the information and application program stored in the memory through the transmission device to perform the following steps:
[0193] Obtaining original execution statements of a database, wherein the original database data includes job information of a batch scheduling task;
[0194] Based on the original execution statement, a first batch scheduling panorama is generated, wherein the first batch scheduling panorama displays the initial dependency relationship between batch jobs in the target application in the form of a visualization diagram;
[0195] By adjusting the first batch scheduling panorama, an adjusted second batch scheduling panorama is obtained, wherein the second batch scheduling panorama displays the updated dependency relationship between the batch jobs in a visual form;
[0196] The second batch scheduling panorama is converted into a target execution statement of the database, and the target execution statement is published to the target production environment.
[0197] It can be understood by those skilled in the art that Fig.13 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Fig.13 The structure of the electronic device is not limited. Fig.13 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Fig.13 Different configurations shown.
[0198] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0199] Example 4
[0200] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the batch scheduling processing method provided in the first embodiment.
[0201] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.
[0202] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the steps of the batch scheduling processing method.
[0203] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0204] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
[0205] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0206] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0207] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0208] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.
[0209] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A batch scheduling processing method, characterized in that: include: Acquire an original execution statement from a database, wherein the original execution statement includes job information of a batch scheduling task; Based on the original execution statement, a first batch scheduling panorama is generated, wherein the first batch scheduling panorama displays the initial dependency relationship between batch jobs in the target application in the form of a visual diagram; By adjusting the first batch scheduling panorama, an adjusted second batch scheduling panorama is obtained, wherein the second batch scheduling panorama displays the updated dependency relationship between the batch jobs in a visual form; The second batch scheduling panorama is converted into a target execution statement of the database, and the target execution statement is published to a target production environment.
2. The method according to claim 1, characterized in that The step of generating a first batch scheduling panorama based on the original execution statement includes: Using the target cursor object, querying the batch job definition information of the target application from the original execution statement; Using the target cursor object, traverse a set of data tables in the test environment database to obtain column data in the query data set; Taking each batch job definition information as row data, generating a first array consisting of the row data and the column data; According to the job grouping information, query the first array to obtain a group of batch job data information with an initial dependency relationship; Based on the set of batch job information with initial dependency, the first batch scheduling panorama is generated.
3. The method according to claim 2, characterized in that The using the target cursor object to query the batch job definition information of the target application from the original execution statement includes: Using the test environment database connection information, create a connection between the target standard library and the database; The target cursor object is created, and batch job definition information of the target application is obtained by cyclically using the target cursor object.
4. The method according to claim 2, characterized in that: The step of generating the first batch scheduling panorama based on the set of batch job information having the initial dependency relationship includes: Identify each batch job as a node object; Determine each dependency in the initial dependency as a line; Based on the node objects and the lines, a first set of graph data is generated, wherein a pointer is used as pointing information between two node objects; Based on the first set of image data, the first batch scheduling panoramic image is drawn.
5. The method according to claim 1, characterized in that The converting the second batch scheduling panorama into a target execution statement of the database, and publishing the target execution statement to the target production environment, includes: Extracting a second set of image data from the second batch of scheduled panoramic images; The second set of graph data is parsed using an N-ary tree structure in a target standard library to obtain element information of each node object, wherein a batch job is regarded as a node object, the element information of each node object represents data information defined by a batch job, and N is a positive integer greater than or equal to 2; The element information of each node object is stored as a row of data to obtain a second array consisting of the element data of multiple node objects; Based on the data information of the second array, the data is filled into a preset program statement structure in the form of a byte stream to obtain the target execution statement, and the target execution statement is published to a target production environment.
6. The method according to claim 5, characterized in that The N-ary tree structure in the target standard library is used to parse the second batch scheduling panorama to obtain element information of each node object, including: Determine the file object corresponding to the second batch scheduling panorama as a target object tree, wherein the target object tree includes a plurality of node objects, each of the node objects includes a plurality of child nodes, and each child node represents a worksheet in the file object; The element information of each node object is obtained by traversing and parsing the child nodes and pointer information of each node object in the multiple node objects, wherein the pointer information represents the dependency relationship between each node object and other node objects.
7. The method according to claim 6, characterized in that The step of traversing and parsing the child nodes and pointer information of each node object in the plurality of node objects to obtain the element information of each node object includes: Obtain each node object in sequence from the multiple node objects as a current node object; By traversing a current group of child nodes and current pointer information of the current node object, the attribute information of the current batch job corresponding to the current node object is obtained, wherein the attribute information includes the job name, execution time and execution mode; The attribute information of the current batch job is stored as a key-value pair, and the element information found based on the key-value pair is determined as the element information of the current node object.
8. The method according to any one of claims 1 to 7, characterized in that The method of adjusting the first batch scheduling panorama to obtain an adjusted second batch scheduling panorama includes: Based on target business requirements, the dependency relationship and job metadata of at least part of the batch jobs in the first batch scheduling panorama are adjusted to obtain the second batch scheduling panorama.
9. The method according to any one of claims 1 to 7, characterized in that After publishing the target execution statement to the target production environment, the method further includes: Based on the target standard library, converting the target execution statement into a third set of graph data; Based on the third set of image data, a third batch scheduling panorama image is generated.
10. A batch scheduling processing device, characterized in that: include: A first acquisition unit, configured to acquire an original execution statement from a database, wherein the original execution statement includes job information of a batch scheduling task; A first processing unit is configured to generate a first batch scheduling panorama based on the original execution statement, wherein the first batch scheduling panorama displays the initial dependency relationship between batch jobs in the target application in the form of a visual diagram; An adjusting unit, configured to obtain an adjusted second batch scheduling panorama by adjusting the first batch scheduling panorama, wherein the second batch scheduling panorama displays the updated dependency relationship between the batch jobs in a visual form; A conversion unit is used to convert the second batch scheduling panorama into a target execution statement of the database, and publish the target execution statement to a target production environment.
11. A processor, characterized in that: The processor is used to run a program, wherein the program executes the method according to any one of claims 1 to 10 when running.
12. An electronic device, characterized in that: The invention comprises one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any one of claims 1 to 10.