Bidirectional conversion method based on Control-M-XML

Abstract

The invention discloses a bidirectional conversion method based on Control-M-XML (Extensible Markup Language). The bidirectional conversion method comprises the following steps: step S1: reading and preprocessing an XML file; s2, recursive traversal and data extraction are carried out; s3, data structuring and conversion are carried out; s4, generating a customized Excel template; and S5, reversely generating XML file analysis. The operation flow logic is visually displayed through a multi-dimensional Excel template, configuration modification through the Excel template is supported, and a compliant Control-M XML file is automatically generated.

Description

Technical Field

[0001] The present invention relates to the field of job scheduling management, and particularly to a bidirectional conversion method based on Control-M-XML. Background Art

[0002] Current technical solutions:

[0003] The Control-M client provides basic XML editing, job monitoring, and simple dependency display. However, the client focuses on supporting basic editing configurations and monitoring, and cannot batch modify parameters (for example, resource queue migration requires line-by-line operation). It does not perform customized parsing on the semantics of Control-M XML (for example, it cannot recognize the hierarchical logic of `SMART_FOLDER`), cannot convert XML into editable tabular data (such as Excel), and lacks in-depth parsing and visualization capabilities for the specific structure of Control-M-XML.

[0004] The technical solutions in the prior art include:

[0005] Lack of intuitiveness: Users understand the job flow by viewing XML tags at the code level, and cannot quickly obtain the overall scheduling logic. There is a lack of an intuitive visual interface to display the logical structure of the Control-M-XML file.

[0006] High collaboration threshold: Due to the complexity of XML syntax and the limitations of tool specialization, non-technical personnel are difficult to directly participate in configuration adjustments, resulting in low cross-team collaboration efficiency.

[0007] Low maintenance efficiency: In scenarios such as migration or version iteration, it is necessary to frequently compare historical configurations and batch modify parameters (such as resource queue adjustment, environment variable migration). The traditional line-by-line editing mode is time-consuming and error-prone (such as missing dependency conditions). Summary of the Invention

[0008] In view of the above problems, the present invention is proposed to provide a bidirectional conversion method based on Control-M-XML that overcomes the above problems or at least partially solves the above problems.

[0009] According to one aspect of the present invention, a bidirectional conversion method based on Control-M-XML is provided. The bidirectional conversion method includes:

[0010] Step S1: XML file reading and preprocessing;

[0011] Step S2: Recursive traversal and data extraction;

[0012] Step S3: Data structuring and conversion;

[0013] Step S4: Generate customized Excel template;

[0014] Step S5: Parse the XML file generated in reverse.

[0015] Optionally, the specific steps of Step S1: Read and preprocess the XML file include:

[0016] Tool selection: Use the xml.etree.ElementTree library in Python to parse the XML file and combine with the lxml library to process complex structures;

[0017] Naming processing: Uniformly process the namespace through regular expressions;

[0018] File loading: Load the XML file to the server path and generate a root node object.

[0019] Optionally, the specific steps of Step S2: Recursively traverse and extract data include:

[0020] Core logic: Recursively traverse XML nodes, record the hierarchical path, recursively process child nodes, and extract key information.

[0021] Optionally, the specific key information includes:

[0022] Job hierarchical structure: The hierarchical relationship between SMART_FOLDER and FOLDER;

[0023] Job definition: Job name, type, execution command, resource requirement parameter attributes;

[0024] Dependency relationship: Predecessor job, trigger condition, and dependency type;

[0025] Scheduling strategy: Execution frequency, time window, and retry strategy.

[0026] Optionally, the specific steps of Step S3: Structurize and transform data include:

[0027] Data storage: Convert the extracted temporary data into a Pandas DataFrame, classify it by module, and the temporary data includes hierarchical structure, job definition, and dependency relationship:

[0028] Field mapping: Ensure that the XML attributes correspond one-to-one with the DataFrame column names.

[0029] Optionally, the specific steps of Step S4: Generate customized Excel template include:

[0030] Tool selection: Use the openpyxl or xlsxwriter library, combined with the ExcelWriter of Pandas to generate a multi-Sheet Excel file;

[0031] Template design:

[0032] Sheet1: SMART_FOLDER: Displays the folder hierarchy path;

[0033] Sheet2: JOB: Lists the job name, type, command, resource queue, and folder path;

[0034] Sheet3: DEPENDENCY: Displays the dependent starting jobs, end jobs, and trigger conditions.

[0035] Optionally, the step S5: reverse generation of XML file parsing specifically includes:

[0036] Data verification: ensure the integrity and logical correctness of the data in the customized EXCEL template;

[0037] Data loading and structure mapping.

[0038] Optionally, the data verification: ensuring the integrity and logical correctness of the data in the customized EXCEL template specifically includes:

[0039] Field check: Check whether there are null values ​​in the JobName, Type, and Command fields of the JOB table in Excel. If key fields are missing, an exception is thrown and the user is prompted to add them.

[0040] Dependency closure verification: Check whether FromJob and ToJob in the DEPENDENCY table exist in the JOB table. If there is an undefined job, record an error log and terminate the generation process;

[0041] Hierarchical path validity: Checks whether the path in the SMART_FOLDER table complies with the format.

[0042] Optionally, the data loading and structure mapping specifically include:

[0043] Read Excel data;

[0044] Field mapping rules, mapping Excel column names to XML attribute names

[0045] XML structure reconstruction, rebuilds the nested structure of Control-M XML according to the hierarchical path and dependencies.

[0046] XML file generation and formatting, write the XML tree structure into the file, and format the output.

[0047] Reverse generation verification to ensure that the generated XML file can be parsed correctly by Control-M.

[0048] A two-way conversion method based on Control-M-XML provided by the present invention, the two-way conversion method comprising: step S1: XML file reading and preprocessing; step S2: recursive traversal and data extraction; step S3: data structuring and conversion; step S4: customized Excel template generation; step S5: reverse generation of XML file parsing. The workflow logic is intuitively displayed through a multi-dimensional Excel template, and it is supported to modify the configuration through the Excel template and automatically generate a compliant Control-M XML file.

[0049] The above description is only an overview of the technical solution of the present invention. In order to be able to understand the technical means of the present invention more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features and advantages of the present invention more obvious and understandable, the following specifically illustrates the specific embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0051] Figure 1 It is a flowchart of a two-way conversion method based on Control-M-XML provided by an embodiment of the present invention;

[0052] Figure 2 It is a flowchart of the detailed implementation of the customized template provided by an embodiment of the present invention;

[0053] Figure 3 It is a schematic diagram of Sheet1 provided by an embodiment of the present invention;

[0054] Figure 4 It is a schematic diagram of Sheet2 provided by an embodiment of the present invention;

[0055] Figure 5 It is a schematic diagram of Sheet3 provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0056] The following will describe the exemplary embodiments of the present disclosure in more detail with reference to the drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be fully conveyed to those skilled in the art.

[0057] In the embodiments of the specification, claims, and drawings of the present invention, the terms "comprising", "having", and any variations thereof are intended to cover non-exclusive inclusion. For example, a series of steps or units are included.

[0058] The following will further describe the technical solutions of the present invention in detail with reference to the drawings and embodiments.

[0059] As Figure 1 and Figure 2 shown, the present invention proposes a bidirectional conversion method based on Control-M-XML. The structure of a Control-M XML file usually contains multiple levels of nested elements (such as SMART_FOLDER, FOLDER, JOB, DEPENDENCY, etc.). The parsing algorithm needs to extract key information and store it in a structured manner.

[0060] Specific implementation steps:

[0061] I. Steps of the XML file parsing algorithm (XML-EXCEL):

[0062] Step 1: XML file reading and preprocessing:

[0063] Tool selection: Use the xml.etree.ElementTree library in Python to parse XML files, and combine with the lxml library to process complex structures.

[0064] Naming processing: Uniformly process namespaces (such as CTM) through regular expressions.

[0065] File loading: Load the XML file to the server path to generate a root node object.

[0066] Step 2: Recursive traversal and data extraction

[0067] Core logic: Recursively traverse XML nodes, record the hierarchical path, recursively process child nodes, and extract the following key information:

[0068] Job hierarchical structure: The hierarchical relationship between SMART_FOLDER and FOLDER.

[0069] Job definition (JOB): Parameter attributes such as job name, type, execution command, and resource requirements.

[0070] Dependency relationship (DEPENDENCY): Predecessor job, trigger condition, and dependency type.

[0071] Scheduling strategy: Execution frequency (such as daily, monthly), time window, and retry strategy.

[0072] Step 3: Data structuring and conversion

[0073] Data storage: Convert the extracted temporary data (hierarchy, job definition, dependencies) into a Pandas DataFrame, categorized by module:

[0074] Field mapping: Make sure that the XML attributes correspond to the DataFrame column names, for example:

[0075] XML attribute JOBNAME → DataFrame column name JobName

[0076] XML attribute QUEUE → DataFrame column name ResourceQueue

[0077] Step 4: Generate customized Excel template

[0078] Tool selection: Use openpyxl or xlsxwriter library, combined with Pandas' ExcelWriter to generate multi-sheet Excel files.

[0079] like Figures 3 - 5 As shown, the template design:

[0080] Sheet1: SMART_FOLDER: Displays the folder hierarchy path (such as / Prod / ETL).

[0081] Sheet2: JOB: Lists the job name, type, command, resource queue, and folder path.

[0082] Sheet3: DEPENDENCY: Displays the dependent starting jobs, end jobs, and trigger conditions.

[0083] 2. Reverse generation of XML file parsing algorithm steps (EXCEL-XML):

[0084] Step 1: Data verification: Ensure the integrity and logical correctness of the data in the customized EXCEL template

[0085] Field check: Check whether there are empty values ​​in the JobName, Type, Command and other fields in the JOB table in Excel. If key fields are missing, an exception is thrown and the user is prompted to supplement them.

[0086] Dependency closure verification: Check whether FromJob and ToJob in the DEPENDENCY table exist in the JOB table. If there are undefined jobs, record an error log and terminate the generation process.

[0087] Hierarchical path legality: Verify whether the paths in the SMART_FOLDER table conform to the format (such as / Prod / ETL) to avoid illegal characters or duplicate paths.

[0088] Step 2: Data loading and structure mapping:

[0089] Read Excel data: Use Pandas to read the data of each sheet (SMART_FOLDER, JOB, DEPENDENCY).

[0090] Field mapping rules: Map the Excel column names to XML attribute names. For example:

[0091] JobName → JOBNAME

[0092] ResourceQueue → QUEUE

[0093] Condition → CONDITION

[0094] XML structure reconstruction: Reconstruct the nested structure of the Control-M XML according to the hierarchical path and dependencies.

[0095] Create the root node: Use xml.etree.ElementTree to create the root node <controlm>, and add a namespace.

[0096] Recursively construct the folder hierarchy: Create SMART_FOLDER or FOLDER nodes layer by layer according to the path in the SMART_FOLDER table (such as / Prod / ETL).

[0097] Insert job definitions: Add the jobs in the JOB table under the corresponding folder nodes.

[0098] Add dependency relationships: Insert DEPENDENCY child nodes within the job nodes to define FROM_JOB and TO_JOB.

[0099] XML file generation and formatting: Write the XML tree structure to a file and format the output.

[0100] Generate an XML string: Convert the node tree to a string using ET.ElementTree.

[0101] Beautify the format: Add indentation and line breaks through minidom to improve readability.

[0102] Write to a file: Save as a.xml file, ensuring the encoding is UTF-8.

[0103] Reverse generation verification: Ensure that the generated XML file can be correctly parsed by Control-M.

[0104] Syntax verification: Use XML Schema (XSD) to verify whether the file format conforms to the Control-M specification.

[0105] Function verification: Import the generated XML into the Control-M client and test whether the job scheduling and dependencies are executed as expected.

[0106] Recursive traversal and performance optimization:

[0107] For large-scale XML files, use an iterative algorithm to replace traditional recursion.

[0108] Use incremental parsing (SAX mode) of lxml to reduce memory usage.

[0109] Handling of complex dependency relationships:

[0110] Dependency relationships may cross folders, and the complete path of the job needs to be associated through the ParentPath field.

[0111] Data integrity verification:

[0112] Both forward parsing and reverse generation have built-in data integrity checks. During the parsing phase, required fields (such as JOBNAME, fields not being empty, and dependency closure) are checked, and exceptions are thrown or logs are recorded when they are missing.

[0113] Maintenance Verification and Reverse Generation

[0114] User Modification Verification:

[0115] Use Pandas to read the modified Excel file and verify the field formats (such as time format and dependency closure).

[0116] Reverse Generate XML: According to the modified DataFrame, regenerate the node tree in the Control-M XML format and write it to the XML file.

[0117] Beneficial Effects:

[0118] 1. Intuitive Display: Provide an intuitive query interface that can clearly display the logical structure of the Control-M-XML file, as well as the job and dependency configuration processes, improving the user's ability to view Control-M historical tasks and conveniently view the visual scheduling historical version records.

[0119] 2. Easy to Use: Simplify the operation process so that non-professional technical personnel can easily understand and modify the job and dependency configurations in the XML file.

[0120] 3. Comprehensive Parsing: Deeply parse the hidden attributes in XML (such as ` <resource>`Queue priority of nodes) to ensure data integrity.

[0121] 4. Bidirectional convertibility: Implement the bidirectional conversion of "XML → Excel → XML", support version comparison, and form a bidirectional conversion closed loop.

[0122] The above specific implementation manners further elaborate on the purpose, technical solution and beneficial effects of the present invention. It should be understood that the above are only specific implementation manners of the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.< / resource> < / controlm>

Claims

1. A bidirectional conversion method based on Control-M-XML, characterized in that: The bidirectional conversion method comprises: Step S1: XML file reading and preprocessing; Step S2: recursive traversal and data extraction; Step S3: data structuring and conversion; Step S4: Generate customized Excel template; Step S5: Reverse generate XML file parsing.

2. A bidirectional conversion method based on Control-M-XML according to claim 1, characterized in that: The step S1: XML file reading and preprocessing specifically includes: Tool selection: Use Python's xml.etree.ElementTree library to parse XML files, combined with the lxml library to handle complex structures; Naming processing: uniformly process namespaces through regular expressions; File loading: Load the XML file to the server path and generate a root node object.

3. A bidirectional conversion method based on Control-M-XML according to claim 1, characterized in that: The step S2: recursive traversal and data extraction specifically includes: Core logic: recursively traverse XML nodes, record hierarchical paths, recursively process child nodes, and extract key information.

4. A bidirectional conversion method based on Control-M-XML according to claim 3, characterized in that: The key information specifically includes: Job hierarchical structure: the hierarchical relationship between SMART_FOLDER and FOLDER; Job definition: job name, type, execution command, resource requirement parameter attributes; Dependencies: Prerequisites, trigger conditions, and dependency types; Scheduling strategy: execution frequency, time window, and retry strategy.

5. The bidirectional conversion method based on Control-M-XML according to claim 1, characterized in that: The step S3: data structuring and conversion specifically includes: Data storage: Convert the extracted temporary data into Pandas DataFrame and classify them by module. The temporary data includes hierarchical structure, job definition, and dependency relationships: Field mapping: Ensure that XML attributes correspond to DataFrame column names.

6. A bidirectional conversion method based on Control-M-XML according to claim 1, characterized in that: The step S4: generating a customized Excel template specifically includes: Tool selection: Use openpyxl or xlsxwriter library, combined with Pandas' ExcelWriter to generate multi-sheet Excel files; Template design: Sheet1: SMART_FOLDER: Displays the folder hierarchy path; Sheet2: JOB: Lists the job name, type, command, resource queue, and folder path; Sheet3: DEPENDENCY: Displays the dependent starting jobs, end jobs, and trigger conditions.

7. The bidirectional conversion method based on Control-M-XML according to claim 1, characterized in that: The step S5: reverse generation of XML file parsing specifically includes: Data verification: ensure the integrity and logical correctness of the data in the customized EXCEL template; Data loading and structure mapping.

8. A bidirectional conversion method based on Control-M-XML according to claim 7, characterized in that: The data verification: ensuring the integrity and logical correctness of the data in the customized EXCEL template specifically includes: Field check: Check whether there are null values ​​in the JobName, Type, and Command fields of the JOB table in Excel. If key fields are missing, an exception is thrown and the user is prompted to add them. Dependency closure verification: Check whether FromJob and ToJob in the DEPENDENCY table exist in the JOB table. If there is an undefined job, record an error log and terminate the generation process; Hierarchical path validity: Checks whether the path in the SMART_FOLDER table complies with the format.

9. The bidirectional conversion method based on Control-M-XML according to claim 7, characterized in that: The data loading and structure mapping specifically include: Read Excel data; Field mapping rules, mapping Excel column names to XML attribute names XML structure reconstruction, rebuilds the nested structure of Control-M XML according to the hierarchical path and dependencies. XML file generation and formatting, write the XML tree structure into the file, and format the output. Reverse generation verification to ensure that the generated XML file can be parsed correctly by Control-M.

Images