Configurable and visualized data processing system and method based on YAML file

Abstract

The invention relates to the technical field of data processing, and provides a configurable and visualized data processing system based on a YAML file, and the data processing system comprises a YAML configuration analysis module which is used for analyzing a YAML configuration file; the data processing engine is used for sequentially executing data processing operation according to a configuration rule and generating an intermediate processing result; the visual interface module is used for displaying an intermediate processing result of the data processing engine in real time; the configuration management module is used for storing, loading and exporting the YAML configuration file; the visual interface module, the configuration management module and the data processing engine cooperate to form a dynamic synchronization mechanism. According to the method, the data processing flow is defined through the YAML configuration file, so that the user can intuitively understand the data processing process, the visual interface reflects the data processing effect in real time, and the user is helped to efficiently complete the data processing task.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing system and method that can be configured and visualized based on a YAML file. Background Art

[0002] With the development of big data and artificial intelligence, data processing has become a critical component across all industries. Many data processing processes rely on code and complex configurations, which are often difficult for non-technical personnel to understand and use. Existing code-based data processing processes often suffer from poor flexibility, scalability, and interoperability. Users often rely on programmers to write and maintain code, making it difficult to quickly adjust data processing logic to meet actual needs.

[0003] Although some data processing tools provide graphical interfaces to assist users, these tools lack sufficient flexibility and customization capabilities, and usually do not support users to quickly define and modify data processing rules through concise configuration files. Summary of the Invention

[0004] In order to help solve the above technical problems, the present application provides a data processing system and method that can be configured and visualized based on YAML files.

[0005] In a first aspect, the present application provides a data processing system that can be configured and visualized based on a YAML file, wherein the data processing system includes: YAML configuration parsing module, used to parse the input configuration, output configuration and data processing configuration in the YAML configuration file; The data processing engine performs data loading, cleaning, conversion, filtering and sorting operations in sequence according to the configuration rules generated by the YAML configuration parsing module, and generates intermediate processing results; A visualization interface module displays the intermediate processing results of the data processing engine in real time, provides interactive controls to dynamically modify YAML configuration parameters, and synchronizes the modified configuration parameters to the YAML configuration parsing module; Configuration management module, used to save, load and export the YAML configuration file; The visualization interface module, configuration management module and data processing engine work together to form a dynamic synchronization mechanism. The visualization interface module converts user interaction operations into YAML configuration change instructions. The configuration management module receives the change instructions and updates the configuration objects in the memory. The data processing engine re-executes the data processing flow according to the updated configuration objects.

[0006] Preferably, the visualization interface module includes a data preview component, a processing step display component and a graphical configuration editing component. The data preview component is used to display the processing results of each step of the data processing configuration in real time. The processing step display component is used to display each step in the YAML configuration file in the form of a visual process node. The graphical configuration editing component is used to enable the user to re-establish the mapping relationship between fields in the field mapping diagram by dragging and dropping, or to modify the logical expression through the input box in the filter condition editor.

[0007] Preferably, the configuration management module includes a configuration parsing submodule, a configuration image maintenance submodule, a configuration verification submodule, and a configuration saving and exporting submodule. The configuration parsing submodule is used to parse the YAML text file into an internally structured configuration object for reading and rendering by the visualization interface module. The configuration image maintenance submodule is used to synchronize the graphical interface interaction with the YAML file.

[0008] Preferably, the data processing configuration includes step types, fields to be processed, filtering conditions, and field mapping rules, and the step types include at least: removing null value rows step type, field conversion step type, condition filtering step type, and sorting step type.

[0009] In a second aspect, the present application provides a data processing method, which adopts the following technical solution: A data processing method of a data processing system configurable and visualized based on a YAML file as described in any one of the first aspects above, wherein the data processing method comprises: S1: Reading and parsing the YAML configuration file of the YAML configuration parsing module through the data processing engine; S2: Processing data by the data processing engine in a sequential order based on the YAML configuration file of the YAML configuration parsing module; S3: Displaying the processing results of each step in step S2 through the visual interface module, wherein the visual interface module converts the user interaction operation into a YAML configuration change instruction; S4: The configuration management module receives the change instruction and updates the configuration object in the memory. Based on the configuration management module modifying the rules in the YAML configuration file through the visual interface, the data processing engine re-executes the data processing flow according to the updated YAML configuration file.

[0010] Preferably, the S1 includes: S11: Read the YAML configuration file and parse the input data, output data targets, and all processing steps; S12: Load input data and convert the input data into a data table or data frame according to the input configuration.

[0011] Preferably, the S2 includes: Step S21: Based on the step type of removing empty value rows, checking whether the field to be processed is empty, and removing the corresponding empty value rows; Step S22: Based on the field conversion step type, the field to be processed is processed according to the field mapping rule; Step S23: Based on the conditional filtering step type, data meeting the condition is screened out according to the filtering condition; Step S24: sorting the data based on the sorting step type; Step S25: Save the processed data in a specified format according to the output configuration.

[0012] Preferably, the S4 includes: After the YAML configuration file is modified, the display result is updated in real time through the visual interface module interface.

[0013] In summary, the method of this application defines the data processing flow through a YAML configuration file and displays each step of the data processing in a visual interface, enabling users to intuitively understand and adjust the data processing process. Users edit the YAML configuration file to define data processing rules, field mappings, filter conditions, conversion steps, and so on. The visual interface simultaneously reflects the data processing results in real time, helping users to more efficiently complete tasks such as data cleaning, conversion, and analysis. This method offers advantages such as flexible configuration, ease of understanding and operation, and convenient user interactive adjustments. It is suitable for use in data processing platforms, data analysis tools, and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a flowchart of an embodiment of a data processing method that can be configured and visualized based on a YAML file of the present application. DETAILED DESCRIPTION

[0015] The present invention will be further described below with reference to the accompanying drawings. The structure and principle of the present invention will be very clear to those skilled in the art. It should be understood that the specific embodiments described herein are only intended to explain the present invention and are not intended to limit the present invention.

[0016] The present application provides a configurable and visual data processing system based on YAML files. The data processing system includes: a YAML configuration parsing module, which is used to parse the input configuration, output configuration and data processing configuration in the YAML configuration file; a data processing engine, which performs data loading, cleaning, conversion, filtering and sorting operations in sequence according to the configuration rules generated by the YAML configuration parsing module, and generates intermediate processing results; a visualization interface module, which displays the intermediate processing results of the data processing engine in real time, provides interactive controls for dynamically modifying YAML configuration parameters, and synchronizes the modified configuration parameters to the YAML configuration parsing module; a configuration management module, which is used to save, load and export YAML configuration files; the visualization interface module, the configuration management module and the data processing engine cooperate to form a dynamic synchronization mechanism, the visualization interface module converts user interaction operations into YAML configuration change instructions, the configuration management module receives the change instructions and updates the configuration objects in the memory, and the data processing engine re-executes the data processing flow according to the updated configuration objects.

[0017] Data processing configuration includes step type, fields to be processed, filtering conditions, and field mapping rules. Step types include at least: removing null value rows step type, field conversion step type, conditional filtering step type, and sorting step type.

[0018] This application also provides a data processing method for the above-mentioned data processing system that can be configured and visualized based on YAML files, Figure 1 This is a flowchart of an embodiment of a data processing method that can be configured and visualized based on a YAML file of the present application.

[0019] The data processing method includes: S1: Reads and parses the YAML configuration file through the data processing engine. S1 includes: S11: Read the YAML configuration file and parse the input data, output data targets, and all processing steps.

[0020] S12: Load input data and convert the input data into a data table or data frame according to the input configuration.

[0021] The YAML configuration file is the core part of this technical solution. All data processing rules, input and output files, step sequence, conditional filtering, etc. are configured through the YAML file.

[0022] The general structure of a YAML configuration file is as follows: input: path: data / input.csv format: csv output: path: data / output.json format: json steps: - type: drop_empty fields: [name, email] - type: transform mappings: name: strip email: lowercase - type: filter condition: "age>= 18" - type: sort field: "age" order: ascending input: Configuration related to input data, including the path and format of the data file (such as CSV, JSON).

[0023] output: The path and format of the output data. Multiple output formats are supported, such as CSV, JSON, Excel, etc.

[0024] steps: The specific steps of data processing, including the processing type (such as cleaning, conversion, filtering, sorting, etc.) and the specific configuration of each step.

[0025] type: step type, such as drop_empty (remove empty rows), transform (field transformation), filter (conditional filtering), sort (sorting), etc.

[0026] fields: The fields to be processed, such as name, email.

[0027] condition: Filter condition, such as age>= 18.

[0028] mappings: field mapping rules, such as name: strip means removing leading and trailing spaces of the name field.

[0029] Pre-writing the logic involved in data processing into the configuration file can improve the efficiency of subsequent data processing.

[0030] The data processing engine is the core component that executes the data processing flow according to the YAML configuration file. It processes the data step by step according to the configuration steps and produces the final output. The engine's work includes two parts: data reading and parsing, and data processing.

[0031] Data reading and analysis: Step S11: Read the YAML configuration file and parse out the input data source, output target, and all processing steps.

[0032] Step S12: Based on the input configuration, load the data source (such as CSV, JSON file) and convert it into a data table or data frame (DataFrame) format for subsequent processing.

[0033] S2: Processes data sequentially through the data processing engine based on the steps in the YAML configuration file. S2 includes: Step S21: Based on the step type of removing empty value rows, check whether the field to be processed is empty, and remove the corresponding empty value rows.

[0034] Step S22: Based on the field conversion step type, the fields to be processed are processed according to the field mapping rules.

[0035] Step S23: Based on the conditional filtering step type, data that meets the conditions is screened out according to the filtering conditions.

[0036] Step S24: Sort the data based on the sorting step type.

[0037] Step S25: Save the processed data in a specified format according to the output configuration.

[0038] Specifically, the data processing process is executed according to the steps in the YAML configuration file. When each step is executed, it processes the data table and updates the data.

[0039] Step S21: Remove empty value rows (drop_empty): Check whether the specified field is empty and remove the corresponding row.

[0040] Step S22: Field transformation: Process the specified field according to the configured mapping rules (such as removing leading and trailing spaces, converting to lowercase).

[0041] Step S23: Data filtering: Filter out data that meets the conditions (such as age>= 18).

[0042] Step S24: Sort: Sort the data according to the specified field, supporting ascending or descending sorting.

[0043] Step S25: Data output: Save the processed data in a specified format (such as CSV, JSON) according to the output configuration.

[0044] S3: The processing results of each step in step S2 are displayed through the visual interface module, which converts user interaction operations into YAML configuration change instructions.

[0045] S4: The configuration management module receives the change instructions and updates the configuration objects in memory. Based on the configuration management module's visual interface, the rules in the YAML configuration file are modified. The data processing engine re-executes the data processing flow based on the updated YAML configuration file. S4 includes: After the YAML configuration file is modified, the visual interface module displays the results in real time.

[0046] Next, we will introduce the visual interface and configuration management parts of this application in detail.

[0047] The visualization interface is used to display the effect of each data processing step, allowing users to dynamically view the processing progress and adjust the processing logic. The visualization interface has the following functions: Data preview and step display.

[0048] 1: Load and display the original data sample, so that users can visually view the input data.

[0049] 2: Display the real-time effect of each processing step. For example, in the "Field Conversion" step, users can see whether the field content has been converted according to the rules (such as removing spaces or converting to lowercase).

[0050] 3: Display the data changes of the current step, and users can preview the results of data processing in real time.

[0051] Interactive adjustments.

[0052] 1: Users can modify the rules in the YAML configuration through the interface. For example, users can modify the mapping relationship of fields by dragging and dropping, or modify the filter conditions (such as changing to age>=25) through the input box.

[0053] 2: After adjusting the configuration, the interface will update the data preview in real time to show the new processing results.

[0054] Real-time preview and adjustment.

[0055] During each step, users can review the results of each step to ensure that the processing is as expected. If necessary, users can roll back to the previous step or modify the processing configuration.

[0056] The configuration management module is responsible for reading, modifying, saving, and exporting YAML configuration files. This module supports the following functions: Read configuration: Read YAML configuration files from the file system or database.

[0057] Edit configuration: Users can modify the rules in the YAML configuration file through a visual interface and save it.

[0058] Export configuration: After completing the configuration, the user can export the configuration as a YAML file for reuse or sharing with other users.

[0059] Construction and underlying logic of the visual interface module: The visualization interface module is built using a web-based technology architecture, such as the Vue front-end framework combined with state management tools (such as Redux) and data visualization libraries (such as D3.js or ECharts). This module mainly includes the following functional components: Data preview component: used to load and display the original input data and the intermediate processing results of each data processing step, supporting page browsing and field filtering functions.

[0060] Processing step display component: displays each step in the YAML configuration file as a visual process node, allowing users to click to view or expand detailed configuration.

[0061] Graphical configuration editing component: Users can re-establish the mapping relationship between fields in the field map by dragging and dropping, or modify the logical expression (such as age>=25) in the filter condition editor through the input box.

[0062] Step change response mechanism: Whenever a user modifies a processing step configuration in the interface, the interface will trigger a configuration change event and dynamically refresh the preview results of downstream steps to ensure the consistency and predictability of the results.

[0063] Each processing step is structured as a separate "step node component" in the interface. This component includes state monitoring logic that automatically refreshes the display based on configuration changes. To improve efficiency, the system uses a difference detection mechanism (diff algorithm) to identify configuration changes, re-rendering and recalculating data only for the affected steps.

[0064] Construction and underlying logic of the configuration management module: The configuration management module is the core module in the system responsible for parsing, maintaining, editing, and exporting YAML configuration files. It mainly includes the following submodules: Configuration parsing submodule: parses the YAML text file into an internally structured configuration object (such as a JSON tree structure) for reading and rendering by the visualization interface module.

[0065] Configuration image maintenance submodule: The system maintains a mirror structure of the YAML configuration in memory, which serves as a synchronization bridge between the graphical interface interaction and the YAML file.

[0066] Bidirectional synchronization mechanism: Configuration images support bidirectional binding with the graphical interface and YAML text. When users modify field mappings or filter conditions in the visual interface, the image automatically updates and is reflected in the YAML text editing area. Conversely, when users edit the YAML text directly, the system also parses and updates the image and interface content.

[0067] Configuration verification submodule: After each user modifies the configuration, the system will call the built-in rule validator (such as based on predefined JSON Schema or regular expression rules) to check the legitimacy of the configuration items and prompt error information in real time in the interface to avoid processing failures caused by configuration errors.

[0068] Configuration saving and export submodule: supports exporting the current configuration as a YAML file. You can also choose to save it as a local file or upload it to a remote configuration center for version management and collaborative editing.

[0069] This configuration management module adopts a modular design with a clear interface, facilitating the subsequent expansion of other configuration formats (such as JSON and TOML). Through a unified management mechanism for graphical interaction and YAML configuration, it enables configuration collaboration between non-technical users and developers, significantly lowering the barrier to entry for using data processing systems.

[0070] In summary, through the specific construction method of the above-mentioned visualization interface and configuration management module, this application can achieve seamless integration of data processing rules defined based on YAML files and graphical interactive configuration. In terms of technical implementation, it is different from traditional static configuration file processing or closed graphical tools, and has high flexibility, real-time and scalability.

[0071] In this embodiment of the present application, the Shanghai Stock Exchange Information Public Service System converts the customer's Excel spreadsheet data into JSON format and maps it to the field format supported by the database. The YAML configuration is as follows: input: path: customers.xlsx format: excel output: path: customers_transformed.json format: json steps: - type: map_fields mappings: "Customer Name":"customer_name" "Contact":"contact" "Contact phone":"phone" - type: transform mappings: phone: normalize_phone - type: filter condition: "len(customer_name)>0" The data processing engine performs the following operations: Map Chinese fields to English; Standardize telephone numbers (e.g., using international area code format); Filter out records with missing customer names.

[0072] The visual interface displays a field mapping relationship diagram and allows users to click on fields to manually correct the mapping and refresh the preview window in real time.

Claims

1. A data processing system that can be configured and visualized based on YAML files, characterized in that: The data processing system includes: YAML configuration parsing module, used to parse the input configuration, output configuration and data processing configuration in the YAML configuration file; The data processing engine performs data loading, cleaning, conversion, filtering and sorting operations in sequence according to the configuration rules generated by the YAML configuration parsing module, and generates intermediate processing results; A visualization interface module displays the intermediate processing results of the data processing engine in real time, provides interactive controls to dynamically modify YAML configuration parameters, and synchronizes the modified configuration parameters to the YAML configuration parsing module; Configuration management module, used to save, load and export the YAML configuration file; The visualization interface module, configuration management module and data processing engine work together to form a dynamic synchronization mechanism. The visualization interface module converts user interaction operations into YAML configuration change instructions. The configuration management module receives the change instructions and updates the configuration objects in the memory. The data processing engine re-executes the data processing flow according to the updated configuration objects.

2. The data processing system according to claim 1, wherein: The visualization interface module includes a data preview component, a processing step display component, and a graphical configuration editing component. The data preview component is used to display the processing results of each step of the data processing configuration in real time. The processing step display component is used to display each step in the YAML configuration file in the form of a visual process node. The graphical configuration editing component is used to enable users to re-establish the mapping relationship between fields in the field mapping diagram by dragging and dropping, or to modify the logical expression through the input box in the filter condition editor.

3. The data processing system according to claim 1, wherein: The configuration management module includes a configuration parsing submodule, a configuration image maintenance submodule, a configuration verification submodule, and a configuration saving and exporting submodule. The configuration parsing submodule is used to parse the YAML text file into an internally structured configuration object for reading and rendering by the visualization interface module. The configuration image maintenance submodule is used to synchronize the graphical interface interaction with the YAML file.

4. The data processing system according to claim 1, wherein: The data processing configuration includes step types, fields to be processed, filtering conditions, and field mapping rules. The step types include at least: a null value row removal step type, a field conversion step type, a conditional filtering step type, and a sorting step type.

5. A data processing method for a data processing system based on a YAML file configurable visualization according to any one of claims 1 to 4, characterized in that: The data processing method includes: S1: Reading and parsing the YAML configuration file of the YAML configuration parsing module through the data processing engine; S2: Processing data by the data processing engine in a sequential order based on the YAML configuration file of the YAML configuration parsing module; S3: Displaying the processing results of each step in step S2 through the visual interface module, wherein the visual interface module converts the user interaction operation into a YAML configuration change instruction; S4: The configuration management module receives the change instruction and updates the configuration object in the memory. Based on the configuration management module modifying the rules in the YAML configuration file through the visual interface, the data processing engine re-executes the data processing flow according to the updated YAML configuration file.

6. The data processing method according to claim 5, characterized in that: Said S1 comprises: S11: Read the YAML configuration file and parse the input data, output data targets, and all processing steps; S12: Load input data and convert the input data into a data table or data frame according to the input configuration.

7. The data processing method according to claim 5, characterized in that: The S2 includes: Step S21: Based on the step type of removing empty value rows, checking whether the field to be processed is empty, and removing the corresponding empty value rows; Step S22: Based on the field conversion step type, the field to be processed is processed according to the field mapping rule; Step S23: Based on the conditional filtering step type, filter out data that meets the conditions according to the filtering conditions; Step S24: sorting the data based on the sorting step type; Step S25: Save the processed data in a specified format according to the output configuration.

8. The data processing method according to claim 5, characterized in that: The S4 includes: After the YAML configuration file is modified, the display result is updated in real time through the visual interface module interface.

Images