Data processing methods and systems

Abstract

A data processing system includes a storage module, a data conversion module, and a data processing module. The data conversion module is used to obtain a data table to be imported from a data source according to a data import instruction, and convert the data table into a converted data table conforming to a predetermined input format. The data processing module is used to receive the converted data table, store the converted data table in the storage module, and execute a processing task according to a data processing instruction to obtain a processing result, and store the processing result and all record files generated during the processing task in the storage module.

Claims

1. A data processing system deployed in a cloud-native containerized environment, comprising: a storage module; and a data conversion module for obtaining a data table to be imported from a data source according to a data import instruction, wherein the data import instruction includes a data source setting indicating the data source to be imported, a target data table setting indicating the data table to be imported, and a number of execution threads setting for obtaining the data table to be imported, wherein... The data import instruction describes the data source settings, the target data table settings, and the number of execution threads in a domain-specific language, and converts the data table to be imported into a converted data table that conforms to a predetermined input format; and a data processing module, connected to the data conversion module and the storage module, is used to receive the converted data table, store the converted data table in the storage module, execute a processing task according to a data processing instruction to obtain a processing result, and store the processing result and all record files generated during the processing of the processing task in the storage module.

2. The data processing system as described in claim 1 further includes: a visualization module connected to the data processing module, used to output the record file generated by the data processing module when performing the processing task in the form of a chart.

3. The data processing system as described in claim 1, wherein, The task is to perform federated learning training based on the transformed data table to build a federated learning model, and use the model parameters of the federated learning model as the result of the operation.

4. The data processing system as described in claim 3, wherein, The computational task is to obtain a prediction result corresponding to the transformed data table using the federated learning model, and to use the prediction result as the computational result.

5. The data processing system as described in claim 4, wherein, The data conversion module obtains the prediction result and converts it into a data table to be exported that conforms to a predetermined output format.

6. A data processing method, executed using a computing device deployed in a cloud-native containerized manner in a cluster environment, includes the following steps: (A) obtaining a data table to be imported from a data source according to a data import instruction, the data import instruction including a data source setting indicating the data source to be imported, a target data table setting indicating the data table to be imported, and a thread count setting for obtaining the data table to be imported, the data import instruction describing the data source setting, the target data table setting, and the thread count setting in a domain-specific language; (B) converting the data table to be imported into a converted data table conforming to a predetermined input format, and storing the converted data table; (C) executing a computation task according to a data computation instruction to obtain a computation result, and storing the computation result and all record files generated during the execution of the computation task; and (D) outputting the record files generated during the execution of the computation task in the form of a chart.

7. The data processing method as described in claim 6, wherein, In step (C), the computational task is to perform federated learning training based on the transformed data table to establish a federated learning model, and the computational result is the model parameters of the federated learning model.

8. The data processing method as described in claim 7, wherein, In step (C), the computational task is to obtain a prediction result corresponding to the transformed data table using the federated learning model based on the transformed data table. The computational result is the prediction result. After step (C), the following steps are also included: (E) converting the prediction result into a data table to be exported that conforms to a predetermined output format.

Images