Data file coding format conversion method and system

Abstract

The invention relates to a data file coding format conversion method and system, and the system comprises a file obtaining module which is used for obtaining an input file uploaded by a user, and recording the uploading time of each input file and a corresponding user name; a thread management module which is used for creating thread tasks for the conversion tasks of the input files, taking out aplurality of thread tasks in parallel according to uploading time and sending the thread tasks to the corresponding analysis modules; an analysis module which is used for reading an input file uploaded by a user by adopting an original encoding format to obtain a to-be-converted file corresponding to the input file; a transcoding module which is used for transcoding the to-be-converted file by adopting a coding format which can be recognized by a big data fusion development system to obtain a new input file; and a file output module which is used for outputting the transcoded new input file toa big data fusion development system for subsequent processing. The method can be widely applied to the field of data transcoding.

Description

technical field [0001] The invention relates to a data file encoding format conversion method and system, belonging to the field of data processing. Background technique [0002] Big data fusion development system is a comprehensive software with functions of data access, data cleaning, data control and automatic analysis. In the process of landing a big data platform, data access is an indispensable key link. In the face of various sources and types of data, data access is needed to integrate these scattered data into a unified big data platform. From the perspective of data types, data access mainly includes structured data (database) access, log data access, IoT data access, and file access. [0003] However, data in different formats cannot be imported into the database without conversion, and cannot be used subsequently. In the existing big data fusion development system, there is no implementation of this function. If the user imports data with wrong format, it will...

AI Technical Summary

Problems solved by technology

[0003] However, data in different formats cannot be imported into the database without conversion, and cannot be used subsequently

In the existing big data fusion development system, there is no realization of this function. If the user imports data with wrong format, it will lead to invalid data in the database.

At the same time, data format conversion also faces the following problems: first, data security, that is, how to ensure that existing data can be completely stored in a new file during the conversion process, the conversion process is more complicated, and there may be many emergencies that threaten the data Security; secondly, the conversion speed, the data files of the long-running system are huge, and the time required for the conversion is also relatively long; finally, the data imported by the user into the database is usually a text with a large amount of data, which also increases the data conversion. Difficulty

Images