Data table maintenance method, device, storage medium and electronic equipment for data synchronization
A data synchronization and data table technology, applied in the field of data management, can solve problems such as affecting the progress of data analysis business, unusable data in data tables, and low efficiency.
Active Publication Date: 2020-12-25
NEUSOFT CORP
View PDF4 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
However, when writing data files to HDFS, garbage files may be generated in HDFS due to data errors, network disconnection, program interruption (disconnection), etc.
At this time, if you continue to write data to the directory of the analysis engine, the data in the entire data table may become unusable, which will affect the data analysis business.
In related technologies, operation and maintenance personnel are usually required to track and maintain the above-mentioned junk files, and the labor cost is high and the efficiency is low
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0058] Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with aspects of the present disclosure as recited in the appended claims.
[0059] Before introducing the data table maintenance of data synchronization provided by the present disclosure, the target application scenario involved in each embodiment of the present disclosure is firstly introduced, the target application scenario includes a business system and one or more analysis engines, and the business system serves as Data sources are used to generate, coll...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention relates to a data table maintenance method and device for data synchronization, a storage medium and electronic equipment. The method comprises the following steps of: when the current data synchronization task is started; according to the execution state of the previous data synchronization task, determining whether to delete a first data file written into an analysis engine data table when execution of the previous data synchronization task fails, and converting the data file currently needing to be synchronized into a temporary data file with a target identifier as a second data file; writing the second data file into the data table through file writing operation and renaming operation, and converting the second data file into an effective data file; and when an error occurs in the file writing operation or the renaming operation, deleting the target data file written into the data table before the error occurs according to the target identifier. According to the method, before data synchronization and in the data synchronization process, junk files generated due to synchronization failure can be cleared, the step of manual intervention in data table maintenance isavoided, and the operation and maintenance efficiency of data synchronization is improved.
Description
technical field [0001] The present disclosure relates to the field of data management, and in particular, to a data synchronization data table maintenance method, device, storage medium and electronic equipment. Background technique [0002] When enterprises do data analysis, they need to synchronize business data to distributed system infrastructure Hadoop through ETL (Extract-Transform-Load, extract-interactive transformation-load) data warehouse technology for analysis engines (for example, Hive or Impala, etc.) use. In the ETL data synchronization process, the JDBC (Java DataBaseConnectivity, Java database connection) operation of the analysis engine is directly performed, and the data transmission speed is very slow. In order to improve the speed of data transmission, the data is usually written to HDFS (Hadoop Distributed File System, Hadoop Distributed File System) in a predetermined format, and then the data is loaded into the data table on the analysis engine by ex...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/27G06F16/25
Inventor 范超李东鸽牟晓光
Owner NEUSOFT CORP