Spark-based dual-fixed-length file loading and unloading method, system, medium and device

Abstract

The present invention provides a Spark-based dual-fixed-length file loading and unloading system, method, medium and device, including: a file loading module: based on Spark, adaptively reads a marker file and a data file in a dual-fixed-length file, parses the marker file and the data file, and saves them to a Hive table; an unloading file module: based on Spark, reads the structure and data in the Hive table, and outputs the structure and data in the Hive table in a standardized manner.

Description

Technical Field

[0001] The present invention relates to the technical field of file systems, and in particular to a Spark-based dual-fixed-length file loading and unloading method, system, medium, and device. Background Art

[0002] Existing technologies can unload data across databases using fixed-length formats according to enterprise standards. However, these technologies only support relational databases like Oracle and Informix, lacking a unified tool for loading files into big data, and also lack support for unloading data from big data.

[0003] Patent document CN1866211B (application number: 200510124633.4) discloses a method for unloading a file system. In the unloading method of the present invention, in response to a forced unloading request for a specific file system, it is determined whether the file system to be unloaded is a root file system. If the file system to be unloaded is not a root file system, other processes are prohibited from accessing the file system to be unloaded and the files belonging to the file system. A first process that is executing a program existing in the file system to be unloaded is terminated. A second process waiting to obtain locks on all open files in the file system to be unloaded is awakened, and the locks are deleted. All open files are closed without affecting a third process that has opened files in the file system to be unloaded. A fourth process that is executing on a directory existing in the file system to be unloaded is notified that the directory is no longer available. The file system is unloaded using an unload operation.

[0004] Therefore, the present invention develops a tool for loading and unloading standard formats based on Spark, which fully utilizes the advantages of large data parallel computing. Summary of the Invention

[0005] In view of the defects in the prior art, the purpose of the present invention is to provide a Spark-based dual-fixed-length file loading and unloading method, system, medium and device.

[0006] According to the present invention, a Spark-based dual-fixed-length file loading and unloading system is provided, comprising:

[0007] File loading module: Based on Spark, it adaptively reads the marker file and data file in the double-fixed-length file, parses the marker file and data file, and saves them to the Hive table;

[0008] Unload file module: Read the structure and data in the Hive table based on Spark, and output the structure and data in the Hive table in a standardized manner.

[0009] Preferably, in the loading file module,

[0010] Module M1.1: Use Spark to read the marker file and data file in the double-fixed-length file, and parse the marker file and data file to obtain the starting position of each field, the read length, and the length of each data record;

[0011] Module M1.2: Read data records through Spark API and get the byte array of each record;

[0012] Module M1.3: Decode the byte array of each record, the starting position and length of each field to obtain the string of each field, generate RDD[Array[String]], and then generate RDD[Row];

[0013] Module M1.4: Get the DataSet object based on the RDD[Row] object through the Spark create dataset API;

[0014] Module M1.5: Save the DataSet object to the Hive table.

[0015] Preferably, in the uninstall file module,

[0016] Module M2.1: Use Spark to read the Hive table to obtain DataSet[Row], and get the length of each field to be exported from the DataSet[Row].

[0017] Module M2.2: Converts DataSet[Row], reads the contents of each field in each row, and then encodes the contents to generate a byte array. If the byte array length is less than the export length, it is padded with spaces, and finally generates RDD[Array[byte]];

[0018] Module M2.3: RDD[Array[byte]] is standardized and output as a file according to each partition, then all the standardized output files of all partitions are merged into one file, and finally the corresponding marked file is generated.

[0019] Preferably, in the module M2.3, data files are stored in partitions, and when data files are read based on the directory to be exported, they are read in parallel using multiple machine partitions.

[0020] According to the present invention, a Spark-based dual-fixed-length file loading and unloading method is provided, comprising:

[0021] File loading step: Based on Spark, the marker file and data file in the double-fixed-length file are adaptively read, the marker file and data file are parsed, and the data files are saved to the Hive table;

[0022] Unloading file steps: Read the structure and data in the Hive table based on Spark, and output the structure and data in the Hive table in a standardized manner.

[0023] Preferably, the file loading step adopts:

[0024] Step S1.1: Use Spark to read the marker file and data file in the double-fixed-length file, and parse the marker file and data file to obtain the starting position of each field, the read length, and the length of each data record;

[0025] Step S1.2: Use Spark to read the data record API and obtain the byte array of each record;

[0026] Step S1.3: Decode the byte array of each record, the starting position and length of each field to obtain the string of each field, generate RDD[Array[String]], and then generate RDD[Row];

[0027] Step S1.4: Get the DataSet object based on the RDD[Row] object through the Spark create dataset API;

[0028] Step S1.5: Save the DataSet object to the Hive table.

[0029] Preferably, in the uninstall file module,

[0030] Step S2.1: Use Spark to read the Hive table to obtain DataSet[Row], and obtain the length of each field to be exported from the DataSet[Row].

[0031] Step S2.2: Convert DataSet[Row], read the contents of each field of each record Row in turn, and then encode the contents to generate a byte array. If the length of the byte array is less than the export length, it is padded with spaces, and finally generate RDD[Array[byte]];

[0032] Step S2.3: RDD[Array[byte]] is standardized and output as a file according to each partition, and then all the standardized output files of all partitions are merged into one file, and finally the corresponding marked file is generated.

[0033] Preferably, in step S2.3, the data files are stored in partitions, and when the data files are read based on the directory to be exported, they are read in parallel using multiple machine partitions.

[0034] According to the present invention, a computer-readable storage medium storing a computer program is provided. When the computer program is executed by a processor, the steps of the method described above are implemented.

[0035] According to the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. The electronic device is characterized in that when the computer program is executed by the processor, the steps of the above-mentioned Spark-based dual-fixed-length file loading and unloading method are implemented.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] 1. The present invention is for the standard format provided by the upstream, and if the format changes, the tool will automatically adapt;

[0038] 2. If the fields in the big data table are changed, the uninstall tool will automatically adapt;

[0039] 3. This tool uses the popular Spark computing to load and unload, which improves the reading and writing efficiency;

[0040] 4. The present invention can read and generate files of different encoding formats by dynamically configuring the encoding and decoding format when reading and writing files. For example, when unloading data files, files in GBK format can be generated, and files in UTF8 format can also be generated;

[0041] 5. Load the file module. By inputting the flag file and data file, the encoding format in the file can be dynamically matched. The file can be parsed to convert the semi-structured data into structured data, and then the data can be processed subsequently.

[0042] 6. Unload file module, through the input table, can output the data in the table in a standardized manner, and generate the corresponding logo file for use by downstream systems;

[0043] 7. The present invention can realize read and write operations on multiple machines with higher efficiency BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0045] Figure 1 This is a flowchart of the Spark-based double-fixed-length file loading and unloading method. DETAILED DESCRIPTION

[0046] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0047] Example 1

[0048] According to the present invention, a Spark-based dual-fixed-length file loading and unloading system is provided, comprising:

[0049] File loading module: Based on Spark, it adaptively reads the marker file and data file in the double-fixed-length file, parses the marker file and data file, and saves them to the Hive table;

[0050] Unload file module: Read the structure and data in the Hive table based on Spark, and output the structure and data in the Hive table in a standardized manner.

[0051] Specifically, in the loading file module,

[0052] Module M1.1: Use Spark to read the marker file and data file in the double-fixed-length file, and parse the marker file and data file to obtain the starting position of each field, the read length, and the length of each data record;

[0053] Module M1.2: Read data records through Spark API and get the byte array of each record;

[0054] Module M1.3: Decode the byte array of each record, the starting position and length of each field to obtain the string of each field, generate RDD[Array[String]], and then generate RDD[Row];

[0055] Module M1.4: Get the DataSet object based on the RDD[Row] object through the Spark create dataset API;

[0056] Module M1.5: Save the DataSet object to the Hive table.

[0057] Specifically, in the uninstall file module,

[0058] Module M2.1: Use Spark to read the Hive table to obtain DataSet[Row], and get the length of each field to be exported from the DataSet[Row].

[0059] Module M2.2: Converts DataSet[Row], reads the contents of each field in each row, and then encodes the contents to generate a byte array. If the byte array length is less than the export length, it is padded with spaces, and finally generates RDD[Array[byte]];

[0060] Module M2.3: RDD[Array[byte]] is standardized and output as a file according to each partition, then all the standardized output files of all partitions are merged into one file, and finally the corresponding marked file is generated.

[0061] Specifically, in the module M2.3, data files are stored in partitions, and when data files are read based on the directory to be exported, they are read in parallel using multiple machine partitions.

[0062] According to a Spark-based dual-fixed-length file loading and unloading method provided by the present invention, Figure 1 Shown, including:

[0063] File loading step: Based on Spark, the marker file and data file in the double-fixed-length file are adaptively read, the marker file and data file are parsed, and the data files are saved to the Hive table;

[0064] Unloading file steps: Read the structure and data in the Hive table based on Spark, and output the structure and data in the Hive table in a standardized manner.

[0065] Specifically, the file loading step adopts:

[0066] Step S1.1: Use Spark to read the marker file and data file in the double-fixed-length file, and parse the marker file and data file to obtain the starting position of each field, the read length, and the length of each data record;

[0067] Step S1.2: Use Spark to read the data record API and obtain the byte array of each record;

[0068] Step S1.3: Decode the byte array of each record, the starting position and length of each field to obtain the string of each field, generate RDD[Array[String]], and then generate RDD[Row];

[0069] Step S1.4: Get the DataSet object based on the RDD[Row] object through the Spark create dataset API;

[0070] Step S1.5: Save the DataSet object to the Hive table.

[0071] Specifically, in the uninstall file module,

[0072] Step S2.1: Use Spark to read the Hive table to obtain DataSet[Row], and obtain the length of each field to be exported from the DataSet[Row].

[0073] Step S2.2: Convert DataSet[Row], read the contents of each field of each record Row in turn, and then encode the contents to generate a byte array. If the length of the byte array is less than the export length, it is padded with spaces, and finally generate RDD[Array[byte]];

[0074] Step S2.3: RDD[Array[byte]] is standardized and output as a file according to each partition, and then all the standardized output files of all partitions are merged into one file, and finally the corresponding marked file is generated.

[0075] Specifically, in step S2.3, the data file is saved in partitions, and when the data file is read based on the directory to be exported, the data file is read in parallel using multiple machine partitions.

[0076] According to the present invention, a computer-readable storage medium storing a computer program is provided. When the computer program is executed by a processor, the steps of the method described above are implemented.

[0077] According to the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. The electronic device is characterized in that when the computer program is executed by the processor, the steps of the above-mentioned Spark-based dual-fixed-length file loading and unloading method are implemented.

[0078] Example 2

[0079] Example 2 is a preferred example of Example 1

[0080] The encoding format of the upstream file may be GBK and UTF8. The tool can automatically identify and parse the text according to different encodings. Traditional data export can only use multi-threading on one machine to improve export efficiency. The present invention is based on big data calculation and can naturally realize read and write operations on multiple machines, which is more efficient.

[0081] The present invention provides a tool for reading and unloading a GUT standard format based on Spark, comprising: reading files and unloading files.

[0082] Read the gut standard format, use Spark to read the flag file in the double-length file, parse the file to obtain the starting position of each field, the read length, and the length of each record; then use Spark's API to read binary records to obtain RDD[bytes[]], and then split each field according to the byte array of each record and the starting position and length of each previous field to obtain an RDD[Row] object; use Spark's API to create a dataset to obtain a DataSet object; finally, save the DataSet object to a Hive table.

[0083] Unload the GUT standard format from big data. Use Spark to read the structure of the Hive metadata table. The table fields contain lengths, and the length of each field to be exported is obtained. Because the exported file is in HDFS, a directory to be exported is first created. Spark reads the table data to obtain a DataSet, reads the content of each field, and based on the length of each field, adds spaces if the actual content length is less than the exported length. The reading is carried out by unloading each partition and finally merging them. Write the data file first and then write the standard file.

[0084] The reading module first reads the flag file, parses the field information, parses the record length, then reads the data file, generates an object structure with a schema, and calls dataset.saveAsTable to save it in the target table;

[0085] Uninstall the module, first read the table information, generate the length information of each field, read each field, generate a string for each field content, write it into the file in sequence, and finally export it in a unified standard format;

[0086] The following is an example of the tag file format:

[0087] Field sequence, English name, type, start position, end position

[0088] 1$$YHBH$$CHAR(9)$$(1,9)

[0089] 2$$RYBH$$CHAR(6)$$(10,15)

[0090] 3$$GYH$$CHAR(6)$$(16,21)

[0091] 4$$YHMC$$VARCHAR2(60)$$(22,81)

[0092] 5$$YHDLM$$VARCHAR2(32)$$(82,113)

[0093] The collection and unloading files of the tenant big data platform of the enterprise financial customer relationship management system use this tool to load and export files; the data platform is responsible for loading data files into the corresponding HDFS directory, and the tenant big data platform is responsible for loading double-fixed-length files and saving them into the corresponding Hive table; when the table data processed by the tenant platform needs to be exported to the downstream system, this tool is used to export the corresponding table into double-fixed-length files and push the files to the downstream system for use

[0094] Those skilled in the art will appreciate that, in addition to implementing the system, device, and various modules provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, and the like by logically programming the method steps. Therefore, the system, device, and various modules provided by the present invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; the modules for implementing various functions can also be considered both software programs for implementing the method and structures within the hardware component.

[0095] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A dual-fixed-length file loading and unloading system based on Spark, characterized in that: include: File loading module: Based on Spark, it adaptively reads the marker file and data file in the double-fixed-length file, parses the marker file and data file, and saves them to the Hive table; Unload file module: read the structure and data in the Hive table based on Spark, and output the structure and data in the Hive table in a standardized manner; In the loading file module, Module M1.1: Use Spark to read the marker file and data file in the double-fixed-length file, and parse the marker file and data file to obtain the starting position of each field, the read length, and the length of each data record; Module M1.2: Read data records through Spark API and get the byte array of each record; Module M1.3: Decode the byte array of each record, the starting position and length of each field to obtain the string of each field, generate RDD[Array[String]], and then generate RDD[Row]; Module M1.4: Get the DataSet object based on the RDD[Row] object through the Spark create dataset API; Module M1.5: Save the DataSet object to the hive table; In the uninstall file module, Module M2.1: Use Spark to read the Hive table to obtain DataSet[Row], and get the length of each field to be exported from the DataSet[Row]. Module M2.2: Converts DataSet[Row], reads the contents of each field in each row, and then encodes the contents to generate a byte array. If the byte array length is less than the export length, it is padded with spaces, and finally generates RDD[Array[byte]]; Module M2.3: RDD[Array[byte]] is standardized and output as a file according to each partition, then all the standardized output files of all partitions are merged into one file, and finally the corresponding marked file is generated.

2. The Spark-based dual-fixed-length file loading and unloading system according to claim 1, characterized in that: In the module M2.3, data files are stored in partitions. When reading data files based on the directory to be exported, they are read in parallel using multiple machine partitions.

3. A method for loading and unloading double-fixed-length files based on Spark, characterized in that: include: File loading step: Based on Spark, the marker file and data file in the double-fixed-length file are adaptively read, the marker file and data file are parsed, and the data files are saved to the Hive table; Unloading file steps: Read the structure and data in the Hive table based on Spark, and output the structure and data in the Hive table in a standardized manner; The file loading step adopts: Step S1.1: Use Spark to read the marker file and data file in the double-fixed-length file, and parse the marker file and data file to obtain the starting position of each field, the read length, and the length of each data record; Step S1.2: Use Spark to read the data record API and obtain the byte array of each record; Step S1.3: Decode the byte array of each record, the starting position and length of each field to obtain the string of each field, generate RDD[Array[String]], and then generate RDD[Row]; Step S1.4: Get the DataSet object based on the RDD[Row] object through the Spark create dataset API; Step S1.5: Save the DataSet object to the Hive table; In the step of uninstalling files, Step S2.1: Use Spark to read the Hive table to obtain DataSet[Row], and obtain the length of each field to be exported from the DataSet[Row]. Step S2.2: Convert DataSet[Row], read the contents of each field of each record Row in turn, and then encode the contents to generate a byte array. If the length of the byte array is less than the export length, it is padded with spaces, and finally generate RDD[Array[byte]]; Step S2.3: RDD[Array[byte]] is standardized and output as a file according to each partition, and then all the standardized output files of all partitions are merged into one file, and finally the corresponding marked file is generated.

4. The Spark-based dual-fixed-length file loading and unloading method according to claim 3, characterized in that: The step S2.3 adopts the method of partitioning and saving the data file. When reading the data file based on the directory to be exported, the data file is read in parallel by using multiple machine partitions.

5. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 3 to 4 are implemented.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is executed by a processor, the steps of the Spark-based double-fixed-length file loading and unloading method described in any one of claims 3 to 4 are implemented.

Images