Concurrent processing method, device and system for format data files

Abstract

The invention provides a concurrent processing method, a device and a system for format data files. A client acquires file fragmentation meta-information sent by a server; the client gradually reads the file fragment content corresponding to the file fragment meta-information from the format record file by using the cache window, so that the read file fragment content ensures that the content withone more separator length is read at the beginning of the file fragment, and the content of at least one more cache window is read at the end of the file fragment; the file fragment contents are analyzed one by one in the read file fragment contents and deleting the first format record; and when the total length of the analyzed content of the file fragment is equal to the length of the file fragment and the analysis of the last format record of the file fragment does not meet the separator, analyzing is continued until the separator is analyzed, stopping the analysis operation, and determining the plurality of format records after the analysis operation as the format record set of the file fragment. According to the method, the format data files can be read concurrently, the integrity offormat records can be guaranteed, and the reliability is guaranteed while the rate is increased.

Description

technical field [0001] The present application relates to the field of communication technology, and in particular to a concurrent processing method, device and system for formatted data files. Background technique [0002] Format data files are usually used for data interaction between heterogeneous platforms. Format data files are composed of multiple format records and are a type of file specially used to store format records. Records of different formats in the format data file are separated by a predefined delimiter (such as a newline character [\n]). [0003] With the gradual development of communication technology, the data volume of the format data file is also gradually increasing (for example, gradually increasing to hundreds of G data volume or even up to T data volume). In order to facilitate the transmission of format data files between heterogeneous platforms, the format data files are usually fragmented. [0004] A current fragment processing solution is tha...

AI Technical Summary

Problems solved by technology

Although the integrity of format records can be guaranteed in this scheme, the speed of serial processing is low

Images