A data flow based file compression process adjustment system and method

Abstract

The application discloses a file compression process adjusting system and method based on data flow, relates to the technical field of data compression, and comprises the following steps: reading an input flow and judging file creation demand; when a file needs to be created, an enhanced output file output handle is created, which is recorded as a first object, a second object is generated by wrapping the first object with a tar package, the second object is wrapped with a gzip object along with its identifier, the identifier is kept after the tar output record of the object is created, the first object control switch is closed, and the flow creation process is completed; data flow is written and file splitting conditions are judged, if the file splitting conditions are met, a flow closing process is entered; the second object and the first object are closed, a tar file header and a tar file body gzip compression segment file pair are generated, it is judged whether a corresponding merging operation needs to be performed on a file system, and the flow closing process is completed, so that the file compression process is adjusted.

Description

Technical Field

[0001] This invention relates to the field of data compression technology, specifically a file compression process adjustment system and method based on data streams. Background Technology

[0002] With the deep penetration of big data technology into finance, the Internet and other fields, the amount of data is growing exponentially. Real-time processing and efficient compression of data streams have become the core links supporting data transmission, storage and backup services. Among them, the tar.gz format has become the mainstream compression format in this scenario due to its strong compatibility and stable compression effect. Its processing efficiency determines the throughput, resource utilization and business response speed of the entire data link, which is of great significance to ensuring business continuity in big data scenarios.

[0003] However, the traditional method of reading data from a data stream and generating a tar.gz file has gradually revealed performance bottlenecks and resource constraints when dealing with high throughput demands (such as data throughput often exceeding 10 GBps). The shortcomings of the traditional method are mainly reflected in four aspects: First, data writing is redundant. The data file needs to undergo at least two write operations. In particular, the temporary file generated in the first stage is in an uncompressed format, which consumes a lot of storage resources and further increases the IO burden. Second, disk IO is limited. Due to the inherent limitations of disk read and write speed, the writing and reading process of temporary files will slow down the software processing efficiency, causing the data processing chain to be stuck. Third, memory resources are overloaded. Although this method reduces disk IO consumption by temporarily storing temporary files in memory, it needs to continuously occupy a large amount of memory until the tar.gz file is generated. The limited total amount of memory restricts the system's concurrent processing capabilities, and memory can easily become a new performance bottleneck in high-concurrency scenarios. Finally, the advantages of streaming processing are lost. The traditional process requires an additional independent packaging and compression step or thread, which not only makes the software design redundant and complex, but also fails to give full play to the characteristics of real-time data stream processing and is difficult to adapt to the low-latency requirements of big data scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide a file compression process adjustment system and method based on data streams to solve the problems raised in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a file compression process adjustment method based on data streams, the method comprising the following steps:

[0006] Read the input stream, determine whether a file needs to be created, and execute the stream creation process if a file needs to be created.

[0007] The execution flow creation process includes: creating an enhanced output file handle, denoted as the first object, and turning on the control switch of the first object; generating a second object by packaging the first object with tar, then packaging the second object with a gzip object and using its identifier, creating a tar output record in the object, and turning off the control switch of the first object;

[0008] After the stream creation process is completed, the data stream writing process is performed.

[0009] After the data stream writing process is completed, the decision chain is entered to determine file segmentation.

[0010] When the file splitting condition is met, the stream closing process begins;

[0011] The stream closing process includes: closing the second object and the first object, generating a compressed fragment of the tar file body; obtaining information from the first object to generate a gzip compressed fragment of the tar file header, forming a fragment pair, determining whether the file system needs to perform corresponding merging operations, completing the stream closing process, and jumping to read the input stream.

[0012] Read the input stream, determine if file creation is needed, and execute the stream creation process if file creation is needed; otherwise, perform a write operation based on the read input stream data. Specific steps include:

[0013] Based on the read data records, the decision-making process is initiated. In the context, it is checked whether a suitable file read / write handle exists. If a suitable file read / write handle exists, the process of writing data streams is initiated; otherwise, the process of stream creation is initiated.

[0014] When entering the data processing stage of writing data stream, data is written to the output file handle. After completion, the decision chain is entered to perform file segmentation judgment: check whether the number of output bytes reaches S, whether the number of output records reaches N, and whether the data recording time span reaches M minutes. If any result meets the condition, the stream is closed. If none of the conditions are met, the data processing process is executed in a loop and the input stream is read. Here, S represents the maximum number of bytes in the generated file, N represents the maximum number of records in the generated file, and M represents the maximum duration of the generated file.

[0015] Based on the read data records, the decision-making process begins. Within the context, a suitable file read / write handle is checked. If a suitable file read / write handle exists, the data stream writing process begins; otherwise, the stream creation process begins. The specific steps of the stream creation process include:

[0016] Create an enhanced output file handle, denoted as the first object;

[0017] Setting the control switch of the first object to true will prevent the output of the tar header. That is, when the control switch of the first object is turned on, all content written to that object will be discarded.

[0018] The process of creating a tar stream is executed, which packages the first object with tar and generates a packaged output object, which is denoted as the second object.

[0019] The process of creating a gzip stream is executed. The second object is wrapped with a gzip object. The wrapped object replaces the wrapped output object obtained by wrapping the first object with tar. The second object's identifier is still used after the replacement.

[0020] The process of creating a tar file output stream is executed, and a tar output record is created in the replaced second object. The identifier of the second object is retained after the update.

[0021] Perform the process to disable the skipped output and turn off the control switch for the first object.

[0022] When entering the data processing stage of writing data to the output file handle, data is written to the output file handle. After completion, the decision chain is entered to determine file segmentation: the output byte count is checked sequentially to see if it reaches S, the output record count reaches N, and the data record time span reaches M minutes. If any of the conditions are met, the stream is closed. The specific steps include:

[0023] Perform the operation to close the tar file stream, close the second object and the first object, and generate a compressed fragment file of the tar file body;

[0024] Perform the operation to generate a tar.gz file header, obtain the number of bytes in the output stream, the number of records processed, the time range of the record content, and the MD5 value of the output stream from the first object, generate the target file name according to the preset requirements, and generate a gzip compressed fragment file containing tar file header information;

[0025] At this point, we have obtained a pair of gzip compressed fragment files consisting of the tar file header and the tar file body.

[0026] Determine if the file system needs to perform a merge operation, complete the stream closing process, and then jump to the data processing process to continue the loop. Specific steps include:

[0027] Determine the file system to which the output is written. If the file system to which the output is written is HDFS, continue to perform the operation of merging the gzip compressed fragments of the tar file header and the tar file body: call the concat method to merge the gzip compressed fragments of the generated tar file header into the gzip compressed fragments of the tar file body to form a tar.gz file;

[0028] Once the stream closing process is complete, the process jumps back to the data processing process to read the input stream and continues to execute the data processing process in a loop.

[0029] A file compression process adjustment system based on data streams includes: an input stream processing module, a stream creation module, a data writing and segmentation judgment module, and a stream closing and fragment processing module. The input stream processing module reads the input stream, determines whether file creation is needed, and executes the stream creation process when file creation is needed; otherwise, it performs a data stream writing operation based on the read input stream data. The stream creation module, when file creation is needed, creates an enhanced output file handle (denoted as a first object), sets its control switch to true to discard the tar header output, generates a second object by tar packaging the first object, then packages the second object with a gzip object while retaining its identifier, and creates and saves the tar output record within this object. Holding the identifier, the control switch of the first object is turned off, completing the stream creation process; the data writing and segmentation judgment module is used to complete the stream creation process, enter the data processing process to write data stream, write data to the output file handle, and after completion, enter the decision chain to perform file segmentation judgment. If the file segmentation condition is met, the stream closing process is entered; otherwise, the loop returns to the reading input stream step to continue processing; the stream closing and fragment processing module is used to enter the stream closing process, close the second object and the first object, and generate a tar file body compressed fragment; obtain information from the first object to generate a tar file header gzip compressed fragment, form a fragment pair, determine whether the file system needs to perform corresponding merging operations, complete the stream closing process, and jump to the data processing process to continue the loop.

[0030] The input stream processing module includes an input stream reading unit and a file creation requirement judgment unit. The input stream reading unit is used to read data records from the target data stream in real time. The file creation requirement judgment unit is used to check the available file read / write handles in the context based on the read data records: if an available handle exists, the data writing and segmentation judgment module is triggered; if no available handle exists, the stream creation module is triggered.

[0031] The stream creation module includes an enhanced output stream creation unit, a tar stream wrapping unit, a gzip stream wrapping unit, a tar output record creation unit, and a switch control unit. The enhanced output stream creation unit is used to create an enhanced output file handle, denoted as the first object. The tar stream wrapping unit is used to wrap the first object using the tar tool to generate a wrapped output object. The gzip stream wrapping unit uses the gzip object to wrap a second object, retaining the identifier of the second object. The tar output record creation unit is used to create a tar output record in the gzip-wrapped second object. The switch control unit is used to control the skip output control switch of the first object in stages. After creating an enhanced output file handle, the control switch of the first object is set to true, so that the tar header is not output. After the tar output record is created, the switch is set to off.

[0032] The data writing and segmentation judgment module includes a data stream writing unit and a segmentation condition judgment unit. The data stream writing unit is used to receive data records in the input stream and write data to the second object in real time. The segmentation condition judgment unit is used to sequentially check whether the number of output bytes reaches S, whether the number of output records reaches N, and whether the data record time span reaches M minutes. If any result meets the condition, the stream is closed. If none of the results meet the condition, the data processing process is executed in a loop and the input stream is read. Here, S represents the maximum number of bytes in the generated file, N represents the maximum number of records in the generated file, and M represents the maximum duration of the generated file.

[0033] The stream closing and fragment processing module includes an output stream closing unit, a compressed fragment generation unit, and a fragment merging unit. The output stream closing unit is used to perform the operation of closing the tar file stream, closing the second object and the first object, and generating a compressed fragment file of the tar file body. The compressed fragment generation unit is used to read statistical information from the first object, generate a target file name according to a preset naming rule, and generate a gzip compressed fragment of the tar file header to form a compressed fragment pair of tar file header and file body. The fragment merging unit is used to determine the file system to which the output is written. If the file system to which the output is written is HDFS, the operation of merging the gzip compressed fragment file pair of tar file header and tar file body is continued: the concat method is called to merge the gzip compressed fragment of the generated tar file header into the gzip compressed fragment of the tar file body to form a tar.gz file. When the stream closing process is completed, the process continues to jump to the process of reading the input stream in the data processing process and continues to loop the data processing process.

[0034] Compared with the prior art, the beneficial effects of the present invention are:

[0035] 1. By splitting the tar file generation process, the compressed tar file body is first written in a streaming manner, and then the tar header information is completed. This achieves only one file write operation and the written content is compressed data, which can save more than 70% of hard disk write operations. Unlike the two data write modes of "writing uncompressed temporary files + writing tar.gz files" in the existing technology, this invention reduces the total amount of data written and avoids the extra occupation of hard disk storage and read / write resources by uncompressed temporary files.

[0036] 2. By optimizing the tar file generation logic, there is no need to temporarily store uncompressed temporary data during the streaming data stream processing, and no additional memory consumption is required, which reduces the server hardware configuration requirements. Unlike the optimization approach of "using memory image files to reduce disk I / O but causing continuous memory occupation and memory overload under high concurrency" in the existing technology, this invention avoids the invalid occupation of memory resources and can release related resources without waiting for the tar.gz file to be fully generated.

[0037] 3. This invention adds auxiliary processing to the file output stream creation and closure nodes of streaming processing, eliminating the need for additional independent steps and threads for tar packaging or gzip compression. Unlike the existing technology that requires additional threads / steps to complete compression and packaging, resulting in redundant software design, this invention simplifies the overall data compression process and avoids the occupation of CPU resources by additional threads. Attached Figure Description

[0038] Figure 1 This is a flowchart illustrating a file compression process adjustment method based on data stream according to the present invention. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] like Figure 1 As shown, the present invention provides a technical solution, a method for adjusting the file compression process based on data streams, the method comprising the following steps:

[0041] Read the input stream, determine whether a file needs to be created, and execute the stream creation process if a file needs to be created.

[0042] The execution flow creation process includes: creating an enhanced output file handle, denoted as the first object, and turning on the control switch of the first object; generating a second object by packaging the first object with tar, then packaging the second object with a gzip object and using its identifier, creating a tar output record in the object, and turning off the control switch of the first object;

[0043] After the stream creation process is completed, the data stream writing process is performed.

[0044] After the data stream writing process is completed, the decision chain is entered to determine file segmentation.

[0045] When the file splitting condition is met, the stream closing process begins;

[0046] The stream closing process includes: closing the second object and the first object, generating a compressed fragment of the tar file body; obtaining information from the first object to generate a gzip compressed fragment of the tar file header, forming a fragment pair, determining whether the file system needs to perform corresponding merging operations, completing the stream closing process, and jumping to read the input stream.

[0047] Read the input stream, determine if file creation is needed, and execute the stream creation process if file creation is needed; otherwise, perform a write operation based on the read input stream data. Specific steps include:

[0048] Based on the read data records, the decision-making process is initiated. In the context, it is checked whether a suitable file read / write handle exists. If a suitable file read / write handle exists, the process of writing data streams is initiated; otherwise, the process of stream creation is initiated.

[0049] When entering the data processing stage of writing data stream, data is written to the output file handle. After completion, the decision chain is entered to perform file segmentation judgment: check whether the number of output bytes reaches S, whether the number of output records reaches N, and whether the data recording time span reaches M minutes. If any result meets the condition, the stream is closed. If none of the conditions are met, the data processing process is executed in a loop and the input stream is read. Here, S represents the maximum number of bytes in the generated file, N represents the maximum number of records in the generated file, and M represents the maximum duration of the generated file.

[0050] Based on the read data records, the decision-making process begins. Within the context, a suitable file read / write handle is checked. If a suitable file read / write handle exists, the data stream writing process begins; otherwise, the stream creation process begins. The specific steps of the stream creation process include:

[0051] Create an enhanced output file handle, denoted as the first object;

[0052] Setting the control switch of the first object to true will prevent the output of the tar header. That is, when the control switch of the first object is turned on, all content written to that object will be discarded.

[0053] The process of creating a tar stream is executed, which packages the first object with tar and generates a packaged output object, which is denoted as the second object.

[0054] The process of creating a gzip stream is executed. The second object is wrapped with a gzip object. The wrapped object replaces the wrapped output object obtained by wrapping the first object with tar. The second object's identifier is still used after the replacement.

[0055] The process of creating a tar file output stream is executed, and a tar output record is created in the replaced second object. The identifier of the second object is retained after the update.

[0056] Perform the process to disable the skipped output and turn off the control switch for the first object.

[0057] When entering the data processing stage of writing data to the output file handle, data is written to the output file handle. After completion, the decision chain is entered to determine file segmentation: the output byte count is checked sequentially to see if it reaches S, the output record count reaches N, and the data record time span reaches M minutes. If any of the conditions are met, the stream is closed. The specific steps include:

[0058] Perform the operation to close the tar file stream, close the second object and the first object, and generate a compressed fragment file of the tar file body;

[0059] Perform the operation to generate a tar.gz file header, obtain the number of bytes in the output stream, the number of records processed, the time range of the record content, and the MD5 value of the output stream from the first object, generate the target file name according to the preset requirements, and generate a gzip compressed fragment file containing tar file header information;

[0060] At this point, we have obtained a pair of gzip compressed fragment files consisting of the tar file header and the tar file body.

[0061] Determine if the file system needs to perform a merge operation, complete the stream closing process, and then jump to the data processing process to continue the loop. Specific steps include:

[0062] Determine the file system to which the output is written. If the file system to which the output is written is HDFS, continue to perform the operation of merging the gzip compressed fragments of the tar file header and the tar file body: call the concat method to merge the gzip compressed fragments of the generated tar file header into the gzip compressed fragments of the tar file body to form a tar.gz file;

[0063] Once the stream closing process is complete, the process jumps back to the data processing process to read the input stream and continues to execute the data processing process in a loop.

[0064] A file compression process adjustment system based on data streams includes: an input stream processing module, a stream creation module, a data writing and segmentation judgment module, and a stream closing and fragment processing module. The input stream processing module reads the input stream, determines whether file creation is needed, and executes the stream creation process when file creation is needed; otherwise, it performs a data stream writing operation based on the read input stream data. The stream creation module, when file creation is needed, creates an enhanced output file handle (denoted as a first object), sets its control switch to true to discard the tar header output, generates a second object by tar packaging the first object, then packages the second object with a gzip object while retaining its identifier, and creates and saves the tar output record within this object. Holding the identifier, the control switch of the first object is turned off, completing the stream creation process; the data writing and segmentation judgment module is used to complete the stream creation process, enter the data processing process to write data stream, write data to the output file handle, and after completion, enter the decision chain to perform file segmentation judgment. If the file segmentation condition is met, the stream closing process is entered; otherwise, the loop returns to the reading input stream step to continue processing; the stream closing and fragment processing module is used to enter the stream closing process, close the second object and the first object, and generate a tar file body compressed fragment; obtain information from the first object to generate a tar file header gzip compressed fragment, form a fragment pair, determine whether the file system needs to perform corresponding merging operations, complete the stream closing process, and jump to the data processing process to continue the loop.

[0065] The input stream processing module includes an input stream reading unit and a file creation requirement judgment unit. The input stream reading unit is used to read data records from the target data stream in real time. The file creation requirement judgment unit is used to check the available file read / write handles in the context based on the read data records: if an available handle exists, the data writing and segmentation judgment module is triggered; if no available handle exists, the stream creation module is triggered.

[0066] The stream creation module includes an enhanced output stream creation unit, a tar stream wrapping unit, a gzip stream wrapping unit, a tar output record creation unit, and a switch control unit. The enhanced output stream creation unit is used to create an enhanced output file handle, denoted as the first object. The tar stream wrapping unit is used to wrap the first object using the tar tool to generate a wrapped output object. The gzip stream wrapping unit uses the gzip object to wrap a second object, retaining the identifier of the second object. The tar output record creation unit is used to create a tar output record in the gzip-wrapped second object. The switch control unit is used to control the skip output control switch of the first object in stages. After creating an enhanced output file handle, the control switch of the first object is set to true, so that the tar header is not output. After the tar output record is created, the switch is set to off.

[0067] The data writing and segmentation judgment module includes a data stream writing unit and a segmentation condition judgment unit. The data stream writing unit is used to receive data records in the input stream and write data to the second object in real time. The segmentation condition judgment unit is used to sequentially check whether the number of output bytes reaches S, whether the number of output records reaches N, and whether the data record time span reaches M minutes. If any result meets the condition, the stream is closed. If none of the results meet the condition, the data processing process is executed in a loop and the input stream is read. Here, S represents the maximum number of bytes in the generated file, N represents the maximum number of records in the generated file, and M represents the maximum duration of the generated file.

[0068] The stream closing and fragment processing module includes an output stream closing unit, a compressed fragment generation unit, and a fragment merging unit. The output stream closing unit is used to perform the operation of closing the tar file stream, closing the second object and the first object, and generating a compressed fragment file of the tar file body. The compressed fragment generation unit is used to read statistical information from the first object, generate a target file name according to a preset naming rule, and generate a gzip compressed fragment of the tar file header to form a compressed fragment pair of tar file header and file body. The fragment merging unit is used to determine the file system to which the output is written. If the file system to which the output is written is HDFS, the operation of merging the gzip compressed fragment file pair of tar file header and tar file body is continued: the concat method is called to merge the gzip compressed fragment of the generated tar file header into the gzip compressed fragment of the tar file body to form a tar.gz file. When the stream closing process is completed, the process continues to jump to the process of reading the input stream in the data processing process and continues to loop the data processing process.

[0069] In this embodiment: the input stream reading unit of the input stream processing module first accesses the real-time data stream of the platform and continuously receives data records pushed by various business systems. Then, the file creation requirement judgment unit searches for available file read / write handles in the context. When no available handles are available, the stream creation module is triggered to enter the output object generation process.

[0070] The enhanced output stream creation unit of the stream creation module generates an enhanced output file handle (first object). This handle simultaneously starts the data statistics function, recording in real time the number of bytes written, the number of data records, and the time range of data reception. The tar stream packaging unit uses the tar tool to package the first object, generating an output object (second object) that can carry tar format data. The gzip stream packaging unit calls the gzip object to package the second object, and uses the identifier of the second object to ensure that the data written later is stored directly in a compressed format. The tar output record creation unit creates a tar output record in the packaged second object, improving the format structure of the tar file body. The switch control unit switches the "skip output control switch" of the first object from the initial off state to on, preventing the tar header information from being written at this time and only retaining the storage channel of the file body.

[0071] After the stream is created, the data writing process begins. The data stream writing unit of the data writing and segmentation judgment module writes the data records in the input stream to the second object one by one. The entire process only performs one write operation, and the written content is a data stream compressed with gzip to reduce hard disk storage consumption. After each batch of data is written, the segmentation condition judgment unit checks the current status in sequence: first, it checks whether the total number of bytes of data written has reached the preset maximum file size; then, it checks whether the cumulative number of data records has reached the preset maximum number of records; and finally, it checks whether the time span from the first write to the current time has reached the preset maximum generation time. If none of the three conditions are met, it returns to the input stream processing stage to continue receiving and writing the next batch of data. If any condition is met (such as the time span after a batch of data is written reaching the preset time), the stream closing module is triggered.

[0072] The output stream closing unit of the stream closing and fragment processing module closes the second and first objects, generating a compressed fragment containing the tar file body. Subsequently, the compressed fragment generation unit extracts information such as the number of bytes, the number of records, and the time range from the statistical function of the first object, combines it with the business type of the data to generate a target file name, and then generates a compressed fragment containing tar header information, forming a "tar header-file body" compressed fragment pair. The fragment merging unit calls the HDFS merging interface to merge the tar header compressed fragment with the file body compressed fragment to generate the final compressed file. After the stream closing process is completed, the process loop control module resets the state to the input stream reading state, guiding the continued reception of the next batch of data streams, realizing continuous loop processing of data compression.

[0073] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for adjusting the file compression process based on data streams, characterized in that: The method includes the following steps: Read the input stream, determine whether a file needs to be created, and execute the stream creation process if a file needs to be created. The execution flow creation process includes: creating an enhanced output file handle, denoted as the first object, and turning on the control switch of the first object; generating a second object by packaging the first object with tar, then packaging the second object with a gzip object and using its identifier, creating a tar output record in the object, and turning off the control switch of the first object; After the stream creation process is completed, the data stream writing process is performed. After the data stream writing process is completed, the decision chain is entered to determine file segmentation. When the file splitting condition is met, the stream closing process begins; The stream closing process includes: closing the second object and the first object, generating a compressed fragment of the tar file body; obtaining information from the first object to generate a gzip compressed fragment of the tar file header, forming a fragment pair, determining whether the file system needs to perform corresponding merging operations, completing the stream closing process, and jumping to read the input stream.

2. The method for adjusting the file compression process based on data streams according to claim 1, characterized in that: Read the input stream, determine if file creation is needed, and execute the stream creation process if file creation is needed; otherwise, perform a write operation based on the read input stream data. Specific steps include: Based on the read data records, the decision-making process is initiated. In the context, it is checked whether a suitable file read / write handle exists. If a suitable file read / write handle exists, the data stream writing process is initiated; otherwise, the stream creation process is initiated. The data stream writing process includes writing data to the output file handle. When entering the data processing stage of writing data stream, data is written to the output file handle. After completion, the decision chain is entered to perform file segmentation judgment: check whether the number of output bytes reaches S, whether the number of output records reaches N, and whether the data recording time span reaches M minutes. If any result meets the condition, the stream is closed. If none of the conditions are met, the data processing process is executed in a loop and the input stream is read. Here, S represents the maximum number of bytes in the generated file, N represents the maximum number of records in the generated file, and M represents the maximum duration of the generated file.

3. The method for adjusting the file compression process based on data streams according to claim 2, characterized in that: Based on the read data records, the decision-making process is initiated. In the context, it is checked whether a suitable file read / write handle exists. If a suitable file read / write handle exists, the process of writing data stream is initiated. Otherwise, proceed to the stream creation process, which includes the following steps: Create an enhanced output file handle, denoted as the first object; Setting the control switch of the first object to true will prevent the output of the tar header. That is, when the control switch of the first object is turned on, all content written to that object will be discarded. The process of creating a tar stream is executed, which packages the first object with tar and generates a packaged output object, which is denoted as the second object. The process of creating a gzip stream is executed. The second object is wrapped with a gzip object. The wrapped object replaces the wrapped output object obtained by wrapping the first object with tar. The second object's identifier is still used after the replacement. The process of creating a tar file output stream is executed, and a tar output record is created in the replaced second object. The identifier of the second object is retained after the update. Perform the process to disable the skipped output and turn off the control switch for the first object.

4. The method for adjusting the file compression process based on data stream according to claim 2, characterized in that: When entering the data processing process, the data stream is written to the output file handle. After completion, the decision chain is entered to perform file segmentation judgment: check whether the number of output bytes reaches S, whether the number of output records reaches N, and whether the data recording time span reaches M minutes. If any result satisfies the condition, the stream closure process is performed, and the specific steps include: Perform the operation to close the tar file stream, close the second object and the first object, and generate a compressed fragment file of the tar file body; Perform the operation to generate a tar.gz file header, obtain the number of bytes in the output stream, the number of records processed, the time range of the record content, and the MD5 value of the output stream from the first object, generate the target file name according to the preset requirements, and generate a gzip compressed fragment file containing tar file header information; At this point, we have obtained a pair of gzip compressed fragment files consisting of the tar file header and the tar file body.

5. The file compression process adjustment method based on data stream according to claim 4, characterized in that: Determine if the file system needs to perform a merge operation, complete the stream closing process, and then jump to the data processing process to continue the loop. Specific steps include: Determine the file system to which the output is written. If the file system to which the output is written is HDFS, continue to perform the operation of merging the gzip compressed fragments of the tar file header and the tar file body: call the concat method to merge the gzip compressed fragments of the generated tar file header into the gzip compressed fragments of the tar file body to form a tar.gz file; Once the stream closing process is complete, the process jumps back to the data processing process of reading the input stream and continues to execute the data processing process in a loop.

6. A file compression process adjustment system based on data streams, applied to the file compression process adjustment method based on data streams as described in any one of claims 1-5, characterized in that: The system includes: an input stream processing module, a stream creation module, a data writing and segmentation judgment module, and a stream closing and fragment processing module. The input stream processing module reads the input stream, determines whether file creation is needed, and executes the stream creation process when file creation is required; otherwise, it performs a data stream writing operation based on the read input stream data. The stream creation module, when file creation is needed, creates an enhanced output file handle, designated as the first object, sets its control switch to true to discard the tar header output, generates a second object by tar packaging the first object, then packages the second object with a gzip object while retaining its identifier, creates a tar output record in this object while maintaining the identifier, and closes the first object. The control switch of the object completes the stream creation process; the data writing and segmentation judgment module is used to complete the stream creation process, enter the data processing process to write data stream, write data to the output file handle, and after completion, enter the decision chain to perform file segmentation judgment. If the file segmentation condition is met, enter the stream closing process; otherwise, return to the reading input stream step to continue processing; the stream closing and fragment processing module is used to enter the stream closing process, close the second object and the first object, and generate tar file body compressed fragments; obtain information from the first object to generate tar file header gzip compressed fragments, form fragment pairs, determine whether the file system needs to perform corresponding merging operations, complete the stream closing process, jump to the data processing process to continue the loop.

7. The file compression process adjustment system based on data stream according to claim 6, characterized in that: The input stream processing module includes an input stream reading unit and a file creation requirement judgment unit. The input stream reading unit is used to read data records from the target data stream in real time. The file creation requirement judgment unit is used to check the available file read / write handles in the context based on the read data records. If an available handle exists, the data writing and segmentation judgment module is triggered. If no available handle is found, trigger the stream creation module.

8. The file compression process adjustment system based on data stream according to claim 7, characterized in that: The stream creation module includes an enhanced output stream creation unit, a tar stream wrapping unit, a gzip stream wrapping unit, a tar output record creation unit, and a switch control unit. The enhanced output stream creation unit is used to create an enhanced output file handle, denoted as the first object. The tar stream wrapping unit is used to wrap the first object using the tar tool to generate a wrapped output object. The gzip stream wrapping unit uses the gzip object to wrap a second object, retaining the identifier of the second object. The tar output record creation unit is used to create a tar output record in the gzip-wrapped second object. The switch control unit is used to control the skip output control switch of the first object in stages. After creating an enhanced output file handle, the control switch of the first object is set to true, so that the tar header is not output. After the tar output record is created, the switch is set to off.

9. A file compression process adjustment system based on data stream according to claim 8, characterized in that: The data writing and segmentation judgment module includes a data stream writing unit and a segmentation condition judgment unit. The data stream writing unit is used to receive data records in the input stream and write data to the second object in real time. The segmentation condition judgment unit is used to sequentially check whether the number of output bytes reaches S, whether the number of output records reaches N, and whether the data record time span reaches M minutes. If any result meets the condition, the stream is closed. If none of the results meet the condition, the data processing process is executed in a loop and the input stream is read. Here, S represents the maximum number of bytes in the generated file, N represents the maximum number of records in the generated file, and M represents the maximum duration of the generated file.

10. A file compression process adjustment system based on data stream according to claim 9, characterized in that: The stream closing and fragment processing module includes an output stream closing unit, a compressed fragment generation unit, and a fragment merging unit. The output stream closing unit is used to perform the operation of closing the tar file stream, closing the second object and the first object, and generating a compressed fragment file of the tar file body. The compressed fragment generation unit is used to read statistical information from the first object, generate a target file name according to a preset naming rule, and generate a gzip compressed fragment of the tar file header to form a compressed fragment pair of tar file header and file body. The fragment merging unit is used to determine the file system to which the output is written. If the file system to which the output is written is HDFS, the operation of merging the gzip compressed fragment file pair of tar file header and tar file body is continued: the concat method is called to merge the gzip compressed fragment of the generated tar file header into the gzip compressed fragment of the tar file body to form a tar.gz file. When the stream closing process is completed, the process continues to jump to the process of reading the input stream in the data processing process and continues to loop the data processing process.

Images