Pure data stream multi-level nested file downloading method

Abstract

The invention discloses a pure data stream multi-level nested file downloading method, which is based on memory stream direct processing and specifically comprises the following steps of: acquiring a primary data ID (Identity) required to be downloaded from a DB (Data Bridge) database through a parent ID and a file path field, performing data preprocessing, logically acquiring all lower-level folders and files of the primary data, and circularly judging the folders and the files; if the path is the folder, adding the path to the ZIP output stream; if the path is a file, processing the file into a file stream, and adding the path and the file stream to a ZIP output stream; and forming a complete ZIP output stream, and downloading a folder compression package. According to the scheme, the problem of downloading the multi-level nested folder data is solved, the requirement of management personnel for downloading electronic archives is met, and the system does not need to create a temporary folder for downloading after nesting, so that the memory space is effectively saved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method for downloading multi-level nested files in pure data streams. Background Technology

[0002] In the agricultural sector, rural data management largely relies on paper documents. As it gradually transitions to an information-based model, electronic archives are created. To facilitate easier use and management, this often involves nested downloads of multi-level folders (containing files). Common data downloads process multiple sets of business data, which cannot support the needs of nested folders and fail to address the pain point of users accessing complete electronic archive data on web systems. Furthermore, unlike common list data downloads, there is no mature solution for downloading multi-level nested folders (containing files). Summary of the Invention

[0003] To address the aforementioned technical problems, this invention provides a pure data stream multi-level nested file download method, implemented based on path fields stored in a database and the ZipOutputStream class in the Java standard library. Through the ZipOutputStream class in the Java standard library and pre-processing, this method can solve the data download problem of nested folders without generating temporary folders that consume memory space.

[0004] This invention is achieved using the following technical solution: A method for downloading multi-level nested files using pure data streams, based on direct memory stream processing, specifically includes the following steps: Step S1: Obtain the ID of the first-level data to be downloaded from the DB database using the parent ID and file path fields, and perform data preprocessing. Logically obtain all subfolders and files of the first-level data, and loop through and judge the folders and files. Step S2: If it is a folder, add the path to the ZIP output stream; if it is a file, process the file into a file stream, and then add the path and the file stream to the ZIP output stream. Step S3: Create a complete ZIP output stream and download the folder archive.

[0005] Specifically, step S1 includes: Step S11: Obtain the target folder and its entire set of sub-contents data, and receive the target folder ID from the front end; Step S12: Using the target folder ID as the root node, traverse all descendant nodes using recursive SQL or a program recursive algorithm; Step S13: Encapsulate the query results into a List<?> This set is used for subsequent processing.

[0006] Specifically, step S1 further includes an initialization step for the ZIP compressed output stream, which is as follows: Get the HTTP response object; Set response headers; Create a ZipOutputStream instance.

[0007] Specifically, the loop judgment in step S1 includes: traversing the data set, writing each item to the ZIP stream, and processing the List.<?> Each element in the collection is processed.

[0008] Specifically, the folder processing in step S2 includes: if it is a folder, generating the corresponding ZIP entry name according to its hierarchical path, and ending with / to indicate the directory, creating a ZipEntry object and putting it into the output stream.

[0009] Specifically, the file processing in step S2 includes: if it is a file, generating a complete path entry name and creating a ZipEntry, reading the input stream through the file path, reading the file content in chunks and writing it to a ZipOutputStream instance to avoid memory overflow.

[0010] Specifically, after step S3 completes the compression, it also includes a resource release step, which specifically includes: Call this function after all entries have been processed. Close the ZipOutputStream instance and the response stream to ensure complete data output; Record the operation log and notify the frontend that the download is complete.

[0011] Specifically, it also includes an exception handling mechanism: capturing IO exceptions during file stream reading or writing, promptly closing resources and returning error information to prevent resource leaks.

[0012] The beneficial effects of this invention are as follows: Efficiently implement multi-level nested folder downloads: By designing a reasonable data table structure (including key fields such as parent ID, subject type, and file path) and combining it with the ZipOutputStream class in the Java standard library, a unified streaming process for multi-level folder structures and their files stored in the database is achieved. Hierarchical ZIP archives can be directly constructed without rebuilding the actual directory structure or generating temporary folders on the server side, solving the problem of traditional methods being unable to handle batch downloads of nested folders.

[0013] Low memory usage and optimized system resources: By adopting a streaming processing mechanism of "reading, compressing, and outputting simultaneously", the large amount of memory consumption and disk I / O overhead caused by the need to create a temporary directory before compression in traditional solutions is avoided. This effectively reduces system resource usage and improves the stability and scalability of the download process.

[0014] High processing efficiency and fast response speed: The recursive query and preprocessing mechanism based on the path field and parent ID can quickly organize the file hierarchy relationship. Combined with real-time encapsulation of Zip compressed stream, it greatly shortens the data packaging time and improves the overall download efficiency, which is especially suitable for high-frequency access scenarios of large-scale electronic archives.

[0015] High compatibility and versatility: This method does not depend on specific databases or third-party tools, but only uses the core classes of the Java standard library. It has good platform portability and system compatibility, and is easy to integrate into various web information systems.

[0016] Promoting the digitalization of agricultural management: In response to the current situation where rural data management still largely relies on paper documents, this invention provides a convenient and reliable way to obtain electronic archives, which helps to break down information silos and improve grassroots governance capabilities.

[0017] Enhancing user experience and service satisfaction: Users can download complete hierarchical electronic documents with a single click through the web system. The operation is simple and intuitive, greatly enhancing the system's usability and service responsiveness, and meeting the management personnel's needs for the integrity and traceability of the archives. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0019] Figure 1 This is a flowchart of a pure data stream multi-level nested file download method according to an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0021] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0022] The following is in conjunction with the appendix Figure 1 The following describes some embodiments of the present invention in detail. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0023] This invention proposes a method for downloading multi-level nested files using pure data streams. In a preferred embodiment, the method directly processes memory streams and specifically includes the following steps: Step S1: Obtain the ID of the first-level data to be downloaded from the DB database using the parent ID and file path fields, and perform data preprocessing. Logically obtain all subfolders and files of the first-level data, and loop through and judge the folders and files. Step S2: If it is a folder, add the path to the ZIP output stream; if it is a file, process the file into a file stream, and then add the path and the file stream to the ZIP output stream. Step S3: Create a complete ZIP output stream and download the folder archive.

[0024] In this embodiment, the key fields for data storage design are: parent ID, subject type (file or folder), and file path. These fields can be improved based on actual business needs. The subject type is mainly used to distinguish whether the subject at this level is a file or a folder, and the file path is used to facilitate the hierarchical conversion to zip compressed output stream.

[0025] The zip compression output stream in the Java standard library, namely the ZipOutputStream class, is accessed through: ZipOutputStreamzos=newZipOutputStream(response.getOutputStream()); Create a zip archive instance object, wrapping all files and folders into a complete zip archive output stream for downloading, such as... Figure 1 As shown, it includes the following steps: S1: Process all data that meets the conditions from the DB database into a data set using the parent ID and file path fields, and proceed to the next step of filtering; S2: The General Body Type field adds folders and files to the zip output stream; S3: Download the folder as a zip archive.

[0026] In one specific embodiment, the present invention fundamentally solves the problems of difficult downloading and high resource consumption of multi-level nested folders through the synergistic effect of "specific data structure design + streaming compression output mechanism". Among them, the data structure design (parent ID, subject type, file path) is the foundation for realizing hierarchical recognition and ordered traversal; the streaming processing mechanism of ZipOutputStream is the core means to achieve zero temporary files and low memory usage.

[0027] For detailed implementation steps, please refer to the appendix. Figure 1 (Flowchart of multi-level folder download): Step 1: Obtain the target folder and its entire subfolder data set: (1) Receive the target folder ID passed from the front end; (2) Using this ID as the root node, traverse all descendant nodes using recursive SQL or program recursive algorithm; (3) Encapsulate the query results into a List<?> This set is used for subsequent processing.

[0028] Step 2: Initialize the ZIP compressed output stream: (1) Obtain the HTTP response object HttpServletResponse; (2) Set the response header; (1) Create a ZipOutputStream instance.

[0029] Step 3: Traverse the data collection, writing each item to a ZIP stream, for the List<?> Each element in the process is processed: (1) If it is a folder (type=="folder"): Generate the corresponding ZIP entry name based on its hierarchical path (e.g., "first-level directory / second-level directory / "). Note that the directory is indicated by a " / ". Create a ZipEntry object and put it into the output stream.

[0030] (2) If it is a file (type=="file"): Generate a complete path entry name (e.g., "first-level directory / second-level directory / report.pdf"); Create ZipEntry; Read input streams (such as FileInputStream) from file paths; The file content is read in chunks and written to a ZipOutputStream to avoid memory overflow.

[0031] Step 4: Complete compression and release resources: (1) Call this function after all entries have been processed; (2) Close ZipOutputStream and response stream to ensure complete data output; (3) Record the operation log and notify the front end that the download is complete.

[0032] The pure data stream multi-level nested file download method proposed in this invention has the following characteristics: No temporary files required: The entire process is based on direct memory stream processing, without relying on the creation of temporary directory structures on the server's local disk, thus avoiding the overhead of generating and cleaning up temporary files; Streaming processing ensures performance: By leveraging the compression-on-output feature of ZipOutputStream, "zero intermediate storage" is achieved, making it suitable for large files and deep-level scenarios; Path consistency guarantee: The ZIP entry path is dynamically constructed using the file_path and hierarchical relationship in the database to ensure that the directory structure inside the compressed package is consistent with the original logic; Exception handling mechanism: Capture IO exceptions during file stream reading or writing, close resources promptly and return error information to prevent resource leaks.

[0033] Taking the electronic archive system of a county agricultural bureau as an example, a certain project contains a three-level directory structure: Project Archives / ├──Conference Materials / │├── Minutes of the First Meeting in 2024.docx │└──Sign-in Sheet.xlsx ├──Inspection of materials / │└──Acceptance Report.pdf └──On-site photos / ├──IMG_001.jpg └──IMG_002.jpg When a user selects the "Project Archives" folder and clicks "Download," the system executes the above process, ultimately generating a compressed file named "Project Archives.zip." The internal directory structure is fully preserved, eliminating the need for manual organization after decompression and greatly improving ease of use.

[0034] The technologies used in this embodiment include: (1) The Java standard library java.util.zip.ZipOutputStream is a built-in class in the JDK and requires no additional dependencies; (2) Database queries are based on standard SQL and are compatible with mainstream relational databases; (3) The file stream processing method conforms to the IO programming specification and its stability and efficiency have been verified in the electronic file management function of the rural land fragmentation problem management system.

[0035] For the foregoing embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application.

[0036] The above embodiments describe the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Modifications and variations made by those skilled in the art without departing from the spirit and scope of the invention should be within the protection scope of the appended claims.

Claims

1. A method for downloading multi-level nested files using pure data streams, characterized in that, The process based on direct memory stream processing includes the following steps: Step S1: Obtain the ID of the first-level data to be downloaded from the DB database using the parent ID and file path fields, and perform data preprocessing. Logically obtain all subfolders and files of the first-level data, and loop through and judge the folders and files. Step S2: If it is a folder, add the path to the ZIP output stream; if it is a file, process the file into a file stream, and then add the path and the file stream to the ZIP output stream. Step S3: Create a complete ZIP output stream and download the folder archive.

2. The method for downloading multi-level nested files using a pure data stream as described in claim 1, characterized in that, Step S1 specifically includes: Step S11: Obtain the target folder and its entire set of sub-contents data, and receive the target folder ID from the front end; Step S12: Using the target folder ID as the root node, traverse all descendant nodes using recursive SQL or a program recursive algorithm; Step S13: Encapsulate the query results into a List<?> This set is used for subsequent processing.

3. The method for downloading multi-level nested files using a pure data stream as described in claim 2, characterized in that, Step S1 further includes an initialization step for the ZIP compressed output stream, specifically: Get the HTTP response object; Set response headers; Create a ZipOutputStream instance.

4. The method for downloading multi-level nested files using a pure data stream as described in claim 3, characterized in that, The loop judgment in step S1 specifically includes: traversing the data set, writing each item to a ZIP stream, and processing the List.<?> Each element in the collection is processed.

5. The method for downloading multi-level nested files using a pure data stream as described in claim 4, characterized in that, The folder processing in step S2 specifically includes: if it is a folder, generating the corresponding ZIP entry name according to its hierarchical path, and ending with / to indicate the directory, creating a ZipEntry object and putting it into the output stream.

6. The method for downloading multi-level nested files using a pure data stream as described in claim 5, characterized in that, The file processing in step S2 specifically includes: if it is a file, generating a complete path entry name and creating a ZipEntry, reading the input stream through the file path, reading the file content in chunks and writing it to a ZipOutputStream instance to avoid memory overflow.

7. The method for downloading multi-level nested files using a pure data stream as described in claim 6, characterized in that, After the compression is completed in step S3, a resource release step is also included, which specifically includes: Call this function after all entries have been processed. Close the ZipOutputStream instance and the response stream to ensure complete data output; Record the operation log and notify the frontend that the download is complete.

8. The method for downloading multi-level nested files using a pure data stream as described in claim 1, characterized in that, It also includes an exception handling mechanism: capturing IO exceptions during file stream reading or writing, closing resources in a timely manner and returning error information to prevent resource leaks.

Images