A method for continuous addressing high efficient fragmented reading of large files

Abstract

The application discloses a method for continuously addressing and efficiently reading large files in fragments, and particularly relates to the technical field of data processing, which comprises the following steps: step 1, opening a large file to be read and reading data in the large file; step 2, continuously reading data and submitting data callback to business processing according to pages; step 3, when the business processing is completed, continuing to acquire next page data according to the path of the large file; and step 4, repeating steps 2 and 3 until the reading and processing of the large file are completed. In the application, a continuous data block is used to store file content in the file, and a pre-calculated index table is used to locate the position of the data block. When reading data, the system directly finds the position of the target data block according to the index table and performs continuous reading operation, thereby avoiding repeated addressing and deep paging problems, improving the efficiency and performance of data reading, and reducing IO operation and resource overhead.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a method for efficient segmented reading of large files using continuous addressing. Background Technology

[0002] When processing large amounts of data, it is often necessary to use files for data transfer and parsing. Due to the limited computer memory, it is usually necessary to use a page-based reading method to gradually obtain the data in the file.

[0003] Traditional pagination methods suffer from several issues, including redundant addressing and deep pagination, leading to performance bottlenecks and reduced efficiency. Redundant addressing occurs because after reading one page of data, the location of the next page needs to be repositioned, resulting in frequent disk seek operations, increasing I / O overhead and latency. This redundant addressing not only wastes time but also places an additional burden on system resources, limiting data processing speed and efficiency. Deep pagination occurs when reading large files; if only one page is read at a time, multiple page flips are required to obtain the complete data. This leads to frequent disk I / O operations and data cache switching, increasing the complexity and latency of data access. Therefore, traditional pagination methods suffer from performance bottlenecks and inefficiency when processing large amounts of data.

[0004] To address this, the present invention proposes a method for efficiently reading large files in segments based on continuous addressing. Summary of the Invention

[0005] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a method for efficient, segmented reading of large files using continuous addressing. This method addresses the pain points of traditional segmented reading methods and offers an efficient and reliable large file reading solution, significantly improving data processing speed and efficiency. Simultaneously, this method also reduces system resource consumption and improves the overall system performance and stability, thereby resolving the problems mentioned in the background section.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for efficient segmented reading of large files using continuous addressing, comprising:

[0007] Step 1. Open the large file to be read and read the data from the large file;

[0008] Step 2. Continuously read data and submit data callbacks to the business processing page by page;

[0009] Step 3. After the business process is completed, continue to retrieve the next page of data based on the large file path;

[0010] Step 4. Repeat steps 2 and 3 until the large file has been read and processed.

[0011] Furthermore, methods for opening large files to be read include:

[0012] Determine the path of the large file to be read, wherein the path is determined by specifying a large file path string, the string containing the complete path of the large file;

[0013] A large file object is created using a specified large file path; the creation method uses the File class in Java.

[0014] After creating a large file object, check whether the large file actually exists;

[0015] If the large file actually exists, the system is allowed to read data from the large file;

[0016] If an exception occurs during the process of opening a large file, the system will log the exception and stop accessing the large file by alerting the user, in order to prevent potential exceptions from affecting the program. Exceptions include large files not existing or inaccessible.

[0017] Furthermore, the reasons for errors occurring during the opening of large files include:

[0018] When the specified large file path is incorrect or does not exist;

[0019] When a large file is already being used by another process, an attempt is made to open the large file in write mode;

[0020] When the file is corrupted or does not conform to the set format;

[0021] When opening large files, you may encounter issues such as insufficient disk space or network connection problems.

[0022] Furthermore, the content of continuously reading data and submitting page-by-page data callbacks to the business processing includes:

[0023] Continuously read data to obtain strings from large files;

[0024] The data block index table is calculated by reading a portion of the file header to obtain the total file size when a large file begins to be read. Based on the set data block size, the number of data blocks is calculated and an index table is created.

[0025] Using information from the index table, the system performs continuous addressing of large file strings. Based on the pre-calculated index table, it finds the starting position of each data block and uses file I / O operations to read the contents of the data block from the starting position.

[0026] The read data blocks are passed to the business processing module for processing, and the business processing module parses or transforms the data according to requirements.

[0027] Furthermore, the index table records the starting position or byte offset of each data block; the data block is a continuous byte stream or other data structure suitable for business processing.

[0028] Furthermore, the contents of the data block index table are calculated as follows:

[0029] If the size of each data block is a fixed value, then the number of data blocks is calculated as the total file size divided by the data block size.

[0030] For each data block, the starting position of the data block is calculated as data block index * data block size;

[0031] If the size of the data block is a variable value, use the markers or delimiters in the large file to determine the starting position of each data block; when reading the large file, find the marker position of the marker or delimiter, and record the marker position as the starting position of the data block. For each data block, record the size and starting position of the data block.

[0032] Furthermore, an index table is stored in the header of the large file to record the starting position or byte offset of each data block, and the size and format of the index table are determined. If the data blocks are generated according to a fixed interval, the file is iterated in a loop and the starting position of each data block is recorded.

[0033] Furthermore, after the business processing is completed, the system continues to read the next page of data based on the large file path. When the business processing module finishes processing a data block, it does not close the file, but keeps the large file at the current position and directly performs the next file reading operation.

[0034] In a second aspect, the present invention provides an electronic device, comprising: a processor and a memory, wherein the memory stores a computer program that can be called by the processor;

[0035] The processor executes the above-described method for efficient segmented reading of large files by calling a computer program stored in the memory.

[0036] Thirdly, the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the above-described method for efficient segmented reading of large files via continuous addressing.

[0037] The technical effects and advantages of this invention are as follows:

[0038] This invention uses contiguous data blocks to store file content and employs a pre-calculated index table to locate the data blocks. When reading data, the system directly finds the target data block based on the index table and performs continuous reading operations, avoiding the problems of repeated addressing and deep paging. This contiguous addressing-based approach significantly improves data reading efficiency and performance while reducing I / O operations and resource overhead. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the large file reading interaction in Example 1;

[0040] Figure 2 This is a flowchart of the large file reading process in Example 1;

[0041] Figure 3 This is a schematic diagram of an electronic device according to Embodiment 2;

[0042] Figure 4 This is a schematic diagram of a computer-readable storage medium according to Embodiment 3. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0044] Furthermore, the accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor methods and / or microcontroller methods.

[0045] It should be understood that although terms such as "first," "second," etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are used merely to distinguish one unit from another. For example, without departing from the scope of the exemplary embodiments, a first unit may be referred to as a second unit, and a similar second unit may be referred to as a first unit. The term "and / or" as used herein includes any and all combinations of one or more of the associated items listed.

[0046] Example 1

[0047] Please see Figure 1 As shown, this embodiment discloses a method for efficient segmented reading of large files using continuous addressing, including:

[0048] Step 1. Open the large file to be read and read the data from the large file:

[0049] Please see Figure 2 As shown, firstly, the system opens the large file to be read, using APIs related to large file operations, such as using Java's FileInputStream or RandomAccessFile classes to open the file.

[0050] Specifically, methods for opening large files to be read include:

[0051] 101. Determine the path of the large file to be read, wherein the path is determined by specifying a large file path string, the string containing the complete path of the large file;

[0052] 102. Create a large file object using the specified large file path; the creation method uses the File class in Java.

[0053] 103. After creating a large file object, check if the large file actually exists to ensure that the large file is found in the specified path;

[0054] 104. If the large file actually exists, the system is allowed to read data from the large file;

[0055] 105. If an exception occurs during the process of opening a large file, log the exception and stop accessing the large file by alerting the user to prevent potential exceptions from affecting the program; exceptions include large files not existing or inaccessible.

[0056] Furthermore, the reasons for errors occurring during the opening of large files include:

[0057] If the specified large file path is incorrect or does not exist, it means that the large file does not exist or the path is wrong;

[0058] If a large file is already being used by another process, attempting to open the large file in write mode indicates that the large file is being used by another process.

[0059] If the file is corrupted or does not conform to the set format, it means that the file is corrupted or the format is incorrect.

[0060] When opening large files, you may encounter errors such as insufficient disk space or network connection problems.

[0061] This step ensures that the system can successfully open the large file to be read and prepares it for subsequent reading operations by correctly opening the large file.

[0062] Step 2. Continuously read data and submit data callbacks to the business logic page by page:

[0063] Specifically, the content of continuously reading data and submitting data callbacks to the business processing page by page includes:

[0064] 201. Continuously read data to obtain large file descriptors;

[0065] 202. Calculate the data block index table. When reading a large file, obtain the total file size by reading part of the file header, calculate the number of data blocks according to the set data block size, and create an index table. The index table records the starting position or byte offset of each data block. The data block is a continuous byte stream or other data structure suitable for business processing.

[0066] 203. Using the information in the index table, perform continuous addressing through large file strings, find the starting position of each data block according to the pre-calculated index table, and use file I / O operations to read the contents of the data block from the starting position;

[0067] 204. The read data block is passed to the business processing module for processing. The business processing module performs parsing, transformation or other preprocessing operations on the data according to the requirements.

[0068] Furthermore, the contents of the data block index table are calculated as follows:

[0069] If the size of each data block is a fixed value, then calculate the number of data blocks: total file size / data block size;

[0070] For each data block, the starting position of the data block is calculated as data block index * data block size;

[0071] If the size of the data block is a variable value, use the markers or delimiters in the large file to determine the starting position of each data block; when reading the large file, find the marker position of the marker or delimiter, and record the marker position as the starting position of the data block. For each data block, record the size and starting position of the data block.

[0072] It should be noted that an index table is stored in the header of a large file to record the starting position or byte offset of each data block, and the size and format of the index table are determined so that the information can be read accurately. If the data blocks are generated according to a fixed interval, the file is iterated in a loop and the starting position of each data block is recorded.

[0073] It should be noted that after opening a large file, the system continuously reads the data in the large file and sends the data back to the business processing module according to the page or other set data block size; the system determines the location of each data block through a pre-calculated index table, thereby realizing continuous addressing and reading.

[0074] Step 3. After the business process is completed, continue to retrieve the next page of data based on the large file path;

[0075] It should be noted that after the business processing is completed, the system continues to read the next page of data based on the large file path. When the business processing module finishes processing a data block, it does not close the file, but keeps the large file at the current position and directly performs the next file reading operation. This allows the system to continue reading the next page of data based on the original address of the large file, so as to maintain the continuity of the business processing module's reading and avoid repeated addressing and data cache switching.

[0076] 4. Repeat steps 2 and 3 until the large file has been read and processed:

[0077] The system repeats steps 2 and 3, continuously reading data blocks and sending them back to the business processing module until the entire large file is read and processed.

[0078] This invention uses contiguous data blocks to store the contents of large files and employs a pre-calculated index table to locate the data blocks. When reading data, the system directly finds the target data block based on the index table and performs continuous read operations, avoiding the problems of repeated addressing and deep pagination. This contiguous addressing-based approach significantly improves the efficiency and performance of data reading while reducing I / O operations and resource overhead.

[0079] Example 2

[0080] Please see Figure 3 As shown, this embodiment provides an electronic device, including: a processor and a memory, wherein the memory stores a computer program that can be called by the processor;

[0081] The processor executes a method for efficient segmented reading of large files using continuous addressing, as described in Embodiment 1, by calling a computer program stored in the memory.

[0082] Example 3

[0083] Please see Figure 4 As shown, this embodiment provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform a method for efficient segmented reading of large files using continuous addressing, as described in Embodiment 1.

[0084] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired network. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0085] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0086] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0087] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only one method, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0088] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0089] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0090] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0091] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0092] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for efficient segmented reading of large files using continuous addressing, characterized in that, include: Step 1. Open the large file to be read and read the data from the large file; Step 2. Continuously read data and submit data callbacks to the business processing page by page, including: Continuously read data to obtain strings from large files; The data block index table is calculated by reading a portion of the file header to obtain the total file size when a large file begins to be read. Based on the set data block size, the number of data blocks is calculated and an index table is created. Using information from the index table, continuous addressing is performed on large file strings. According to the pre-calculated index table, the starting position or byte offset of each data block is recorded. The data block is a continuous byte stream data structure. The starting position of each data block is found, and continuous reading operations are performed to avoid repeated addressing and deep paging. File I / O operations are used to read the contents of the data block from the starting position. The read data blocks are passed to the business processing module for processing. The business processing module parses or transforms the data according to requirements. Step 3. After the business process is completed, continue to retrieve the next page of data based on the large file path; After the business processing is completed, continue reading the next page of data based on the large file path. When the business processing module finishes processing a data block, do not close the file, keep the large file at the current position, and directly perform the next file reading operation. Step 4. Repeat steps 2 and 3 until the large file has been read and processed.

2. The method for efficient segmented reading of large files using continuous addressing according to claim 1, characterized in that, Methods for opening large files to be read include: Determine the path of the large file to be read, wherein the path is determined by specifying a large file path string, the string containing the complete path of the large file; A large file object is created using a specified large file path; the creation method uses the File class in Java. After creating a large file object, check whether the large file actually exists; If the large file actually exists, the system is allowed to read data from the large file; If an exception occurs during the process of opening a large file, log it and stop accessing the large file by notifying the user; the exception may include the large file not existing or being inaccessible.

3. The method for efficient segmented reading of large files using continuous addressing according to claim 2, characterized in that, Reasons for errors occurring during the opening of large files include: When the specified large file path is incorrect or does not exist; When a large file is already being used by another process, an attempt is made to open the large file in write mode; When the file is corrupted or does not conform to the set format; When opening large files, you may encounter issues such as insufficient disk space or network connection problems.

4. The method for efficient segmented reading of large files using continuous addressing according to claim 1, characterized in that, The contents of the computed data block index table include: If the size of each data block is a fixed value, then the number of data blocks is calculated as the total file size divided by the data block size. For each data block, the starting position of the data block is calculated as data block index * data block size; If the size of the data block is a variable value, use the markers or delimiters in the large file to determine the starting position of each data block; when reading the large file, find the marker position of the marker or delimiter, and record the marker position as the starting position of the data block. For each data block, record the size and starting position of the data block.

5. The method for efficient segmented reading of large files using continuous addressing according to claim 4, characterized in that, An index table is stored at the beginning of a large file to record the starting position or byte offset of each data block. The size and format of the index table are determined. If the data blocks are generated at fixed intervals, the file is iterated through in a loop, and the starting position of each data block is recorded.

6. An electronic device, characterized in that, include: A processor and a memory, wherein the memory stores a computer program that can be called by the processor; The processor executes a method for efficient segmented reading of large files using continuous addressing, as described in any one of claims 1-5, by calling a computer program stored in the memory.

7. A computer-readable storage medium, characterized in that, The system stores instructions that, when executed on a computer, cause the computer to perform a method for efficient segmented reading of large files using continuous addressing as described in any one of claims 1-5.

Images