CSV file reading method and device, program product and storage medium
By dividing the CSV file into row blocks and using dynamic row block size and error threshold judgment, the problem of low reading efficiency of large CSV files is solved, and efficient and accurate data reading is achieved.
Patent Information
- Application Number
- CN202411325440.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-23
- Publication Date
- 2024-12-27
AI Technical Summary
The reading efficiency of large CSV files is low. The traditional line-by-line reading method is inefficient when processing large-scale CSV files, and consumes a lot of time and system resources.
By dividing the CSV file into preset size row blocks for reading, avoiding separate judgments for each row, reducing the number of judgments and repeated operations, dynamic adjustment of row block size and error threshold judgment are used to ensure data integrity and efficiency.
It significantly improves the efficiency of data reading, reduces the number of judgments, optimizes the utilization of system resources, and ensures the accuracy and completeness of data.
Smart Images
Figure CN119201858A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, program product, and storage medium for reading CSV files. Background Technology
[0002] Comma-Separated Values (CSV) files are files that store data records in plain text format, separated by newline characters. CSV files are frequently used in this field to store data records with the same fields. For example, in new energy vehicle projects, the vehicle bus continuously sends data records, and the received data records are stored in a large CSV file. Typically, CSV files are read line by line, but this method is very inefficient for large CSV files. This presents a technical problem of low reading efficiency for large CSV files. Summary of the Invention
[0003] This invention provides a method, apparatus, program product, and storage medium for reading CSV files, solving the technical problem of low reading efficiency for large CSV files.
[0004] In a first aspect, the present invention provides a method for reading a CSV file, the method comprising: S10, obtaining the starting position and size of the row block to be read, wherein the row block size is the number of rows of the CSV file read each time; S20, reading the CSV file based on the starting position and size of the row block to complete the reading of the data of the current row block; S30, determining whether the number of valid rows in the current row block is greater than a preset row count error threshold; S40, if the number of valid rows is greater than the row count error threshold, returning to execute the above steps S10 to S30 to read the data of the next row block; S50, if the number of valid rows is less than or equal to the row count error threshold, ending the reading of the CSV file.
[0005] In some embodiments, the starting position of the row block includes the row number of the CSV file. The step of reading the CSV file based on the starting position of the row block and the row block size to complete the data reading of the current row block includes: reading the CSV file line by line starting from the row corresponding to the starting position of the row block; after reading the data of the row block size, obtaining the row number of the last row of data; if the difference between the row number and the row number corresponding to the starting position of the row block is equal to the value of the row block size minus 1, then it is determined that the data reading of the current row block has been completed.
[0006] In some embodiments, before determining whether the number of valid rows in the current row block is greater than a preset row number error threshold, the method further includes: removing invalid rows from the current row block to obtain valid rows in the current row block, wherein invalid rows include at least one of blank rows, garbled rows, and rows with non-preset format; and determining the number of valid rows.
[0007] In some embodiments, the row number error threshold ranges from 1 / 5 to 1 / 2 of the row block size.
[0008] In some embodiments, before obtaining the starting position and size of the row block to be read, the method further includes: obtaining the total number of rows in the CSV file, the memory space size, and the expected reading time; determining the minimum number of blocks based on the memory space size, and determining the maximum number of blocks based on the expected reading time; dividing the total number of rows by the minimum number of blocks to obtain the maximum row block size, and dividing the total number of rows by the maximum number of blocks to obtain the minimum row block size; the value of the row block size is selected between the maximum row block size and the minimum row block size.
[0009] In some embodiments, the step of obtaining the total number of rows in a CSV file includes: obtaining the file size of the CSV file; obtaining the data volume of a preset number of rows in the CSV file; and multiplying the ratio of the file size to the data volume by the preset number of rows to obtain the total number of rows in the CSV file.
[0010] In some embodiments, when reading a row block for the first time, the row block size is set to 1 / 2 of the total number of rows; before returning to the step of executing the above steps S10 to S30 to read the data of the next row block, the method further includes: setting the row block size of the next row block to 1 / 2 of the row block size of the current row block, until the row block size of the next row block is less than the minimum row block size; when the row block size of the next row block is less than the minimum row block size, the row block size of the next row block is modified to the minimum row block size.
[0011] Secondly, the present invention provides a CSV file reading device, the device comprising: an acquisition module, configured to acquire the starting position and size of the row block to be read, wherein the row block size is the number of rows of the CSV file read each time; a reading module, configured to read the CSV file based on the starting position and size of the row block, and complete the reading of the data of the current row block; and a judgment module, configured to judge whether the number of valid rows in the current row block is greater than a preset row count error threshold, and if the number of valid rows is greater than the row count error threshold, then read the data of the next row block.
[0012] Thirdly, the present invention provides a computer program product, including the steps of a computer program for implementing a CSV file reading method according to any of the above aspects.
[0013] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the CSV file reading method of any of the above aspects.
[0014] This invention provides a method, apparatus, program product, and storage medium for reading CSV files. The method includes: S10, obtaining the starting position and block size of the desired row block to be read, wherein the block size is the number of rows to be read in each CSV file; S20, reading the CSV file based on the starting position and block size to complete the reading of the current row block; S30, determining whether the number of valid rows in the current row block is greater than a preset row count error threshold; S40, if the number of valid rows is greater than the row count error threshold, returning to steps S10-S30 to read the data of the next row block; S50, if the number of valid rows is less than or equal to the row count error threshold, ending the reading of the CSV file. By dividing the file into row blocks of a preset size for reading, individual judgment for each row is avoided, reducing the number of judgments and repetitive operations, and improving the efficiency of data reading. Attached Figure Description
[0015] The invention will now be described in more detail with reference to embodiments and the accompanying drawings:
[0016] Figure 1 This is a flowchart illustrating a method for reading a CSV file provided in an embodiment of this application;
[0017] Figure 2 This is a schematic diagram of the structure of a CSV file reading system provided in an embodiment of this application;
[0018] Figure 3 This is a flowchart illustrating a method for reading a CSV file provided in an embodiment of this application.
[0019] In the accompanying drawings, the same parts are referred to by the same reference numerals, and the drawings are not drawn to scale. Detailed Implementation
[0020] To enable those skilled in the art to better understand the present invention and to fully understand and implement the process of how the present invention uses technical means to solve technical problems and achieve corresponding technical effects, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. The embodiments of the present invention and the various features therein can be combined with each other without conflict, and the resulting technical solutions are all within the protection scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.
[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0022] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.
[0023] Comma-Separated Values (CSV) files are files that store data records in plain text format, separated by newline characters. CSV files are frequently used in this field to store data records with the same fields. For example, in new energy vehicle projects, the vehicle bus continuously sends data records, and the received data records are stored in a large CSV file. Typically, CSV files are read line by line, but this method is very inefficient for large CSV files. This presents a technical problem of low reading efficiency for large CSV files.
[0024] The technical solution of this application will be described below with reference to specific embodiments.
[0025] Example 1
[0026] Figure 1 This is a flowchart illustrating a method for reading a CSV file provided in an embodiment of this application, as shown below. Figure 1As shown in the technical solution of this embodiment, a method for reading a CSV file is provided. The method includes: S10, obtaining the starting position and size of the row block to be read, wherein the row block size is the number of rows of the CSV file read each time; S20, reading the CSV file based on the starting position and size of the row block to complete the data reading of the current row block; S30, determining whether the number of valid rows in the current row block is greater than a preset row count error threshold; S40, if the number of valid rows is greater than the row count error threshold, returning to execute the above steps S10 to S30 to read the data of the next row block; S50, if the number of valid rows is less than or equal to the row count error threshold, ending the reading of the CSV file.
[0027] Valid and invalid rows constitute a CSV file. The number of valid rows is the total number of valid rows. When reading a CSV file, this application only needs to obtain the valid row records. That is, this application reads and counts the valid rows, while invalid rows, such as blank rows, garbled rows, and non-record data rows that do not conform to the format, are not counted. The number of valid rows in the row block is counted, and invalid rows are excluded.
[0028] This embodiment addresses the technical problem of efficiently reading CSV files. Traditional line-by-line reading and judgment methods are inefficient when processing large-scale CSV files because frequent line-by-line operations and judgments consume significant time and system resources. The technical solution of this embodiment avoids individual judgments on each line by dividing the file into line blocks of a preset size, reducing the number of judgments and repetitive operations. For example, for a CSV file with millions of rows of data, judging each line once would result in millions of judgments. However, by using line block reading, assuming a line block size of 10,000 rows, the number of judgments can be reduced to one ten-thousandth of the original, and the total time for reading the CSV file will be significantly shortened. Similarly, for a CSV file with millions of rows of data, judging each line once would result in millions of judgments. However, by using line block reading, assuming a line block size of 1,000 rows, the number of judgments can be reduced to one-thousandth of the original. Changing the line block size can further reduce the number of judgments. Simultaneously, this block reading method can better utilize the system's caching and parallel processing capabilities, improving data reading efficiency.
[0029] Example 2
[0030] Based on the above embodiments, the row block start position includes the row number of the CSV file. The steps of reading the CSV file based on the row block start position and the row block size to complete the data reading of the current row block include: starting from the row corresponding to the row block start position, reading the CSV file line by line; after reading the data of the row block size, obtaining the row number of the last row of data; if the difference between the row number and the row number corresponding to the row block start position is equal to the row block size minus 1, then it is determined that the data reading of the current row block has been completed.
[0031] This embodiment addresses the technical problem of accurately determining whether a desired row block has been read completely. In reading CSV files, accurately determining when a row block has been read is crucial for ensuring data integrity and accuracy. Inaccurate determination may lead to missing data or duplicate readings. Assuming the row block size is 100, then subtracting 1 from the row block size gives 99. In one implementation, the difference between the current row number 99 and the row number 0 corresponding to the start of the row block is also equal to 99, meaning the difference between the current row number and the row number corresponding to the start of the row block is equal to the row block size minus 1, thus determining that the current row block has been read completely. In another implementation, the difference between the current row number 199 and the row number 100 corresponding to the start of the row block is also equal to 99, meaning the difference between the current row number and the row number corresponding to the start of the row block is equal to the row block size minus 1, thus determining that the current row block has been read completely. This embodiment provides a reliable determination method by calculating the difference between the row number of the last row and the starting row number and comparing it with the row block size. This precise judgment method can effectively avoid reading errors caused by calculation errors or fuzzy judgments, ensuring that the reading of each row block is accurate.
[0032] Example 3
[0033] Based on the above embodiments, before determining whether the number of valid rows in the current row block is greater than a preset row number error threshold, the method further includes: removing invalid rows from the current row block to obtain valid rows in the current row block, wherein invalid rows include at least one of blank rows, garbled rows, and non-preset format rows; and determining the number of valid rows.
[0034] This embodiment addresses the technical problem of extracting valid rows. When processing CSV files, the presence of invalid rows can interfere with determining whether the CSV file has been completely read. This embodiment's solution predefines characteristics of invalid rows, such as blank lines, garbled lines, and lines not in a preset format, to accurately filter and remove rows when reading row blocks. When the number of valid rows is below a threshold, it indicates that there are a large number of invalid rows in the row block, and this row block represents the end of the CSV file. When reading a CSV file, this embodiment's solution requires obtaining the valid row record data. Specifically, it reads and counts the valid rows, while invalid rows, such as blank lines, garbled lines, and non-record data lines that do not conform to the format, are not counted. The number of valid rows in the row block is counted, and invalid rows are excluded. By removing blank lines, garbled lines, and lines not in a preset format, this embodiment can identify whether the current row block is the end of the CSV file, thus achieving the technical effect of accurately identifying the end of the CSV file.
[0035] Example 4
[0036] Based on the above embodiments, the range of the row number error threshold is 1 / 5 to 1 / 2 of the row block size.
[0037] Determining an appropriate error threshold is crucial for accurately judging the file's reading status during CSV file reading. If the threshold is set too high, a small number of invalid rows or special cases may lead to false positives indicating the file has been read completely, resulting in data loss. Conversely, if the threshold is set too low, a large number of invalid rows may be overlooked, wasting reading time and reducing efficiency. The technical solution in this embodiment sets the error threshold within the range of 1 / 5 to 1 / 2 of the row block size, fully considering various possible scenarios within the file.
[0038] In one implementation, for a block size of 1000 rows, the row count error threshold is set to half the block size. For example, setting the row count error threshold to 500 rows ensures that the next block is read only when the number of valid rows exceeds 500; otherwise, reading the CSV file stops. This guarantees that the read CSV file contains a large amount of valid text, achieving the technical effect of ensuring data quality while reading the CSV file. It avoids misjudging the file as finished reading due to a small number of invalid rows or special cases, thus preventing data loss. Furthermore, for cases where test data is affected by noise and generates a large number of invalid rows, the technical solution of this implementation can quickly detect and terminate the reading of the CSV file, achieving the technical effect of identifying and removing low-quality test data.
[0039] In one implementation, for a row block size of 1000 rows, the row count error threshold is set to 1 / 5 of the row block size. For example, setting the row count error threshold to 200 rows means that the next row block is read only when the number of valid rows in the current row block is greater than 200, thus allowing as much CSV data as possible to be read. If the data in a CSV file is determined to be important, setting the row count error threshold to 1 / 5 of the row block size achieves the technical effect of reading as complete CSV data as possible within a limited time, avoiding the neglect of a large number of invalid rows in the file, which would lead to wasted reading time and reduced efficiency.
[0040] For a block size of 1000 rows, the technical solution of this embodiment sets the error threshold in the range of 1 / 5 to 1 / 2 of the block size, that is, in the range of 200 to 500. Such a threshold setting can better adapt to the characteristics of different files and possible abnormal situations while ensuring the accuracy of judgment. It can determine whether the CSV file has been read completely, thus improving the efficiency and accuracy of reading.
[0041] Example 5
[0042] Based on the above embodiments, before obtaining the starting position and size of the row block to be read, the method further includes: obtaining the total number of rows in the CSV file, the memory space size, and the expected reading time; determining the minimum number of blocks based on the memory space size, and determining the maximum number of blocks based on the expected reading time; dividing the total number of rows by the minimum number of blocks to obtain the maximum row block size, and dividing the total number of rows by the maximum number of blocks to obtain the minimum row block size; the value of the row block size is selected between the maximum row block size and the minimum row block size.
[0043] This embodiment addresses the technical problem of determining the range of the number of blocks and the value of the row block size based on the memory space size and the expected read time. When processing large-scale CSV files, reasonably determining the number of blocks and the row block size is crucial for optimizing read efficiency and system resource utilization. The memory space size directly limits the amount of data that can be processed simultaneously; if the blocks are too large, it may lead to memory overflow. The expected read time reflects the required read speed; if the blocks are not properly allocated, it may result in excessively long read times. This embodiment's technical solution first comprehensively obtains three key factors: the total number of rows in the CSV file, the memory space size, and the expected read time. Based on the memory space size, the minimum number of blocks is calculated to ensure that each block does not exceed the memory limit during reading and processing. For example, for a CSV file with 100,000 rows, if the memory can only hold 50,000 rows of data, then the minimum number of blocks will be determined as 100,000 / 50,000 = 2 blocks to ensure that each block can be accommodated in memory. Simultaneously, the maximum number of blocks is calculated based on the expected read time to meet the read speed requirements. For example, if a read operation is required to be completed within 5 seconds, the maximum number of blocks is calculated based on the data volume and processing speed. For instance, assuming each block takes at least 50 milliseconds (0.05 seconds) to read, the maximum number of blocks is 5 / 0.05 = 100 blocks. Then, the total number of rows is divided by the minimum and maximum number of blocks to obtain the maximum and minimum row block sizes. Finally, a suitable row block size is selected from this range, for example, between 1000 and 50000 rows, choosing a value that satisfies both memory requirements and read speed. This comprehensive consideration and meticulous calculation finds the optimal balance between memory constraints and read speed requirements, thus determining the most suitable row block size and ensuring a highly efficient and stable read process.
[0044] Example 6
[0045] Based on the above embodiments, the step of obtaining the total number of rows in a CSV file includes: obtaining the file size of the CSV file; obtaining the data volume of a preset number of rows in the CSV file; and multiplying the ratio of the file size to the data volume by the preset number of rows to obtain the total number of rows in the CSV file.
[0046] The technical problem addressed in this embodiment is how to obtain the total number of rows in a CSV file without reading the entire file. When processing large CSV files, directly reading the entire file to obtain the total number of rows is an extremely time-consuming and resource-intensive method. This embodiment's solution estimates the total number of rows by using the file's physical size and a preset number of rows. First, the actual file size of the CSV file on the storage medium is obtained, reflecting the storage space occupied by the file. Then, a representative preset number of rows of data is obtained. By calculating the ratio of the file size to the preset number of rows of data, the approximate number of rows can be obtained. For example, if the file size is 20MB and the preset number of rows is 20KB, then the ratio is 20MB / 20KB = 1000, and the estimated total number of rows is approximately 1000 * 100 = 100,000 rows. This method eliminates the need to read the file content line by line, greatly saving reading time and system resources, and enabling a fast and efficient approximation of the total number of rows.
[0047] Example 7
[0048] Based on the above embodiments, when reading a row block for the first time, the row block size is set to 1 / 2 of the total number of rows; before returning to execute the above steps S10 to S30 to read the data of the next row block, the method further includes: setting the row block size of the next row block to 1 / 2 of the row block size of the current row block, until the row block size of the next row block is less than the minimum row block size; when the row block size of the next row block is less than the minimum row block size, the row block size of the next row block is modified to the minimum row block size.
[0049] The technical problem addressed in this embodiment is how to further optimize the reading process, reduce the number of loops, and accelerate the reading progress. Optimizing the reading strategy to reduce the number of loops and increase reading speed are key objectives in reading CSV files. This embodiment employs a dynamic strategy of gradually reducing the row block size. At the start of reading, the row block size is set to half the total number of rows, enabling rapid processing of most of the file's data. For example, for a file with 10,000 rows, the initial row block size is 5,000 rows. As reading progresses, the size of the next row block is set to half the current row block size, gradually narrowing the reading range and allowing for more precise processing of the remaining data. For example, the second row block size becomes 2,500 rows, and the third becomes 1,250 rows. When the row block size is less than a pre-set minimum value, such as 100, the row block size is fixed at 100 to avoid excessive loops and judgments caused by excessively small row blocks. This strategy of gradually reducing the row block size effectively adapts to different stages of the file reading process. By flexibly switching between rapid processing in the early stage and fine reading in the later stage, it reduces unnecessary calculations and time consumption, greatly improves reading efficiency, and enables the reading of large-scale CSV files to be completed more efficiently and quickly.
[0050] Example 8
[0051] Figure 2 This is a schematic diagram of the structure of a CSV file reading device provided in an embodiment of this application, as shown below. Figure 2 As shown in the technical solution of this embodiment, a CSV file reading device is provided. The device includes: an acquisition module, used to acquire the starting position and size of the row block to be read, wherein the row block size is the number of rows of the CSV file read each time; a reading module, used to read the CSV file based on the starting position and size of the row block to complete the data reading of the current row block; and a judgment module, used to judge whether the number of valid rows in the current row block is greater than a preset row number error threshold. If the number of valid rows is greater than the row number error threshold, the data of the next row block is read.
[0052] This embodiment addresses the technical problem of efficiently reading CSV files. Traditional line-by-line reading and judgment methods are inefficient when processing large-scale CSV files because frequent line-by-line operations and judgments consume significant time and system resources. The technical solution of this embodiment reads the file by dividing it into line blocks of a preset size, avoiding individual judgments for each line and reducing the number of judgments and repetitive operations. For example, for a CSV file with millions of rows of data, judging each line once would result in millions of judgments. However, by using line block reading, assuming a block size of 10,000 rows, the number of judgments can be reduced to one ten-thousandth of the original, and the total reading time will be significantly shortened. Changing the block size can further reduce the number of judgments. Simultaneously, this block reading method better utilizes the system's caching and parallel processing capabilities, improving data reading efficiency. Other technical solutions and effects of this embodiment correspond to the above embodiments and will not be repeated here.
[0053] Example 9
[0054] In the technical solution of this embodiment, a computer program product is provided, including steps for a computer program to implement any of the methods in the above embodiments.
[0055] In the technical solution of this embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the methods in the above embodiments.
[0056] Based on the above embodiments, this embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the above embodiments.
[0057] The processor may include, but is not limited to, one or more processors or microprocessors. Each processor may be implemented as an Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor, or other electronic component, for executing the methods described in the above embodiments.
[0058] Computer-readable storage media can be implemented by any type of volatile or non-volatile storage device or a combination thereof. Computer-readable storage media may include, but are not limited to, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, and computer storage media (e.g., hard disks, floppy disks, solid-state drives, removable disks, CD-ROMs, DVD-ROMs, Blu-ray discs, etc.).
[0059] Computer-readable storage media may also store at least one computer-executable program / instruction, such as computer-readable instructions. Computer-readable storage media include, but are not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Computer-readable storage media may include, for example, read-only memory (ROM), hard disk, flash memory, etc. For example, a non-transitory computer-readable storage medium may be connected to a computing device such as a computer, and then, when the computing device executes the computer-readable instructions stored on the computer-readable storage medium, the various methods described above can be performed.
[0060] In addition, the computer device may include (but is not limited to) a data bus, an input / output (I / O) bus, a display, and input / output devices (e.g., keyboard, mouse, speakers, etc.).
[0061] The processor can communicate with external devices via the I / O bus through wired or wireless networks.
[0062] In one embodiment, the at least one computer-executable instruction may also be compiled into or comprise a software product / computer program product, wherein one or more computer-executable instructions are executed by a processor to perform the steps of the various functions and / or methods in the embodiments described herein.
[0063] Example 10
[0064] Based on the above embodiments, this embodiment provides an application example.
[0065] Comma-Separated Values (CSV) files are plain text files that store data records separated by newline characters. Records consist of fields, and typically contain the same sequence of fields. CSV files are frequently used in research and development projects to store data records with identical fields because they are supported by most programs and offer good adaptability for data exchange. For example, in new energy vehicle projects, the vehicle bus continuously sends data records, and the received data records are stored in a very large CSV file.
[0066] For typical CSV files, the data is usually read line by line. However, if the CSV file is too large, this line-by-line reading method is inefficient.
[0067] This application provides a method for reading CSV files with large amounts of data, improving the efficiency of reading such CSV files. Here, "large amounts of data" refers to hundreds of thousands or even millions of rows of records. For such large volumes of data, reading line by line is extremely time-consuming and difficult to perform efficiently.
[0068] In related technologies, a header file containing information such as the number of original information records and a preset length is generated, along with the original information records and fixed-length, fixed-placeholders to generate corresponding actual information records to read and obtain CSV files. However, this can only be used for CSV files that meet certain prerequisites.
[0069] This application presents a reading method that enables more efficient reading of data records from CSV files. The basic concept involves designing pre-sized row blocks and reading the CSV data in blocks. A set judgment value is used to determine whether the CSV file has been completely read, thus looping through the entire file. This method is simpler, requires no header file generation, and has a wider range of applications.
[0070] In experimental development and testing projects, data records with a standardized format are often generated and stored in comma-separated value files. When accessing the data, these CSV files with a large number of records need to be read. These CSV files may vary in length and format. Current reading methods require checking each line as it is read, resulting in inefficiency and limited adaptability due to the specific format requirements of CSV files.
[0071] The present invention aims to design a reading method that can effectively meet the needs of reading large CSV files, with higher efficiency and applicability to a wider range of scenarios, thereby solving the technical problem of the inability to reasonably and efficiently read large CSV file data in project development and testing.
[0072] When receiving data from a project, the received data consists of rows with identical field characteristics. If a traditional row-by-row reading method is used, the number of reads required would be extremely large, and each read would require checking whether the CSV file has been completely read, presenting technical problems of low feasibility and efficiency.
[0073] This application provides a reading method that, when writing to a CSV file, sequentially inputs each data record as a row. When reading the CSV file, the rows are divided into pre-defined blocks, which are then read iteratively to read the entire file. This approach reduces the number of conditional checks, improves reading efficiency and adaptability, and enables effective reading of large CSV files. Applying this method to reading large CSV files effectively solves the aforementioned technical problems.
[0074] To meet the needs of project development experiments, this invention designs and uses a method for reading large CSV files in blocks. By dividing the entire CSV file into fixed-size blocks of valid data rows, a flag is set for reading each block. The method determines whether the entire file has been read based on the progress of each block, thus simplifying the judgment process, reducing the number of judgments, and improving reading efficiency. This block-based reading logic reduces the judgment operations present in traditional line-by-line reading and provides stronger reading continuity, greatly improving the feasibility and efficiency of file reading.
[0075] Figure 3 This is a flowchart illustrating a method for reading a CSV file provided in an embodiment of this application. Figure 3 As shown, firstly, the data to be saved is obtained from the data source in the project. The received data is saved as a CSV file in row-by-row format. After the data is saved, a large CSV file to be read is obtained. Each data record in this CSV file is on a separate row, and the data fields are theoretically arranged in a fixed order. Based on the total number of rows in the CSV file, an appropriate row block size is set, and the entire file is divided into blocks.
[0076] The line block record keeps track of the first line position, effectively recording the line block reading process and ensuring that the entire block is read, with a fixed number of lines read each time. A flag is set to point to the currently read line number. By using the flag line number and the first position of the current line block, the number of lines read in the current block can be obtained, allowing subsequent operations to proceed.
[0077] A boolean parameter is set to determine the size of the currently read row block when the difference between the row number in the flag bit and the row number in the first position equals the preset row block size. Because this application only needs to obtain valid row records when reading a CSV file, it reads and counts the valid rows, ignoring invalid rows such as blank lines, garbled lines, and non-record data lines that do not conform to the format. The number of valid rows in the row block is counted, and invalid rows are excluded.
[0078] A threshold is set to handle potential errors. The logic for determining whether a CSV file has been completely read is as follows: after reading the current row block, the number of valid rows read is used to determine if the CSV file has been completely read. If the number of valid rows read is significantly lower than the actual size of the row block, it indicates that there are many blank lines, and therefore the file can be considered completely read, meaning that the remaining lines are blank. Since the file may also contain blank lines or invalid lines such as garbled characters in the middle, the aforementioned error threshold needs to be set.
[0079] Set a reasonable error threshold (e.g., half the block size), and compare the number of valid lines read with the error threshold. If the number of valid lines is greater than or equal to the error threshold, the file is considered not yet fully read, and the previously set Boolean parameter is set to 0, indicating that the file reading is not complete; otherwise, it is 1, indicating that the reading is complete. Therefore, it is necessary to continue reading the next block and repeat the above block reading operation until a block is read and the Boolean value is 1, indicating that the entire file has been read, and the reading ends.
[0080] For the data source records, a large CSV file is obtained. Assuming there is a CSV file with 100,030 rows of data to be read, a fixed row block size is set, for example, 100. 100 rows of data constitute one row block. A record bit is set at the beginning of each row block to record the starting position of the row block. Simultaneously, an external flag is created during reading, with its value pointing to the current row position being read, thus indicating how many rows of data have been read in the current row block. The difference between the current flag bit and the row block starting position recorded by the record bit is equal to 100, indicating that the row block has been completely read. An external Boolean judgment bit and a numerical parameter are used to record the number of valid rows read. This is used to determine whether the current row block is valid. This application presets an error threshold, for example, half the row block size, such as 50. The number of valid rows read is compared with the error threshold. If the number of valid rows is greater than the error threshold of 50, it is determined that the CSV file has not been completely read, the judgment bit is set to 0, and the external parameters are reset to start reading the next row block. When the last line block is read, the valid line is 30, which is less than or equal to 50. The judgment position is 1, indicating that the CSV file has been read and the reading is complete.
[0081] In the embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0082] It should be noted that, in this invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element limited by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0083] While the embodiments disclosed in this invention are as described above, the above content is merely for the purpose of facilitating understanding of this invention and is not intended to limit the invention. Any person skilled in the art to which this invention pertains may make any modifications and changes in form and detail of the implementation without departing from the spirit and scope disclosed in this invention; however, the scope of patent protection of this invention shall still be determined by the scope defined in the appended claims.
Claims
1. A method for reading a CSV file, characterized in that: The method comprises: S10, obtaining the starting position and size of the row block to be read, wherein the row block size is the number of rows of the CSV file read each time; S20, reading the CSV file based on the row block starting position and the row block size, completing data reading of the current row block; S30, determining whether the number of valid rows in the current row block is greater than a preset row number error threshold; S40, if the valid number of rows is greater than the row number error threshold, returning to execute the above steps S10 to S30 to read the data of the next row block; S50: If the number of valid rows is less than or equal to the row number error threshold, then the reading of the CSV file ends.
2. The method for reading a CSV file according to claim 1, wherein: The row block starting position includes a row number of the CSV file, and the step of reading the CSV file based on the row block starting position and the row block size to complete the data reading of the current row block includes: Starting from the row corresponding to the starting position of the row block, the CSV file is read line by line; After reading the data of the row block size, obtaining the row number of the row where the last row of data is located; If the difference between the row number and the row number corresponding to the starting position of the row block is equal to the value of the row block size minus 1, it is determined that the data of the current row block has been read.
3. The method for reading a CSV file according to claim 1, wherein: Before the step of determining whether the number of valid rows in the current row block is greater than a preset row number error threshold, the method further includes: Eliminate invalid rows in the current row block to obtain valid rows in the current row block, wherein the invalid rows include at least one of blank rows, garbled rows, and rows not in a preset format; The number of the valid rows is determined.
4. The method for reading a CSV file according to claim 1, wherein: The value range of the row number error threshold is 1 / 5 to 1 / 2 of the row block size.
5. The method for reading a CSV file according to claim 1, wherein: Before the step of obtaining the starting position and size of the row block to be read, the method further includes: Get the total number of rows, memory size, and expected reading time of the CSV file; Determine the minimum number of blocks based on the memory space size and the maximum number of blocks based on the expected reading time; Dividing the total number of rows by the minimum number of blocks to obtain the maximum row block size, and dividing the total number of rows by the maximum number of blocks to obtain the minimum row block size; The value of the row block size is selected between the maximum row block size and the minimum row block size.
6. The method for reading a CSV file according to claim 5, characterized in that: The step of obtaining the total number of rows of the CSV file includes: Get the file size of the CSV file; Get the data volume of the preset number of rows of the CSV file; The total number of rows in the CSV file is obtained by multiplying the ratio of the file size to the data volume by the preset number of rows.
7. The method for reading a CSV file according to claim 5, wherein: When reading a row block for the first time, the row block size is set to 1 / 2 of the total number of rows; Before returning to execute the above steps S10 to S30 to read the data of the next row block, the method further includes: Setting the row block size of the next row block to 1 / 2 of the row block size of the current row block until the row block size of the next row block is smaller than the minimum row block size; When the row block size of the next row block is smaller than the minimum row block size, the row block size of the next row block is modified to the minimum row block size.
8. A CSV file reading device, characterized in that: The device comprises: An acquisition module is used to acquire the starting position and size of the row block to be read, wherein the row block size is the number of rows of the CSV file read each time; A reading module, used for reading the CSV file based on the row block starting position and the row block size, and completing the data reading of the current row block; The judging module is used to judge whether the number of valid rows in the current row block is greater than a preset row number error threshold, and if the number of valid rows is greater than the row number error threshold, read the data of the next row block.
9. A computer program product, comprising the steps of the computer program for implementing the method for reading a CSV file according to any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for reading a CSV file according to any one of claims 1 to 7 are implemented.