Data reading method, electronic equipment and computer readable storage medium

Abstract

The invention discloses a data reading method, electronic equipment and a computer readable storage medium, and the method comprises the steps: obtaining a Parquet file, and reading data blocks from the Parquet file in sequence; the Parquet file comprises a plurality of data blocks, the data blocks comprise metadata heads and data values, and block offset addresses of the data blocks, data offset addresses of the data values and verification information are stored in the metadata heads; in response to normal reading of the current data block, feedback information of a data value in the current data block is acquired, and a verification result of the current data block is determined based on the feedback information and the verification information; in response to reading abnormity of the current data block, taking the data offset address of the data value which is read abnormally as a reading starting point, and re-reading the data value in the current data block within a preset time condition; and skipping the current data block based on the block offset address of the current data block in response to the check result that the block is damaged. According to the scheme, the continuity and throughput of data processing can be guaranteed.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data reading method, electronic device, and computer-readable storage medium. Background Technology

[0002] With the development of big data technology and the increase in data volume, real-time and efficient data storage has become crucial. Parquet (an open-source columnar data storage format) is a popular columnar storage format, characterized by its compact structure and high space utilization. However, currently, when reading Parquet files, any error encountered during the reading process will cause the entire file reading task to fail, potentially halting the entire data processing pipeline and requiring system rescheduling. This compromises the continuity and throughput of data processing. Therefore, ensuring the continuity and throughput of data processing has become a pressing issue. Summary of the Invention

[0003] The main technical problem addressed by this application is to provide a data reading method, electronic device, and computer-readable storage medium that can ensure the continuity and throughput of data processing.

[0004] To address the aforementioned technical problems, this application provides a data reading method, comprising: acquiring a Parquet file and sequentially reading data blocks from the Parquet file; wherein the Parquet file includes multiple data blocks, each data block includes a metadata header and multiple data values, the metadata header storing the block offset address of the data block, the data offset address of the data value, and verification information; in response to a normal read of the current data block, acquiring feedback information of the data value in the current data block, and determining the verification result of the current data block based on the feedback information and the verification information; in response to an abnormal read of the current data block, using the data offset address of the abnormally read data value as the starting point, rereading the data value in the current data block within a preset time condition; and in response to the verification result indicating block corruption, skipping the current data block based on the block offset address of the current data block.

[0005] To address the aforementioned technical problems, a second aspect of this application provides an electronic device including a memory and a processor coupled to each other, wherein the memory stores program instructions and the processor executes the program instructions to implement the method described in the first aspect.

[0006] To address the aforementioned technical problems, a third aspect of this application provides a computer-readable storage medium storing program instructions executable by a processor, the program instructions being used to implement the method described in the first aspect.

[0007] The above scheme obtains the Parquet file to be read and reads data blocks sequentially from it. The Parquet file includes multiple data blocks, each containing a metadata header and multiple data values. The metadata header stores the block offset address, the data offset address of each data value, and verification information. When the current data block is read successfully, feedback information is obtained from the data values ​​within that block. Based on this feedback and verification information, the verification result for the current data block is obtained. When the current data block is read abnormally, the data offset address of the abnormal data value is used as the starting point for rereading the data values ​​within a preset time condition. If the verification result indicates block corruption, the current data block is skipped based on its block offset address. When a data block reads abnormally, it is not necessary to traverse the entire file or data block; the individual abnormal data value can be directly located through the data value offset address in the metadata header, rather than blindly retrying the entire data block. Furthermore, relying on the block offset address to accurately skip bad blocks aligns with the columnar storage and block management characteristics of Parquet, significantly reducing invalid I / O operations and thus ensuring the continuity and throughput of data processing. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart illustrating one implementation method of the data reading method of this application; Figure 2 This is a flowchart illustrating another embodiment of the data reading method of this application; Figure 3 This is a schematic diagram of the structure of one embodiment of the electronic device of this application; Figure 4 This is a schematic diagram of one embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0009] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments, and different implementation methods can be adaptively combined. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0010] In this paper, the terms "system" and "network" are often used interchangeably. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this paper means two or more.

[0011] Please see Figure 1 , Figure 1 This is a flowchart illustrating one implementation method of the data reading method. The data reading method includes: S101: Obtain the Parquet file and read data blocks sequentially from the Parquet file; wherein, the Parquet file includes multiple data blocks, each data block includes a metadata header and multiple data values, and the metadata header stores the block offset address of the data block, the data offset address of the data value, and verification information.

[0012] Specifically, the Parquet file to be read is obtained, and data blocks are read sequentially from the Parquet file. The Parquet file includes multiple data blocks, and each data block includes a metadata header and multiple data values. The metadata header stores the block offset address of the data block, the data offset address of the data value, and the verification information.

[0013] In one application method, each data block has a corresponding block identifier. After obtaining the Parquet file to be read, the data blocks are read sequentially from the Parquet file according to the block identifiers.

[0014] In another application, the data blocks correspond to a storage order. After obtaining the Parquet file to be read, the data blocks are read from the Parquet file in the storage order.

[0015] S102: In response to the current data block being read normally, obtain feedback information on the data values ​​in the current data block, and determine the verification result of the current data block based on the feedback information and the verification information.

[0016] Specifically, when the current data block being read is read normally, feedback information of the data values ​​in the current data block is obtained, and the verification result of the current data block is obtained based on the feedback information and the verification information.

[0017] It should be noted that when the number of data values ​​read in the current data block is the same as the number of data values ​​stored in the metadata header, the read status of the current data block is determined to be normal.

[0018] In one application mode, when the current data block being read is read normally, feedback information is generated based on all data values ​​in the current data block and their corresponding metadata information. Based on the feedback information and the verification information, the verification result of the current data block is determined.

[0019] In one application scenario, the verification information includes a checksum and verification metadata. A hash value is calculated for all data values ​​in the current data block to obtain the verification value in the feedback information. When the verification value and checksum match, the first verification sub-result is considered successful. The metadata information corresponding to all data values ​​in the current data block is then matched with the verification metadata. If a match is successful, the second verification sub-result is considered successful. If both verification sub-results are successful, the current data block is considered intact; otherwise, it is considered corrupted.

[0020] In another application, when the current data block being read is read normally, feedback information is generated based on the feature values ​​of all data values ​​in the current data block. Based on the feedback information and the verification information, the verification result of the current data block is determined.

[0021] In one application scenario, the verification information includes a preset data verification length and a first verification byte. The data length of all data values ​​in the current data block and the first byte are used as feedback information. When the data length of all data values ​​is consistent with the data verification length and the first byte is consistent with the first verification byte, the verification result of the current data block is determined to be a complete block; otherwise, the block is considered corrupted.

[0022] S103: In response to a current data block read error, the data offset address of the data value with the read error is used as the starting point for rereading the data value in the current data block within a preset time condition.

[0023] Specifically, when an error occurs while reading the current data block, the data offset address of the data value that was read incorrectly is used as the starting point for reading, and the data value in the current data block is reread within a preset time condition.

[0024] It should be noted that due to factors such as network fluctuations, read exceptions may occur during the reading of the current data block, such as input / output exceptions (IOException) and network timeout exceptions (SocketTimeoutException).

[0025] Understandably, if the current data block continues to show a read error within the preset time period, the verification result will also indicate that the block is corrupted.

[0026] In one application mode, when an error occurs while reading the current data block, the offset address of the data with the error is used as the starting point for reading. Within a preset time window, the retry interval increases exponentially with the number of retries until the maximum interval is reached or a timeout occurs, at which point reading the data value in the current data block stops.

[0027] In another application, when an error occurs while reading the current data block, the offset address of the data with the error is used as the starting point for reading. The first attempt is immediately retried. If it fails, it continues to retry at fixed time intervals until the preset total time is exhausted, at which point reading the data value in the current data block is stopped.

[0028] S104: In response to the check result indicating that the block is corrupt, skip the current data block based on the block offset address of the current data block.

[0029] Specifically, when the verification result indicates that the block is corrupt, the current data block is skipped based on the block offset address of the current data block.

[0030] Understandably, when the verification result indicates that the block is complete, the next data block in the Parquet file is read.

[0031] In one application mode, an offset table is set up for the block offset address of all data blocks. This offset table is stored in the local storage system. When the verification result is that the block is corrupt, the block offset address of the next data block is directly obtained by looking up the offset table, and the corrupted current data block is skipped.

[0032] In another application, each data block has its own block offset address and the block offset address of the next data block embedded in a fixed position. For example, the first 8 bytes of the header of each data block embed its own block offset address, and the last 4 bytes of the tail embed the block offset address of the next data block. When the verification result is that the block is corrupted, the block offset address of the next data block embedded in the data block is read directly.

[0033] The above scheme obtains the Parquet file to be read and reads data blocks sequentially from it. The Parquet file includes multiple data blocks, each containing a metadata header and multiple data values. The metadata header stores the block offset address, the data offset address of each data value, and verification information. When the current data block is read successfully, feedback information is obtained from the data values ​​within that block. Based on this feedback and verification information, the verification result for the current data block is obtained. When the current data block is read abnormally, the data offset address of the abnormal data value is used as the starting point for rereading the data values ​​within a preset time condition. If the verification result indicates block corruption, the current data block is skipped based on its block offset address. When a data block reads abnormally, it is not necessary to traverse the entire file or data block; the individual abnormal data value can be directly located through the data value offset address in the metadata header, rather than blindly retrying the entire data block. Furthermore, relying on the block offset address to accurately skip bad blocks aligns with the columnar storage and block management characteristics of Parquet, significantly reducing invalid I / O operations and thus ensuring the continuity and throughput of data processing.

[0034] In one implementation, please refer to Figure 2 , Figure 2 This is a flowchart illustrating another embodiment of the data reading method of this application. The data reading method includes: S201: Obtain the Parquet file and read data blocks sequentially from the Parquet file.

[0035] Specifically, obtain the Parquet file to be read, and read the data blocks from the Parquet file sequentially.

[0036] In one implementation scenario, the Parquet file also includes at least one row group unit, which includes multiple data blocks. Step S201, which reads data blocks sequentially from the Parquet file, specifically includes reading data blocks from the Parquet file according to the granularity order of the row group units.

[0037] Specifically, data blocks in the row group units are read sequentially from the Parquet file according to the granularity of the row group units. Since the row group unit is the smallest independent read / write unit in the Parquet file, when reading data blocks, it is only necessary to load all data blocks in the current row group unit into the cache, rather than loading the entire Parquet file at once. This can significantly reduce cache usage and improve data reading efficiency.

[0038] S202: In response to the current data block being read normally, feedback information is obtained based on all data values ​​in the current data block; the feedback information includes checksum and metadata information.

[0039] Specifically, when the current data block being read is read normally, feedback information is obtained based on all the data values ​​in the current data block. This feedback information includes checksums and metadata information.

[0040] In one implementation scenario, hash values ​​are calculated for all data values ​​in the current data block to obtain a check value, and metadata information corresponding to all data values ​​in the current data block is obtained.

[0041] S203: Based on the check value and the built-in check code, obtain the first check result; based on the metadata information and the check metadata, obtain the second check result.

[0042] Specifically, the verification information includes a built-in checksum and verification metadata. The verification value is compared with the built-in checksum to obtain the first verification result, and the metadata information is matched with the verification metadata to obtain the second verification result.

[0043] S204: Determine the verification result of the current data block based on the results of the first and second checksums.

[0044] Specifically, the verification result of the current data block is determined jointly based on the first verification result and the second verification result. The first verification result, obtained by comparing the verification value with the built-in verification code, can verify the integrity of the data content. The second verification result, obtained by matching the metadata information with the verification metadata, can verify the logical validity of the data. Thus, the combination of "content + attribute" dual verification greatly reduces the probability of misjudgment and improves the comprehensiveness and accuracy of data verification.

[0045] In one implementation scenario, step S204 specifically includes: in response to at least one of the first check result and the second check result being a check failure, determining that the check result is a block corruption and determining the error type; storing the identifier, block offset address and error type of the current data block in the local storage system.

[0046] Specifically, when at least one of the first and second checksum results fails, the checksum result is determined to be block corruption. Specifically, when the checksum and the built-in checksum fail to match, the first checksum result is determined to be a checksum failure. When the metadata information and the checksum metadata do not match, the second checksum result is determined to be a checksum failure. If the first checksum result fails, the corresponding error type is data content abnormality. If the second checksum result fails, the corresponding error type is metadata abnormality. The identifier of the current data block, the block offset address, and the corresponding error type are stored in the local storage system to facilitate fault location and improve the efficiency of subsequent fault investigation and repair.

[0047] S205: In response to a current data block read error, the data offset address of the data value with the read error is used as the starting point for rereading the data value in the current data block within a preset time condition.

[0048] Specifically, when an error occurs while reading the current data block, the data offset address of the data value that was read incorrectly is used as the starting point for reading, and the data value in the current data block is reread within a preset time condition.

[0049] In one implementation scenario, step S205, which involves rereading the data value in the current data block within a preset time condition, specifically includes: rereading the data value in the current data block at preset time intervals within a threshold time window, and determining the corresponding number of retries; and stopping the reading of the data value in the current data block in response to the number of retries exceeding the threshold.

[0050] Specifically, within a threshold time window, the data value in the current data block is reread at preset time intervals, and the corresponding number of retries is determined. When the number of retries exceeds the threshold, reading the data value in the current data block stops. By retrying at preset intervals within the threshold time window, the random errors of a single read can be compensated for, increasing the probability of successfully obtaining valid data and avoiding direct judgment of data loss due to a single read failure. This ensures the integrity of data blocks in the data reading or storage link. Furthermore, setting a "number threshold" as the upper limit for retries prevents the system from falling into an infinite loop of read retries. If the data block continues to fail to read due to irreparable reasons such as hardware damage, permanent link interruption, or damage to the data block itself, the system will automatically stop retries after reaching the number threshold, thereby avoiding the ineffective occupation of system resources such as CPU, memory, and communication bandwidth, and ensuring the normal operation of other business processes in the system.

[0051] It should be noted that, for data block reading anomalies caused by network fluctuations, a partial data packet retransmission mechanism can be added to avoid data block anomalies triggered by a single network jitter.

[0052] In a specific implementation scenario, the threshold time window includes multiple sequentially ordered time intervals, and these sequentially ordered time intervals grow exponentially.

[0053] Specifically, the threshold time window includes multiple sequentially ordered time intervals, and these time intervals increase exponentially with the number of retries. For example, there is a threshold time window of 10 seconds, and the threshold number of retries is 3. The first retry is at the 1st second, the second retry is at the 4th second, and the third retry is at the 9th second.

[0054] In other implementation scenarios, retries can also be performed at fixed time intervals. For example, there is a threshold time window of 6 seconds, and the threshold number of retries is 3 times. The first retry is at 1 second, the second retry is at 3 seconds, and the third retry is at 5 seconds. This application does not impose specific restrictions here.

[0055] In one implementation scenario, after the step of stopping reading the data value in the current data block in response to the retry count exceeding the threshold, the method further includes: determining that the verification result is that the block is corrupted, and skipping the current data block based on the block offset address of the current data block.

[0056] Specifically, when the number of retries exceeds the threshold, the reading of data values ​​in the current data block is stopped, and the verification result of the current data block is determined to be corrupted. The current data block is skipped based on its block offset address. This retry mechanism can stop the operation after the number of retries exceeds the threshold, avoiding the ineffective consumption of system resources and ensuring the efficient operation of the system. At the same time, accurately skipping bad blocks based on the block offset address allows the system to directly advance to the reading of subsequent normal data blocks, without blocking the overall process due to a single bad block, thereby further ensuring the continuity and throughput of data processing.

[0057] In one implementation scenario, before the step of rereading the data value in the current data block at preset time intervals within a threshold time window, the method further includes: storing the data offset address of the abnormally read data value in the current data block to the local storage system.

[0058] Specifically, before rereading the data value in the current data block at preset time intervals within the threshold time window, the data offset address of the abnormal data value in the current data block can be stored in the local storage system. The stored data offset address allows the retry operation to accurately locate the target abnormal data position without having to traverse the entire data block again to find the abnormal data, reducing the address addressing overhead during retry and improving the efficiency of retry reading.

[0059] S206: In response to a check result indicating block corruption, skip the current data block based on the block offset address of the current data block.

[0060] Specifically, when the verification result indicates that the block is corrupt, the current data block is skipped based on the block offset address of the current data block.

[0061] The above scheme obtains the Parquet file to be read, and reads data blocks sequentially from the Parquet file. When the current data block being read is read normally, feedback information is obtained based on all data values ​​in the current data block. This feedback information includes a checksum and metadata information. The checksum information includes a built-in checksum and checksum metadata. The checksum and the built-in checksum are compared to obtain a first checksum result. The metadata information and the checksum metadata are matched to obtain a second checksum result. The checksum result of the current data block is determined based on the first checksum result and the second checksum result. When the current data block being read is read abnormally, the data offset address of the abnormal data value is used as the starting point for reading, and the reading continues for a preset time bar. The system rereads the data value in the current data block. When the verification result indicates that the block is corrupted, it skips the current data block based on the block offset address. The first verification result obtained by comparing the verification value with the built-in checksum can verify the integrity of the data content. The second verification result obtained by matching the metadata information with the verification metadata can verify the logical validity of the data. Thus, the combination of "content + attribute" dual verification greatly reduces the probability of false positives, improving the comprehensiveness and accuracy of data verification. At the same time, accurately skipping bad blocks based on the block offset address allows the system to directly advance to the reading of subsequent normal data blocks, preventing the overall process from being blocked by a single bad block. This further ensures the continuity and throughput of data processing.

[0062] Please see Figure 3 , Figure 3This is a schematic diagram of an embodiment of the electronic device of this application. The electronic device 30 includes a memory 300 and a processor 302 coupled to each other. The memory 300 stores program data (not shown). The processor 302 calls the program data to implement the method in any of the above embodiments. For related descriptions, please refer to the detailed description of the above method embodiments, which will not be repeated here. Specifically, the electronic device 30 includes: desktop computers, laptops, tablet computers, servers, etc., which are not limited here. In addition, the processor 302 can also be called a CPU (Center Processing Unit). The processor 302 may be an integrated circuit chip with signal processing capabilities. The processor 302 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. In addition, the processor 302 can be implemented by integrated circuit chips.

[0063] Please see Figure 4 , Figure 4 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 40 stores program data 400. When the program data 400 is executed by a processor, it implements the method in any of the above embodiments. For a detailed description of the relevant content, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0064] It should be noted that the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0065] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0066] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0067] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A data reading method, characterized in that, include: Obtain a Parquet file and read data blocks sequentially from the Parquet file; wherein the Parquet file includes multiple data blocks, each data block includes a metadata header and multiple data values, and the metadata header stores the block offset address of the data block, the data offset address of the data value, and verification information; In response to the current data block being read normally, feedback information of the data value in the current data block is obtained, and the verification result of the current data block is determined based on the feedback information and the verification information. In response to the current data block read error, the data offset address of the read error data value is used as the starting point for reading, and the data value in the current data block is reread within a preset time condition; In response to the verification result indicating that the block is corrupt, the current data block is skipped based on the block offset address of the current data block.

2. The method according to claim 1, characterized in that, The verification information includes a built-in verification code and verification metadata; The step of obtaining feedback information on the data values ​​in the current data block, and determining the verification result of the current data block based on the feedback information and the verification information, includes: The feedback information is obtained based on all data values ​​in the current data block; wherein, the feedback information includes a checksum and metadata information; Based on the verification value and the built-in verification code, a first verification sub-result is obtained; based on the metadata information and the verification metadata, a second verification sub-result is obtained. Based on the first and second verification results, the verification result of the current data block is determined.

3. The method according to claim 2, characterized in that, Determining the verification result of the current data block based on the first verification result and the second verification result includes: In response to at least one of the first check sub-result and the second check sub-result being a check failure, the check result is determined to be block corruption and the error type is determined; The identifier, block offset address, and error type of the current data block are stored in the local storage system.

4. The method according to claim 1, characterized in that, The step of rereading the data value in the current data block within a preset time condition includes: Within a threshold time window, the data values ​​in the current data block are reread at preset time intervals, and the corresponding number of retries is determined. In response to the retry count exceeding the threshold, reading data values ​​from the current data block is stopped.

5. The method according to claim 4, characterized in that, The threshold time window includes multiple sequentially ordered time intervals, and these sequentially ordered time intervals increase exponentially.

6. The method according to claim 4, characterized in that, After responding to the retry count exceeding the threshold and stopping reading the data value in the current data block, the method further includes: If the verification result indicates that the block is corrupted, the current data block is skipped based on its block offset address.

7. The method according to claim 1, characterized in that, Before rereading the data values ​​in the current data block at preset time intervals within the threshold time window, the method further includes: Store the data offset address of the abnormal data value read in the current data block to the local storage system.

8. The method according to claim 1, characterized in that, The Parquet file also includes at least one row group unit, which includes a plurality of the data blocks; The step of sequentially reading data blocks from the Parquet file includes: The data blocks are read from the Parquet file in granular order of the row group units.

9. An electronic device, characterized in that, The method includes a memory and a processor coupled to each other, the memory storing program instructions, and the processor executing the program instructions to implement the method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The system stores program instructions that can be executed by a processor, the program instructions being used to implement the method according to any one of claims 1-8.

Images