Data reading method and device, equipment and storage medium

Abstract

The invention discloses a data reading method and device, equipment and a storage medium, relates to the technical field of storage, is applied to a data reading controller, and comprises the steps of determining a target physical address corresponding to a target data block based on a received first index value, reading a target data block from a storage medium of a rear-end storage unit according to the target physical address; and based on the first index value and verification information, verifying a data integrity verification field in the target data block, and if verification succeeds, returning the target data block to a user side. The problem of I / O conflict between the data reading operation and other modification operations can be solved, and the data reading performance is improved.

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and in particular to a data reading method, device, equipment and storage medium. Background Art

[0002] For a stand-alone storage system or a distributed storage system, data is generally placed in the storage medium according to custom rules. When reading is required, the storage system first queries the physical address corresponding to the data through the index, and then reads the corresponding data from the storage medium through the physical address.

[0003] However, during the reading process, the foreground I / O (Input / Output) operation may conflict with the background migration, writing and other I / O operations. Generally, the consistency of reading is ensured by serializing the same data block, that is, no modification operation is allowed to this data block during the period from the query of the index physical address to the return of the data. In addition, the technology to ensure data I / O consistency also includes MVCC (Multi-Version Concurrency Control). The idea of ​​MVCC is generally that each update operation on the data will create a new data version instead of directly overwriting the original data, so that multiple versions of data can be retained in the database. Each version is identified by a timestamp or version number. When a transaction needs to read data, the database system will select the appropriate data version according to the timestamp or version number of the transaction start, thereby achieving read consistency. However, these technologies usually assume that the data may be modified in advance and make additional restrictions. Such redundant control measures will lead to reduced reading performance or redundant I / O operations.

[0004] Therefore, the above technical problems need to be solved by those skilled in the art urgently. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a data reading method, device, equipment and storage medium, which can solve the I / O conflict problem between data reading operation and other modification operations and improve the performance of data reading. The specific scheme is as follows:

[0006] A first aspect of the present application provides a data reading method, comprising:

[0007] Determine a target physical address corresponding to the target data block based on the received first index value, and read the target data block from a storage medium of a backend storage unit according to the target physical address;

[0008] The data integrity check field in the target data block is checked based on the first index value and the check information, and if the check succeeds, the target data block is returned to the user end.

[0009] Optionally, the data reading method further includes:

[0010] Parsing the data read request sent by the client to obtain the first index value;

[0011] If the data read request also carries verification type information, the verification information is determined based on the verification type information; if the data read request does not carry verification type information, the verification information is determined based on a preset reading mode of the data read controller.

[0012] Optionally, the verification type information includes a first verification type and a second verification type, and the verification types corresponding to the preset read mode are the first verification type and the second verification type; wherein, the verification information corresponding to the first verification type is the first offset information carried in the data read request, and the verification information corresponding to the second verification type includes the first offset information carried in the data read request and a first verification code obtained by calculating the data information in the target data block using a preset verification algorithm.

[0013] Optionally, when the check information is the first offset information, checking the data integrity check field in the target data block based on the first index value and the check information includes:

[0014] Parsing the data integrity check field in the target data block to obtain a second index value and second offset information;

[0015] Determine whether the second index value is consistent with the first index value, and determine whether the second offset information is consistent with the first offset information;

[0016] If the second index value is consistent with the first index value, and the second offset information is consistent with the first offset information, the verification succeeds, otherwise the verification fails.

[0017] Optionally, when the check information is the first offset information and the first check code, checking the data integrity check field in the target data block based on the first index value and the check information includes:

[0018] Parsing the data integrity check field in the target data block to obtain a second check code, a second index value, and second offset information;

[0019] Determine whether the second check code is consistent with the first check code, and determine whether the second index value and the second offset information are consistent with the first index value and the first offset information respectively;

[0020] If the first check code is consistent with the second check code, and the second index value and the second offset information are respectively consistent with the first index value and the first offset information, the check succeeds; otherwise, the check fails.

[0021] Optionally, the data reading method further includes:

[0022] When storing the data information into the storage medium, the data information is calculated using the preset verification algorithm to obtain the second verification code, and the second verification code is written into the data integrity verification field; wherein the preset verification algorithm is a cyclic redundancy check algorithm or an information summary algorithm.

[0023] Optionally, after verifying the data integrity check field in the target data block based on the first index value and the check information, the method further includes:

[0024] If the verification fails, the target data block is read from the storage medium of the backend storage unit based on a preset read consistency rule, and the target data block is returned to the user end; wherein the preset read consistency rule is a serialization technology or a multi-version concurrency control technology.

[0025] Optionally, determining a target physical address corresponding to the target data block based on the received first index value includes:

[0026] Searching for a physical address corresponding to the first index value from an index buffer; the index buffer stores a mapping relationship between the index value and the physical address;

[0027] The physical address corresponding to the first index value is determined as the target physical address corresponding to the target data block.

[0028] Optionally, the process of searching the index buffer for the physical address corresponding to the first index value further includes:

[0029] If the first index value does not exist in the index buffer, the target data block is read from the storage medium of the back-end storage unit based on a preset read consistency rule, and the target data block is returned to the user end; wherein the preset read consistency rule is a serialization technology or a multi-version concurrency control technology;

[0030] The data integrity check field in the target data block is parsed to obtain corresponding index information, and the index information is added to the index buffer to update the index buffer.

[0031] Optionally, the data reading method further includes:

[0032] When an index update request sent by the user is obtained, the index update request is sent to the back-end storage unit so that the back-end storage unit returns the corresponding index information based on the data to be read in the index update request.

[0033] The index information corresponding to the data to be read is added to the index buffer to update the index buffer.

[0034] A second aspect of the present application provides a data reading device, comprising:

[0035] A data reading module, configured to determine a target physical address corresponding to a target data block based on the received first index value, and read the target data block from a storage medium of a back-end storage unit according to the target physical address;

[0036] A verification module is used to verify the data integrity verification field in the target data block based on the first index value and the verification information, and if the verification is successful, return the target data block to the user end.

[0037] A third aspect of the present application provides a distributed storage system, comprising a plurality of clients, a server, and a plurality of storage terminals, wherein the server terminal is respectively connected to the plurality of clients, and the server terminal is respectively connected to the plurality of storage terminals;

[0038] Any client includes a first data reading controller and a first data verifier, the first data reading controller is connected to the server and the first data verifier respectively, and the first data reading controller is also used to connect to the user end in the client; each storage end includes at least one storage medium, and the multiple storage ends are deployed in a distributed manner;

[0039] The first data checker is used to check the data integrity check field in the data block based on the index value and the check information, and send the check result and the data block to the first data reading controller;

[0040] The first data reading controller is used to execute the aforementioned data reading method.

[0041] A fourth aspect of the present application provides an electronic device, comprising a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the aforementioned data reading method.

[0042] A fifth aspect of the present application provides a computer-readable storage medium, comprising a front-end service layer, a transmission service layer, and a back-end storage layer connected in sequence;

[0043] The front-end service layer includes a second data reading controller and a second data verifier, the second data reading controller is connected to the transmission service layer and the second data verifier respectively, and the second data reading controller is also used to connect to the user end; the back-end storage layer includes at least one storage medium deployed locally;

[0044] The second data checker is used to check the data integrity check field in the data block based on the index value and the check information, and send the check result and the data block to the second data reading controller;

[0045] The second data reading controller is used to execute the aforementioned data reading method.

[0046] In the present application, the target physical address corresponding to the target data block is first determined based on the received first index value, and the target data block is read from the storage medium of the back-end storage unit according to the target physical address; the data integrity check field in the target data block is checked based on the first index value and the check information, and if the check succeeds, the target data block is returned to the user end. It can be seen that the present application first determines the target physical address corresponding to the target data block based on the received first index value, and then reads the target data block from the storage medium of the back-end storage unit according to the target physical address, and further, the data integrity check field in the target data block is checked using the first index value and the check information, and if the check succeeds, the data block is returned to the user end, so as to realize an optimistic locking mechanism based on the data integrity check field in the data read-through scenario. Through the above scheme, there is no need to impose additional restrictions on the data, which solves the I / O conflict problem between the data reading operation and other modification operations and improves the performance of data reading. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0048] Figure 1 A schematic diagram of a system framework applicable to a data reading method disclosed in this application;

[0049] Figure 2 A schematic diagram of a system framework applicable to another data reading method disclosed in this application;

[0050] Figure 3 A flow chart of a data reading method disclosed in this application;

[0051] Figure 4 A flow chart of a method for obtaining index values ​​and verification information disclosed in this application;

[0052] Figure 5 A schematic diagram of the structure of a distributed storage system disclosed in this application;

[0053] Figure 6 A schematic diagram of the structure of a single-node storage host disclosed in this application;

[0054] Figure 7 A schematic diagram of a data integrity check domain disclosed in this application;

[0055] Figure 8 This is a schematic diagram of the structure of a data reading device disclosed in this application. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] In the prior art, data is generally placed in a storage medium according to user-defined rules. When data needs to be read, the storage system first queries the physical address corresponding to the data through an index, and then reads the corresponding data from the storage medium through the physical address.

[0058] For example, for a stand-alone file system, the general reading process is as follows: 1. File system metadata positioning: When a file needs to be read, the file storage location is first located through the file system metadata (such as inode); 2. Physical address resolution: Find the physical address of the file data block through inode; 3. Storage medium reading: Read the corresponding data block from the storage medium through the physical address, and then assemble it into a complete file content.

[0059] However, during the reading process, the foreground I / O operation may conflict with the background migration, writing and other I / O operations. Common technologies to ensure data I / O consistency include serialization and MVCC multi-version concurrency control. However, these technologies usually assume that the data may be modified in advance and make additional restrictions. Such redundant control measures will lead to reduced reading performance or generate redundant I / O operations.

[0060] To this end, the present application provides a data reading method, apparatus, device and storage medium, which can solve the I / O conflict problem between data reading operations and other modification operations and improve the performance of data reading.

[0061] The system framework used in the data reading solution of this application can be found in Figure 1 As shown, it may specifically include a data reading controller, an index buffer, a data checker, a data transmission channel and a storage medium.

[0062] The solution of the present application can be applied to distributed storage systems and can also be used for single-node storage hosts. In a distributed storage system, the data read controller can generally be located at the client of the distributed storage system; in a single-node storage host, the data read controller is generally located at the service layer within the node, that is, the front end of the single-machine storage service, to maximize the performance optimization of the read path; the data read controller is mainly used to obtain data read requests sent by the user end, and read data from the storage medium based on the data read request.

[0063] The index buffer is used to cache the mapping relationship between the index value of the read data and the physical address; the data checker is mainly used to check the read data block based on the first index value and the check information. It should be noted that the index buffer and the data checker can be as follows Figure 1 As shown in the figure, as a peripheral module to interact with the data reading controller, it can also be as follows Figure 2 As shown in the figure, it is integrated inside the data reading controller. The data transmission channel includes Figure 1 , Figure 2The preset direct path on the left side of the figure reads data blocks directly from the storage medium, and the slow path on the right side reads data blocks based on the preset read consistency rules; the preset direct path can be understood as a direct IO path for fast reading and writing data in the medium. For distributed storage systems, it can be a message network supporting RDMA (Remote Direct Memory Access) from the client to the storage host, an IO channel supporting NVMeoF (NVMe over Fabrics, a protocol for transmitting NVMe commands in data centers and storage networks; NVMe is Non-Volatile Memory Express, non-volatile memory host controller interface specification) protocol, etc. to accelerate information transmission. For a single-node storage host, it can be an IO channel such as NVMe, SAS (Serial Attached SCSI, serial attached SCSI interface; SCSI is Small Computer System Interface, small computer system interface) or PCIe (peripheralcomponent interconnect express, high-speed serial computer expansion bus standard) within the same host, directly accessing the storage medium without going through mutually exclusive routing and other logical methods. It should be pointed out that the direct IO path is not limited to the above-mentioned direct acceleration method. The storage medium can be a persistent medium such as an SSD (Solid State Disk) or an HDD (Hard Disk Drive), which has the functions of writing and reading data.

[0064] Figure 3 A flow chart of a data reading method provided in an embodiment of the present application. Figure 2 As shown, the data reading method includes:

[0065] Step S11: determining a target physical address corresponding to a target data block based on the received first index value, and reading the target data block from a storage medium of a back-end storage unit according to the target physical address.

[0066] In this embodiment, the target physical address corresponding to the target data block is first determined based on the received first index value, and then the target data block can be read from the storage medium of the back-end storage unit according to the target physical address.

[0067] In a specific implementation, the determination of the target physical address corresponding to the target data block based on the received first index value includes: searching the index buffer for the physical address corresponding to the first index value; the index buffer stores a mapping relationship between the index value and the physical address; and determining the physical address corresponding to the first index value as the target physical address corresponding to the target data block. It can be understood that the present application sets up an index buffer for storing the mapping relationship between the index value of the data and the physical address, so the physical address corresponding to the first index value can be searched from the index buffer, and the physical address corresponding to the first index value can be determined as the target physical address corresponding to the target data block.

[0068] Further, in the process of searching the physical address corresponding to the first index value from the index buffer, it also includes: if the first index value does not exist in the index buffer, the target data block is read from the storage medium of the back-end storage unit based on the preset read consistency rule, and the target data block is returned to the user end; wherein the preset read consistency rule is a serialization technology or a multi-version concurrency control technology; the data integrity check field in the target data block is parsed to obtain the corresponding index information, and the index information is added to the index buffer to update the index buffer. That is, when searching the physical address corresponding to the first index value from the index buffer, if the first index value does not exist in the index buffer, the target data block can only be read from the storage medium of the back-end storage unit based on the preset read consistency rule, and the preset read consistency rule can specifically be a serialization technology or a multi-version concurrency control technology. Further, after reading the target data block, the data integrity check field in the target data block is parsed to obtain the corresponding index information, so as to add the index information to the index buffer to update the index buffer. It should be pointed out that in addition to adding the index information to the index buffer, the corresponding physical address information also needs to be updated. In this way, when the same data block is read later, the corresponding physical address can be queried from the index buffer using the index value, and then the data can be directly read from the storage medium of the back-end storage unit through the preset direct path. This update method can be understood as a passive update of the index buffer. It can be understood that the index value is stored in the index buffer, and as the data is read, the index buffer is updated according to the LRU (Least Recently Used) strategy, and the index update strategy can be the passive update strategy mentioned above.

[0069] In addition, the index update strategy also includes an active update strategy. The embodiment of the present application specifically discloses: when an index update request sent by the user end is obtained, the index update request is sent to the back-end storage unit, so that the back-end storage unit returns the corresponding index information based on the data to be read in the index update request, and adds the index information corresponding to the data to be read to the index buffer to update the index buffer. That is, the active update strategy is actively initiated by the user end. When the user perceives that a certain part of the data needs to be read directly, an index update request is asynchronously initiated, and the data reading controller sends the index update request to the back-end storage unit, so that the back-end storage unit returns the corresponding index information based on the data to be read in the index update request, and the data reading controller then adds the index information corresponding to the data to be read to the index buffer to update the index buffer.

[0070] Step S12: verifying the data integrity check field in the target data block based on the first index value and the check information, and if the check succeeds, returning the target data block to the user end.

[0071] In this embodiment, the data integrity check field (DIF, Data Integrity Field) in the read target data block is checked using the first index value and the check information. If the check succeeds, the target data block is returned to the user end to implement an optimistic locking mechanism based on the data integrity check field in a data read-through scenario. Among them, DIF is a technology used to ensure data integrity in storage devices or storage networks. It adds additional metadata fields to the data block to store data checksums and other related information so as to detect and correct data errors during data transmission and storage. Through the above scheme, there is no need to impose additional restrictions on the data, which solves the I / O conflict problem between data reading operations and other modification operations and improves the performance of data reading.

[0072] In addition, after verifying the data integrity check field in the target data block based on the first index value and the check information, it also includes: if the verification fails, reading the target data block from the storage medium of the back-end storage unit based on the preset read consistency rule, and returning the target data block to the user end; wherein the preset read consistency rule is a serialization technology or a multi-version concurrency control technology. That is, if the verification fails, the slow path determined based on the preset read consistency rule reads the target data block from the storage medium of the back-end storage unit, and returns the target data block to the user end to ensure that the read data is consistent. The preset read consistency rule can specifically be a serialization technology or a multi-version concurrency control technology. It can be understood that the slow path is the one that passes through Figure 1 and Figure 2The read-write consistency processing layer in the storage medium is used to read the target data block from the storage medium and return it. The read-write consistency processing layer can be specifically set based on serialization technology or based on multi-version concurrency control technology.

[0073] It can be seen that in the present application, the target physical address corresponding to the target data block is first determined based on the received first index value, and the target data block is read from the storage medium of the back-end storage unit according to the target physical address; the data integrity check field in the target data block is checked based on the first index value and the check information, and if the check succeeds, the target data block is returned to the user end. It can be seen that the present application first determines the target physical address corresponding to the target data block based on the received first index value, and then reads the target data block from the storage medium of the back-end storage unit according to the target physical address, and further, the data integrity check field in the target data block is checked using the first index value and the check information, and if the check succeeds, the data block is returned to the user end, so as to realize an optimistic locking mechanism based on the data integrity check field in the data read-through scenario. Through the above scheme, there is no need to impose additional restrictions on the data, which solves the I / O conflict problem between the data reading operation and other modification operations and improves the performance of data reading.

[0074] In a specific embodiment, Figure 4 As shown, the process of obtaining the first index value and the verification information specifically includes the following steps:

[0075] Step S21: parsing the data read request sent by the client to obtain the first index value.

[0076] In this embodiment, after obtaining the data read request from the user end, the data read request is first parsed to obtain the first index value it carries. It is understandable that when the data is written to the storage medium, the user end will record the corresponding first index value, so when the user end needs to read a block of data, the data read request sent by the user end will carry the corresponding first index value. It should be noted that the user end here refers to the user of this basic component, generally speaking, the various business parties that connect to this component.

[0077] In a distributed storage system, the user end is located in the client, at the upper layer of the data reading controller, as shown in the following example: Figure 5 As shown in , in a single-node storage host, the user end is independent of the single-node device and is connected to the data reading controller, as shown in Figure 6 As shown in . Among them, Figure 5 and Figure 6 The index buffer and the data verifier are used as an example to perform data interaction with the data reading controller as a peripheral module.

[0078] Step S22: if the data read request also carries verification type information, determining the verification information based on the verification type information; if the data read request does not carry verification type information, determining the verification information based on a preset reading mode of the data read controller.

[0079] In this embodiment, the data read request initiated by the upper user terminal can choose whether to carry the verification type information. Therefore, after obtaining the data read request, if the data read request carries the verification type information, the verification information is determined based on the verification type information. If the data read request does not carry the verification type information, the verification information is determined based on the preset reading mode of the data read controller. In this way, when the data read request does not carry the verification type information, the verification can also be performed according to the preset reading mode configured by the system to respond to the user's fast reading requirements or high consistency requirements.

[0080] In a specific implementation, the verification type information includes a first verification type and a second verification type, and the verification types corresponding to the preset read mode are the first verification type and the second verification type; wherein, the verification information corresponding to the first verification type is the first offset information carried in the data read request, and the verification information corresponding to the second verification type includes the first offset information carried in the data read request and the first verification code obtained by calculating the data information in the target data block using a preset verification algorithm. It can be understood that the integrity check needs to include at least the first index value and the offset information, and can also include the verification code. When the verification information does not include the verification code, the verification process of the verification code can be executed by the upper layer service of the user end after the target database is sent to the user end, for example, by the application program that initiates the data read request. In this way, when the upper layer service of the user end performs the verification code verification, the data reading controller only needs to perform two verifications, thereby speeding up the reading speed; when the application does not perform the verification code verification, before the data is returned to the user end, three verifications including the verification code verification need to be completed to ensure data consistency and accuracy.

[0081] In a specific embodiment, when the verification information is the first offset information, the data integrity check field in the target data block is verified based on the first index value and the verification information, including: parsing the data integrity check field in the target data block to obtain a second index value and a second offset information; determining whether the second index value is consistent with the first index value, and determining whether the second offset information is consistent with the first offset information; if the second index value is consistent with the first index value, and the second offset information is consistent with the first offset information, the verification is successful, otherwise the verification fails.

[0082] First, the layout of data in the storage medium is introduced, such as Figure 7 As shown, the read target data block contains not only the actual data information, but also the corresponding data integrity check field. The data integrity check field specifically includes the index value, offset information and check value corresponding to the data information, where the index value is the unique number of the target data, and generally every 4KB of data carries a DIF number.

[0083] When the verification information is the first offset information, the data integrity check field in the target data block is verified based on the first index value and the first offset information. The specific steps are to parse the data integrity check field in the target data block to obtain the second index value and the second offset information, and then respectively determine whether the second index value is consistent with the first index value, and whether the second offset information is consistent with the first offset information. If the second index value is consistent with the first index value, and the second offset information is consistent with the first offset information, the verification is successful, otherwise the verification fails. That is, the present application verifies whether the index value and offset information in the DIF are consistent with the index value and offset information carried in the request by parsing the DIF in the target data block. If they are consistent, the reading is considered successful and the target data block is returned to the user end. If there is any inconsistency, the read consistency mechanism is triggered, that is, a data read request is sent to the back-end storage unit, and then the data read is guaranteed to be consistent through a more complex internal consistency processing mechanism.

[0084] In another specific embodiment, when the verification information is the first offset information and the first check code, the verification of the data integrity check field in the target data block based on the first index value and the check information includes: parsing the data integrity check field in the target data block to obtain a second check code, a second index value, and a second offset information; determining whether the second check code is consistent with the first check code, and determining whether the second index value and the second offset information are respectively consistent with the first index value and the first offset information; if the first check code is consistent with the second check code, and the second index value and the second offset information are respectively consistent with the first index value and the first offset information, the verification is successful, otherwise the verification fails.

[0085] That is, when the verification information is the first offset information and the first check code, the data integrity check field in the target data block is verified based on the first index value, the first offset information and the first check code. The specific steps are to parse the data integrity check field in the target data block to obtain the second check code, the second index value and the second offset information, and then determine whether the second check code is consistent with the first check code, and determine whether the second index value and the second offset information are respectively consistent with the first index value and the first offset information; if the first check code is consistent with the second check code, and the second index value and the second offset information are respectively consistent with the first index value and the first offset information, then it means that the verification is successful, otherwise the verification fails. In other words, only when the first index value, the first offset information and the first check code carried in the data read request are respectively consistent with the second index value, the second offset information and the second check code in the DIF, it means that the verification has passed, and the data read controller will then return the complete and consistent data to the user end.

[0086] In addition, the above method also includes: when storing the data information in the storage medium, the data information is calculated using the preset verification algorithm to obtain the second verification code, and the second verification code is written into the data integrity verification domain; wherein the preset verification algorithm is a cyclic redundancy check algorithm or a message digest algorithm. That is, when the data information is stored in the storage medium, the data information has been calculated using the preset verification algorithm to obtain the first verification code, and the first verification code is written into the data integrity verification domain. Specifically, the verification code of the data is generally calculated once every 4KB and written into the DIF. Among them, the preset verification algorithm adopted is a cyclic redundancy check algorithm (Cyclic redundancy check, i.e. CRC) or a message digest algorithm (Message-Digest Algorithm 5, i.e. MD5).

[0087] It can be seen that in the present application, the data read request initiated by the upper user terminal can choose whether to carry the verification type information. When the data read request does not carry the verification type information, it can also be verified according to the preset reading mode configured by the system to respond to the user's fast reading requirements or high consistency requirements. In addition, the integrity check needs to include at least the first index value and the offset information, and can also include a check code. When the verification information does not include the check code, the verification process of the check code can be executed by the upper-layer business of the user terminal after the target database is sent to the user terminal. In this way, when the upper-layer business of the user terminal performs the check code verification, the data reading controller only needs to perform two checks, thereby speeding up the reading speed; when the application does not perform the check code verification, before the data is returned to the user terminal, three checks including the check code verification need to be completed to ensure data consistency and accuracy.

[0088] See also Figure 8 As shown, the embodiment of the present application also discloses a data reading device, including:

[0089] A data reading module 11 is used to determine a target physical address corresponding to a target data block based on the received first index value, and read the target data block from a storage medium of a back-end storage unit according to the target physical address;

[0090] The verification module 12 is used to verify the data integrity verification field in the target data block based on the first index value and the verification information, and if the verification is successful, return the target data block to the user end.

[0091] It can be seen that the present application discloses a data reading module, which is used to determine the target physical address corresponding to the target data block based on the received first index value, and read the target data block from the storage medium of the back-end storage unit according to the target physical address; a verification module, which is used to verify the data integrity verification field in the target data block based on the first index value and verification information, and if the verification is successful, the target data block is returned to the user end. It can be seen that the present application first determines the target physical address corresponding to the target data block based on the received first index value, and then reads the target data block from the storage medium of the back-end storage unit according to the target physical address. Further, the first index value and verification information are used to verify the data integrity verification field in the target data block. If the verification is successful, the data block is returned to the user end to implement an optimistic locking mechanism based on the data integrity verification field in the data read-through scenario. Through the above scheme, there is no need to impose additional restrictions on the data, which solves the I / O conflict problem between the data reading operation and other modification operations and improves the performance of data reading.

[0092] In some specific embodiments, the data reading device further includes:

[0093] A parsing unit, configured to parse the data read request sent by the client to obtain the first index value;

[0094] A verification information determination unit is used to determine the verification information based on the verification type information if the data read request also carries the verification type information, and to determine the verification information based on a preset reading mode of the data read controller if the data read request does not carry the verification type information.

[0095] In some specific embodiments, the verification type information includes a first verification type and a second verification type, and the verification types corresponding to the preset read mode are the first verification type and the second verification type; wherein, the verification information corresponding to the first verification type is the first offset information carried in the data read request, and the verification information corresponding to the second verification type includes the first offset information carried in the data read request and a first verification code obtained by calculating the data information in the target data block using a preset verification algorithm.

[0096] In some specific embodiments, when the verification information is the first offset information, the verification module 12 may specifically include:

[0097] A first parsing unit, configured to parse the data integrity check field in the target data block to obtain a second index value and a second offset information;

[0098] a first judging unit, configured to judge whether the second index value is consistent with the first index value, and judge whether the second offset information is consistent with the first offset information;

[0099] The first result determination unit is used to verify that the verification is successful if the second index value is consistent with the first index value and the second offset information is consistent with the first offset information, and that the verification fails otherwise.

[0100] In some specific embodiments, when the verification information is the first offset information and the first verification code, the verification module 12 may specifically include:

[0101] A second parsing unit, configured to parse the data integrity check field in the target data block to obtain a second check code, a second index value, and second offset information;

[0102] a second judgment unit, configured to judge whether the second check code is consistent with the first check code, and judge whether the second index value and the second offset information are consistent with the first index value and the first offset information respectively;

[0103] The second result determination unit is used to determine that the verification is successful if the first verification code is consistent with the second verification code, and the second index value and the second offset information are respectively consistent with the first index value and the first offset information; otherwise, the verification fails.

[0104] In some specific embodiments, the data reading device is also used to calculate the data information using the preset verification algorithm to obtain the second verification code when storing the data information into the storage medium, and write the second verification code into the data integrity check field; wherein the preset verification algorithm is a cyclic redundancy check algorithm or an information digest algorithm.

[0105] In some specific embodiments, after verifying the data integrity check field in the target data block based on the first index value and the check information, the device is also used to read the target data block from the storage medium of the back-end storage unit based on a preset read consistency rule if the check fails, and return the target data block to the user end; wherein the preset read consistency rule is a serialization technology or a multi-version concurrency control technology.

[0106] In some specific embodiments, the data reading module 11 may specifically include:

[0107] A search unit, configured to search an index buffer for a physical address corresponding to the first index value; the index buffer stores a mapping relationship between the index value and the physical address;

[0108] The address determining unit is used to determine the physical address corresponding to the first index value as the target physical address corresponding to the target data block.

[0109] In some specific embodiments, the data reading device is also used to read the target data block from the storage medium of the back-end storage unit based on a preset read consistency rule and return the target data block to the user end if the first index value does not exist in the index buffer; wherein the preset read consistency rule is a serialization technology or a multi-version concurrency control technology; parse the data integrity check field in the target data block to obtain the corresponding index information, and add the index information to the index buffer to update the index buffer.

[0110] In some specific embodiments, the data reading device further includes:

[0111] An index acquisition unit is used to send the index update request sent by the user end to the back-end storage unit when the index update request is obtained, so that the back-end storage unit returns the corresponding index information based on the data to be read in the index update request

[0112] The index updating unit is used to add the index information corresponding to the data to be read to the index buffer to update the index buffer.

[0113] The present application also discloses a method such as Figure 5The distributed storage system shown includes multiple clients, a server, and multiple storage terminals, wherein the server is connected to multiple clients respectively, and the server is connected to multiple storage terminals respectively; any client includes a first data reading controller and a first data verifier, wherein the first data reading controller is connected to the server and the first data verifier respectively, and the first data reading controller is also used to connect to the user terminal in the client; each storage terminal includes at least one storage medium, and the multiple storage terminals are deployed in a distributed manner; the first data verifier is used to verify the data integrity verification domain in the data block based on the index value and verification information, and send the verification result and the data block to the first data reading controller; the first data reading controller is used to execute the data reading method disclosed in any of the aforementioned embodiments. In addition, the first data verifier can be integrated into the first data reading controller, or it can be used as an external module to interact with the data reading controller.

[0114] The present application also discloses a method such as Figure 6 The electronic device shown may be a single-node storage host or a single-node storage server, including a front-end service layer, a transmission service layer and a back-end storage layer connected in sequence; the front-end service layer includes a second data reading controller and a second data verifier, the second data reading controller is connected to the transmission service layer and the second data verifier respectively, and the second data reading controller is also used to connect to the user end; the back-end storage layer includes at least one locally deployed storage medium; the second data verifier is used to verify the data integrity verification field in the data block based on the first index value and the verification information, and send the verification result and the target data block to the second data reading controller; the second data reading controller is used to execute the data reading method disclosed in any of the aforementioned embodiments. In addition, the second data verifier can be integrated into the second data reading controller, or it can be used as a peripheral module to interact with the data reading controller.

[0115] Furthermore, an embodiment of the present application also discloses a storage medium, in which a computer program is stored. When the computer program is loaded and executed by a processor, the steps of the data reading method disclosed in any of the aforementioned embodiments are implemented.

[0116] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0117] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0118] The data reading method, device, equipment and storage medium provided by the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A data reading method, characterized in that: include: Determine a target physical address corresponding to the target data block based on the received first index value, and read the target data block from a storage medium of a backend storage unit according to the target physical address; The data integrity check field in the target data block is checked based on the first index value and the check information, and if the check succeeds, the target data block is returned to the user end.

2. The data reading method according to claim 1, characterized in that: Also includes: Parsing the data read request sent by the client to obtain the first index value; If the data read request also carries verification type information, the verification information is determined based on the verification type information; if the data read request does not carry verification type information, the verification information is determined based on a preset reading mode of the data read controller.

3. The data reading method according to claim 2, characterized in that: The check type information includes a first check type and a second check type, and the check types corresponding to the preset read mode are the first check type and the second check type; wherein, the check information corresponding to the first check type is the first offset information carried in the data read request, and the check information corresponding to the second check type includes the first offset information carried in the data read request and a first check code obtained by calculating the data information in the target data block using a preset check algorithm.

4. The data reading method according to claim 3, characterized in that: When the check information is the first offset information, checking the data integrity check field in the target data block based on the first index value and the check information includes: Parsing the data integrity check field in the target data block to obtain a second index value and second offset information; Determine whether the second index value is consistent with the first index value, and determine whether the second offset information is consistent with the first offset information; If the second index value is consistent with the first index value, and the second offset information is consistent with the first offset information, the verification succeeds, otherwise the verification fails.

5. The data reading method according to claim 3, characterized in that: When the check information is the first offset information and the first check code, checking the data integrity check field in the target data block based on the first index value and the check information includes: Parsing the data integrity check field in the target data block to obtain a second check code, a second index value, and second offset information; Determine whether the second check code is consistent with the first check code, and determine whether the second index value and the second offset information are consistent with the first index value and the first offset information respectively; If the first check code is consistent with the second check code, and the second index value and the second offset information are respectively consistent with the first index value and the first offset information, the check succeeds; otherwise, the check fails.

6. The data reading method according to claim 5, characterized in that: Also includes: When storing the data information into the storage medium, the data information is calculated using the preset verification algorithm to obtain the second verification code, and the second verification code is written into the data integrity verification field; wherein the preset verification algorithm is a cyclic redundancy check algorithm or an information summary algorithm.

7. The data reading method according to claim 1, characterized in that: After verifying the data integrity check field in the target data block based on the first index value and the check information, the method further includes: If the verification fails, the target data block is read from the storage medium of the backend storage unit based on a preset read consistency rule, and the target data block is returned to the user end; wherein the preset read consistency rule is a serialization technology or a multi-version concurrency control technology.

8. The data reading method according to any one of claims 1 to 7, characterized in that: The determining a target physical address corresponding to the target data block based on the received first index value includes: Searching for a physical address corresponding to the first index value from an index buffer; the index buffer stores a mapping relationship between the index value and the physical address; The physical address corresponding to the first index value is determined as the target physical address corresponding to the target data block.

9. The data reading method according to claim 8, characterized in that: The process of searching the index buffer for the physical address corresponding to the first index value further includes: If the first index value does not exist in the index buffer, the target data block is read from the storage medium of the back-end storage unit based on a preset read consistency rule, and the target data block is returned to the user end; wherein the preset read consistency rule is a serialization technology or a multi-version concurrency control technology; The data integrity check field in the target data block is parsed to obtain corresponding index information, and the index information is added to the index buffer to update the index buffer.

10. The data reading method according to claim 8, characterized in that: Also includes: When an index update request sent by the user is obtained, the index update request is sent to the back-end storage unit so that the back-end storage unit returns the corresponding index information based on the data to be read in the index update request. The index information corresponding to the data to be read is added to the index buffer to update the index buffer.

11. A data reading device, characterized in that: include: A data reading module, configured to determine a target physical address corresponding to a target data block based on the received first index value, and read the target data block from a storage medium of a back-end storage unit according to the target physical address; A verification module is used to verify the data integrity verification field in the target data block based on the first index value and the verification information, and if the verification is successful, return the target data block to the user end.

12. A distributed storage system, characterized in that: It includes multiple clients, a server, and multiple storage terminals, wherein the server is connected to multiple clients respectively, and the server is connected to multiple storage terminals respectively; Any client includes a first data reading controller and a first data verifier, the first data reading controller is connected to the server and the first data verifier respectively, and the first data reading controller is also used to connect to the user end in the client; each storage end includes at least one storage medium, and the multiple storage ends are deployed in a distributed manner; The first data checker is used to check the data integrity check field in the data block based on the index value and the check information, and send the check result and the data block to the first data reading controller; The first data reading controller is used to execute the data reading method according to any one of claims 1 to 10.

13. An electronic device, characterized in that: It includes a front-end service layer, a transmission service layer and a back-end storage layer connected in sequence; The front-end service layer includes a second data reading controller and a second data verifier, the second data reading controller is connected to the transmission service layer and the second data verifier respectively, and the second data reading controller is also used to connect to the user end; the back-end storage layer includes at least one storage medium deployed locally; The second data checker is used to check the data integrity check field in the data block based on the index value and the check information, and send the check result and the data block to the second data reading controller; The second data reading controller is used to execute the data reading method according to any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that: Used to store computer executable instructions, which, when loaded and executed by a processor, implement the data reading method according to any one of claims 1 to 10.

Images