Data reading processing method and device based on Ceph storage system

By segmenting and verifying the data in the Ceph storage system, and directly outputting or repairing abnormal sharded data, the problems of silent data corruption inspection lag and low repair efficiency are solved, real-time and efficient data repair is achieved.

CN118981391AInactive Publication Date: 2024-11-19CHINA TELECOM ARTIFICIAL INTELLIGENCE TECHNOLOGY (BEIJING) CO LTD

Patent Information

Application Number
CN202410990707.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-22
Publication Date
2024-11-19
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The prior art has a lag when checking for corruption of silent data, resulting in low repair efficiency of silent damaged data and cannot be automatically or intervally restored.

Method used

By segmenting the target data in the Ceph storage system, using a preset verification algorithm to calculate the verification information of the sharded data, and search and compare the verification information in the key-value pair table to directly output or repair abnormal sharded data to ensure data consistency.

Benefits of technology

It avoids performance losses caused by full data inspection and recovery, realizes real-time detection and repair of silent data corruption, and improves data repair efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118981391A_ABST
    Figure CN118981391A_ABST
Patent Text Reader

Abstract

The invention discloses a data reading processing method and device based on a Ceph storage system. The method comprises the following steps: in response to a data reading request initiated by a client, obtaining target data corresponding to the data reading request from a Ceph storage system; segmenting the target data to obtain multiple pieces of fragmented data, and performing operation on each piece of fragmented data by adopting a preset verification algorithm to obtain first verification information corresponding to each piece of fragmented data; taking the object name of the object where the fragmented data is located and the offset of the fragmented data in the corresponding object as keys, and searching second verification information corresponding to each piece of fragmented data in a preset key value pair table; and executing corresponding reading processing operation on the target data according to a first comparison result of the first verification information and the second verification information of each piece of fragmented data. According to the method and the device, the technical problem that the restoration efficiency of the silent damaged data is relatively low due to hysteresis when the silent data damage is checked in the prior art is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of big data management technology, and more specifically, to a data reading and processing device based on a Ceph storage system. Background Art

[0002] With the continuous development of emerging technologies such as AI and big data, massive amounts of business data need to be stored in large-scale storage clusters. However, during the data storage process, silent data corruption (SDC) may occur due to disk bad sectors, data bit flipping, etc., resulting in data inconsistency, which seriously affects the normal use of customers.

[0003] Currently, there are several solutions for silent data corruption recovery:

[0004] Solution 1: Use global scrub scanning to check for silent data corruption, periodically compare the data of each replica, read the master and slave data, complete data consistency check, and when the data is abnormal, the inconsistent data will be collected and reported, and the data will be fully restored manually. However, this method still has the following problems: (1) The periodic check mechanism causes a lag in recovery time, that is, when silent data corruption has occurred but has not been scanned, the user cannot perceive the abnormality when reading the data. Therefore, the data provided to the user is erroneous data, resulting in serious consequences; (2) The full check requires reading all global data, which not only squeezes normal business traffic, but also wastes resources and affects the performance of the big data storage cluster; (3) Silent data corruption requires manual full recovery, so this solution can neither automatically recover nor recover data intervals.

[0005] Solution 2: Use parallel program failure detection to detect silent data corruption, that is, the central monitor sends a heartbeat check request through a periodic timer, obtains hardware fault check information of all nodes through the distributed status monitoring system, and determines whether silent data corruption has occurred and the hardware where silent data corruption has occurred. However, this method still has the following problems: (1) There is a lag in the use of periodic timers for checking, that is, when silent data corruption has occurred but has not been scanned, the user cannot perceive the abnormality when reading the data; (2) The granularity of the silent data corruption check is hardware, so it is impossible to accurately locate the specific location where the data anomaly occurs.

[0006] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0007] The embodiment of the present application provides a data reading processing and device based on the Ceph storage system to at least solve the technical problem that the related technology has a lag when checking for silent data damage, resulting in low efficiency in repairing silently damaged data.

[0008] According to one aspect of an embodiment of the present application, a data reading processing method based on a Ceph storage system is provided, comprising: in response to a data reading request initiated by a client, obtaining target data corresponding to the data reading request from the Ceph storage system; dividing the target data to obtain a plurality of shard data, and using a preset verification algorithm to operate on each shard data to obtain first verification information corresponding to each shard data; using the object name of the object where the shard data is located and the offset of the shard data in the corresponding object as a key, searching for second verification information corresponding to each shard data in a preset key-value pair table, wherein the key-value pair table stores the second verification information corresponding to each shard data. A plurality of key-value pairs are stored, and the key-value pairs are based on the object name of the object where each shard data obtained by splitting the target data and the offset of each shard data in the corresponding object as the key in the key-value pair when writing the target data, and the second verification information obtained by operating each shard data by using a preset verification algorithm is used as the value in the key-value pair; a first comparison result between the first verification information and the second verification information of each shard data is determined, and a corresponding read processing operation is performed on the target data according to the first comparison result of each shard data, wherein the read processing operation includes: direct output or output after repairing the abnormal shard data in the target data.

[0009] Optionally, before obtaining the target data corresponding to the data read request from the Ceph storage system, the method also includes: responding to a data write request initiated by the client, wherein the data write request carries at least the target data to be written; starting from the starting position of the target data, dividing the target data according to a preset alignment granularity to obtain multiple shard data, and encapsulating each shard data into multiple objects respectively, determining the object name of the object where the multiple shard data are located and the offset of the shard data in the corresponding object; using a preset verification algorithm to operate on each shard data to obtain second verification information of each shard data; using the object name of the object where each shard data is located and the offset of each shard data in the corresponding object as a key and the second verification information of each shard data as a value to form multiple key-value pairs, and the multiple key-value pairs form a key-value pair table; storing the target data and the key-value pair table in multiple copies in the Ceph storage system in a multi-copy manner, wherein the types of copies include: primary copy and secondary copy.

[0010] Optionally, after obtaining multiple shard data, the method also includes: storing the multiple shard data in a temporary data cache list respectively; determining the position of each shard data in the target data; recording the position of each shard data in the target data and the correspondence between each shard data and the temporary data cache list in a memory shard mapping table.

[0011] Optionally, a first comparison result of the first verification information and the second verification information of each shard data is determined, and a corresponding read processing operation is performed on the target data based on the first comparison result of each shard data, including: based on the position of each shard data recorded in the memory shard mapping table in the target data, the first verification information and the second verification information of each shard data are compared in turn to obtain a corresponding first comparison result; when the first comparison results of each shard data are the same, it is determined that each shard data of the target data is in a normal state, and the target data is directly output; when the first comparison results of any shard data are different, it is determined that the shard data is abnormal shard data, the abnormal shard data is repaired using the secondary copy of the target data, and the target data is updated according to the repaired abnormal shard data, and the updated target data is output.

[0012] Optionally, directly outputting the target data includes: taking the temporary data cache list corresponding to the first shard data of the target data as the benchmark cache list, and appending the temporary data cache list corresponding to each other shard data to the end of the benchmark cache list in turn according to the position of each shard data in the target data recorded in the memory shard mapping table, to obtain the target data cache list, and performing data processing on the target data through the target data cache list, wherein the other shard data are shard data other than the first shard data among the multiple shard data obtained by splitting the target data.

[0013] Optionally, the abnormal shard data is repaired using the secondary copy of the target data, and the target data is updated according to the repaired abnormal shard data, including: traversing each secondary copy of the target data, and executing the following steps: obtaining interval data corresponding to the abnormal shard data from the secondary copy according to the data length of the abnormal shard data, the object name of the object where the abnormal shard data is located, and the offset of the abnormal shard data in the corresponding object, and using a preset verification algorithm to operate on the interval data to obtain corresponding third verification information; comparing the third verification information with the first target verification information of the abnormal shard data to obtain a second comparison result; when the second comparison result is the same, updating the data recorded in the temporary data cache list corresponding to the abnormal shard data recorded in the memory shard mapping table to the interval data to update the target data; when the second comparison result is different, continuing to use other secondary copies of the target data to repair the abnormal shard data.

[0014] Optionally, the preset check algorithm includes at least one of the following: a cyclic redundancy check algorithm, a Hamming code, and a parity check algorithm.

[0015] According to another aspect of the embodiment of the present application, a data reading and processing device based on a Ceph storage system is also provided, including: an acquisition module, which is used to respond to a data reading request initiated by a client and obtain target data corresponding to the data reading request from the Ceph storage system; a first determination module, which is used to divide the target data into multiple shard data, and use a preset verification algorithm to operate on each shard data to obtain first verification information corresponding to each shard data; a second determination module, which is used to use the object name of the object where the shard data is located and the offset of the shard data in the corresponding object as a key, and search for the second verification information corresponding to each shard data in a preset key-value pair table, wherein , a plurality of key-value pairs are stored in the key-value pair table, and the key-value pairs are the object name of the object where each shard data obtained by splitting the target data and the offset of each shard data in the corresponding object as the key in the key-value pair when writing the target data, and the second verification information obtained by operating each shard data by using a preset verification algorithm is used as the value in the key-value pair; a data reading and processing module is used to determine a first comparison result of the first verification information and the second verification information of each shard data, and perform corresponding reading and processing operations on the target data according to the first comparison result of each shard data, wherein the reading and processing operations include: direct output or output after repairing the abnormal shard data in the target data.

[0016] According to another aspect of an embodiment of the present application, a non-volatile storage medium is also provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned data reading and processing method based on the Ceph storage system by running the computer program.

[0017] According to another aspect of an embodiment of the present application, a computer program product is also provided, which includes a stored computer program, wherein when the computer program is executed by a processor, the above-mentioned data reading and processing method based on the Ceph storage system is implemented.

[0018] In an embodiment of the present application, when a client needs to read target data from a Ceph storage system, in order to verify whether silent data corruption occurs to the target data during the storage process, the target data can be divided into multiple shard data of the same size as the target data when the target data is written, and a preset verification algorithm is used to operate on each shard data to obtain the first verification information corresponding to each shard data; determine the first comparison result of the first verification information and the second verification information of each shard data, and perform a corresponding read processing operation on the target data according to the first comparison result, wherein the read processing operation includes: directly outputting or repairing the abnormal shard data in the target data before outputting. The above-mentioned data reading processing process can check each shard data of the data to be read, and repair the abnormal shard data with silent data corruption, thereby avoiding performance loss caused by full data inspection and full recovery, thereby solving the technical problem that the related technology has a lag in checking silent data corruption, resulting in low efficiency in repairing silently damaged data. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 It is a hardware structure block diagram of an optional computer terminal (mobile device) for implementing a data reading and processing method based on a Ceph storage system according to an embodiment of the present application;

[0021] Figure 2 It is a flowchart of an optional data reading and processing method based on a Ceph storage system according to an embodiment of the present application;

[0022] Figure 3 is a flow chart of generating an optional key-value pair according to an embodiment of the present application;

[0023] Figure 4 This is a schematic diagram of multiple copies of storage of an optional key-value pair according to an embodiment of the present application;

[0024] Figure 5 is a flow chart of an optional verification logic according to an embodiment of the present application;

[0025] Figure 6 It is a structural schematic diagram of an optional data reading and processing device based on a Ceph storage system according to an embodiment of the present application;

[0026] Figure 7 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] In addition, the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. For example, an interface is set up between this system and the relevant user or organization. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0030] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0031] SDC (Silent Data Corruption): refers to the storage system providing corrupted data to the application without any warning. For example, due to media damage, a bad block appears in the disk. When the application reads the block, it will read wrong data. Usually, silent data corruption may occur in the following places: software involved in operating, moving or storing data (such as disk drives, memory bars), and various levels of the storage stack (such as disk controllers, memory, switches, network interface cards, data processing units, host bus adapters, disk drives, solid-state drives).

[0032] CRC (Cyclic Redundancy Check): It is a hash function that generates a short fixed-bit check code based on network data packets or computer files to detect or verify errors that may occur after data transmission or storage. It usually consists of n data bits, and k check bits are obtained through mathematical polynomial calculation to achieve error detection and correction. It is widely used in data transmission verification and hard disk storage verification.

[0033] Ceph is a reliable, automatically rebalanced, and data-consistent distributed storage system, which is widely used in the field of big data storage. Among them, Object is the lowest-level storage unit in the Ceph storage system. Each Object contains metadata and original data. When users want to store data in the Ceph cluster, the stored data will be divided into multiple Objects; in addition, Bufferlist is the basic structure of Ceph storage system to transmit data in memory. It is used to temporarily store data and is responsible for managing all memory in the Ceph data flow processing process.

[0034] splic method: It is an important function in bufferlist. Its function is to move a part of the data in bufferlist to another bufferlist. The splic method needs to limit three parameters, namely: off (the starting offset in the current bufferlist), len (the length of the data to be moved), and claim_by (the target bufferlist). Therefore, the splic method can be interpreted as: starting from the offset off in the current bufferlist, move the data of the moving length len to the end of the target bufferlist specified by claim_by. This operation is a move operation, not a copy operation, that is, the original data will be deleted from the original bufferlist after being moved.

[0035] Append method: It is an important function in bufferlist, which is used to append data to the end of an existing data block or list.

[0036] Example 1

[0037] According to an embodiment of the present application, a method embodiment of a data reading processing method based on a Ceph storage system is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal (or mobile device) for implementing a data reading and processing method based on a Ceph storage system is shown. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0039] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0040] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data reading and processing method based on the Ceph storage system in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the data reading and processing method based on the Ceph storage system of the above-mentioned application is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0041] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0042] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0043] In the above operating environment, Figure 2 is a flow chart of an optional data reading and processing method based on a Ceph storage system according to an embodiment of the present application, such as Figure 2 As shown, the method at least includes steps S202-S208, wherein:

[0044] Step S202, in response to a data read request initiated by the client, obtain target data corresponding to the data read request from the Ceph storage system.

[0045] In the technical solution provided in step S202, when the user triggers a read operation on the target data on the client, the processing system can first obtain the target data from the primary copy of the Ceph storage system that stores the target data.

[0046] Step S204, dividing the target data into multiple slice data, and using a preset verification algorithm to calculate each slice data to obtain first verification information corresponding to each slice data.

[0047] In the technical solution provided in step S204, when writing the target data to the Ceph storage system, in order to make full use of the storage space of each sector, the multiple shard data obtained after the target data is segmented are usually encapsulated into an object (Object), and each object is stored in the corresponding sector, and the object name of the object where each shard data is located and the offset in the object are recorded. At the same time, in order to facilitate the subsequent verification of whether each shard data has silent data corruption, the present application can also use a preset verification algorithm to calculate the second verification information corresponding to each shard data. Therefore, when reading the target data, in order to verify whether the target data has silent data corruption, the processing system can also segment the target data to obtain multiple shard data, and use a preset verification algorithm to calculate each shard data to obtain the first verification information corresponding to each shard data.

[0048] Step S206: using the object name of the object where the shard data is located and the offset of the shard data in the corresponding object as a key, searching for the second verification information corresponding to each shard data in the preset key-value pair table.

[0049] In the technical solution provided in step S206, the processing system can use the object name of the object where each shard data of the target data is located and the offset of the shard data in the corresponding object as the key when writing the target data, and match the key with a preset key-value pair table to obtain the second verification information of each shard data when writing, wherein the above-mentioned key-value pair table includes multiple key-value pairs, and the key-value pair uses the object name of the object where each shard data obtained by splitting the target data when writing and the offset of each shard data in the corresponding object as the key in the key-value pair, and uses the second verification information obtained by operating each shard data using a preset verification algorithm as the value in the key-value pair.

[0050] Step S208, determining a first comparison result of the first verification information and the second verification information of each shard data, and performing a corresponding read processing operation on the target data according to the first comparison result of each shard data.

[0051] In the technical solution provided in step S208, the processing system can compare the first verification information and the second verification information of each shard data in the target data to obtain a first comparison result of each shard data, wherein when the first comparison results of all shard data are the same, it indicates that no silent data corruption has occurred in the target data, and at this time the target data to be read can be directly fed back to the client; and when the first comparison results of any shard data are different, it indicates that the shard data is abnormal shard data, and at this time the abnormal shard data can be repaired, and the target data to be read can be updated according to the repaired abnormal shard data, so as to finally feed back the complete target data to the client.

[0052] The above method of this embodiment is further introduced below.

[0053] As an optional implementation, in the technical solution provided in the above step S202, before the processing system obtains the target data corresponding to the data read request from the Ceph storage system, it can also write the target data into the Ceph storage system according to the following method, including:

[0054] Step S1, responding to a data writing request initiated by a client, wherein the data writing request carries at least target data to be written.

[0055] Step S2, starting from the starting position of the target data, the target data is segmented according to the preset alignment granularity to obtain multiple shard data, and each shard data is encapsulated into multiple objects respectively, and the object name of the object where the multiple shard data are located and the offset of the shard data in the corresponding object are determined.

[0056] Specifically, in order to ensure that the size of the written data is aligned with the physical sectors of the storage device in the Ceph storage system, so as to reduce the sector boundary problem in the read and write operations and improve the efficiency of large data read and write, the target data to be written can be segmented according to the preset alignment granularity to obtain multiple shard data. It should be noted that since the storage nodes in the Ceph storage system, such as hard disk drives (HDDs) and solid-state drives (SSDs), are usually based on 4K (i.e., 4096 bytes) as a sector, therefore, in the embodiment of the present application, it is preferred to use 4K granularity (i.e., 4096 bytes) to segment the target data to be written.

[0057] Then, each shard data is encapsulated into the lowest-level storage unit of the Ceph storage system, namely, an object, wherein each object contains the original data of the shard data and metadata related to the shard data (i.e., information describing the characteristics and attributes of the shard data, such as creation timestamp, permissions, object name, object size, object version, etc.). Therefore, the processing system can determine the object name of the object where each shard data is located and the offset of the shard data in the corresponding object, wherein the offset is used by the client to read or write data in the object.

[0058] In addition, after the processing system divides the target data to be written into multiple shard data through the above step S2, the processing system can also: store the multiple shard data in the temporary data cache list respectively; determine the position of each shard data in the target data; record the position of each shard data in the target data and the corresponding relationship between each shard data and the temporary data cache list in the memory shard mapping table. In other words, the processing system stores each shard data after segmentation in the temporary bufferlist respectively, and records the position order of each shard data in the target data and the corresponding temporary bufferlist through the memory shard map.

[0059] Step S3, using a preset verification algorithm to operate on each slice data to obtain second verification information of each slice data.

[0060] Specifically, the processing system can operate on each shard data through a preset verification algorithm to obtain second verification information corresponding to each shard data, wherein the preset verification algorithm used above includes but is not limited to: cyclic redundancy check algorithm, Hamming code, parity check algorithm, etc.

[0061] Among them, the above-mentioned verification algorithms are all calculated using the slice data itself as the calculation data. Taking the cyclic redundancy check algorithm CRC32 as an example, the CRC check code (i.e., the second verification information) of the slice data can be calculated according to the following formula:

[0062] g(x)=x 32 +x 26 +x 23 +x 22 +x 16 +x 12 +x 11 +x 10 +x 8 +x 7 +x 5 +x 4 +x 2 +x+1

[0063] Among them, the remainder obtained by performing modulo-2 division of the binary polynomial corresponding to the slice data and g(x) is the CRC check code of the slice data. The same generating polynomial g(x) needs to be used when writing data and reading data to obtain the check code of the slice data.

[0064] Furthermore, the processing system can also use the append method of Ceph bufferlist to traverse each shard data of the target data according to the memory shard map, and use the temporary bufferlist corresponding to the first shard data of the target data as the benchmark bufferlist, and splice the temporary bufferlists corresponding to each shard data to the end of the benchmark bufferlist in sequence according to the position of the shard data in the target data. This process is repeated until all shard data positions are traversed, and finally the temporary bufferlist corresponding to the obtained target data as a whole is appended to the write data bufferlist in the Ceph storage system, wherein the write data bufferlist can also be used as the read data bufferlist during the data reading stage.

[0065] Step S4, using the object name of the object where each shard data is located and the offset of each shard data in the corresponding object as the key, and the second verification information of each shard data as the value, to form multiple key-value pairs, and the multiple key-value pairs form a key-value pair table.

[0066] Specifically, the processing system can use the object name of the object where the shard data is located and the offset of the shard data in the corresponding object as the key, and the second verification information of each shard data as the value to form multiple key-value pairs, and the multiple key-value pairs form a key-value pair table.

[0067] For example, Figure 3 is an optional key-value pair generation flow chart according to an embodiment of the present application, such as Figure 3As shown. For a data to be written into the Ceph storage system (data size is 8K), the data to be written is aligned and split into the corresponding temporary bufferlist according to the 4K granularity through the splic method of Cephbufferlist, and the CRC32c algorithm is used to calculate the two 4K fragments to obtain the corresponding CRC check codes, which are recorded as: CRC_val1, CRC_val2, and the two fragments are encapsulated into objectA, and the offset of the first fragment in objectA is 8K, and the offset of the second fragment in objectA is 16K, so that the object name of objectA where the two fragments are located and the offset in objectA are used as the key, and the CRC check codes of the two fragments are used as the value to form two key-value pairs:<objectA_8K;CRC_val1> ,<objectA_16K;CRC_val2> .

[0068] Step S5, storing the target data and the key-value pair table in multiple copies in the Ceph storage system in a multi-copy manner.

[0069] Specifically, the processing system can distribute the key-value pairs obtained in the above step S4 together with the target data to be written to multiple replicas in the Ceph storage system, wherein the types of the replicas include: primary replicas and secondary replicas. It should be noted that the division of the above primary replicas and secondary replicas is relative. If the user reads data directly from replica A, then replica A is the primary replica, and other replicas B storing the data to be read are all secondary replicas.

[0070] For example, Figure 4 is a schematic diagram of multiple copies of storage of an optional key-value pair according to an embodiment of the present application, such as Figure 4 As shown, it can be<objectA_8K;CRC_val1> Ceph_omap (i.e. object mapping, which is used to store distributed key-value pairs describing metadata information such as object properties) stored in multiple replicas. When all replicas have completed the write operation of the key-value pair and the target data, a prompt message indicating the successful write can be fed back to the client, thereby effectively preventing the loss of verification data.

[0071] Furthermore, after completing the write operation of the target data through the above steps, when the user triggers a read operation on the target data on the client, the processing system can first obtain the target data from the primary copy of the Ceph storage system that stores the target data, and divide the target data to be read according to the same alignment granularity as when writing the data to obtain multiple shard data, and obtain the object name of the object encapsulated by each shard data when writing and the offset (offset) in the object, and at the same time use a preset algorithm to operate on each shard data to obtain the first verification information corresponding to each shard data; using the object name of the object encapsulated by each shard data when writing and the offset (offset) in the object as the key value, search for the second verification information corresponding to the key name in the Ceph_omap in the primary copy, and compare whether the first verification information of each shard data when reading is the same as the second verification information of each shard data when writing.

[0072] For example, Figure 5 is a flow chart of an optional verification logic according to an embodiment of the present application, such as Figure 5 As shown, obtain the data from the primary copy in the Ceph storage system Figure 3 The target data is written and aligned and split into multiple temporary bufferlists according to the 4K granularity. The CRC32c algorithm is used to calculate the CRC check code of one of the shard data, recorded as: newCRC_val1, and the key name of the shard data objectA_8K is used to search the CRC check code CRC_val1 corresponding to the key name in Ceph_omap in the primary copy, and compare the two check information to see if they are the same.

[0073] As an optional implementation, in the technical solution provided in the above step S208, the processing system may read and process the target data according to the following rules, including:

[0074] According to the position of each shard data recorded in the memory shard mapping table in the target data, the first verification information and the second verification information of each shard data are compared in sequence to obtain a corresponding first comparison result;

[0075] The first case: if the first comparison results of each shard data are the same, it is determined that each shard data of the target data is in a normal state (ie, no silent data corruption occurs), and the processing system can directly output the target data.

[0076] The above situation can be understood as: when the first verification information and the second verification information of each shard data are verified in sequence and are the same, the processing system can use the temporary data cache list corresponding to the first shard data of the target data as the benchmark cache list, and according to the position of each shard data recorded in the memory shard mapping table in the target data, append the temporary data cache list corresponding to each other shard data to the end of the benchmark cache list in sequence to obtain the target data cache list (that is, the read data bufferlist), and perform data processing on the target data through the target data cache list, wherein the other shard data are the shard data other than the first shard data in the multiple shard data obtained by splitting the target data.

[0077] It should be noted that if the client needs to read multiple target data at the same time, the target data cache list (i.e., read data bufferlist) of each target data can be obtained in turn according to the above method, and the target data cache list of each target data can be spliced ​​in turn according to the order of reading each target data, and the spliced ​​read data bufferlist can be fed back to the client.

[0078] The second case: if the first comparison result of any shard data is different, it can be determined that there is silent data corruption in the shard data and it is marked as abnormal shard data. At this time, the processing system checks and repairs the abnormal shard data using the secondary copy of the stored target data, and updates the target data based on the repaired abnormal shard data, and finally outputs the updated target data.

[0079] Specifically, the processing system may traverse each secondary copy storing the target data, and execute the following steps in a loop until interval data identical to the first verification information of the abnormal shard data is found from the multiple secondary copies:

[0080] First, according to the target offset of the abnormal shard data in the corresponding object and the data length of the abnormal shard data, the interval data corresponding to the abnormal shard data is obtained from the secondary copy, and the interval data is calculated using a preset verification algorithm to obtain the corresponding third verification information;

[0081] Then, the third verification information is compared with the first target verification information of the abnormal shard data to obtain a second comparison result;

[0082] When the second comparison result is the same, the data recorded in the temporary data cache list corresponding to the abnormal shard data recorded in the memory shard mapping table is updated to the interval data to update the target data;

[0083] When the second comparison result is different, continue to use other secondary copies storing the target data to repair the abnormal shard data.

[0084] Therefore, the above repair steps can also be understood as: the processing system can first automatically trigger the CephRecovery process according to the object name of the object where the abnormal shard data with silent data corruption is located and the offset of the abnormal shard data in the corresponding object. At this time, the processing system can traverse the i-th (i=1, 2, ..., n, n is the total number of secondary copies) remote copy storing the target data, and obtain the corresponding interval data from the i-th remote copy according to the data length of the abnormal shard data, the object name of the object where the abnormal shard data is located, and the offset of the abnormal shard data in the corresponding object; then, use the preset verification algorithm to calculate the third verification information of the interval data; then, compare the first verification information corresponding to the key name of the abnormal shard data recorded in Ceph omap, wherein: if the two verification information are the same, replace the data in the temporary bufferlist of the abnormal shard data in the memory shard mapping table with the interval data of the remote copy; and if the two verification information are different, set i=i+1 to continue traversing the i+1-th remote copy for index loop verification until the interval data identical to the second verification information of the abnormal shard data is found for repair.

[0085] It should be noted that if there are multiple abnormal shard data in the target data, each abnormal shard data can be repaired in turn according to the above steps, and then according to the position of each shard data recorded in the memory shard mapping table and the bufferlist of each shard data, the bufferlist of the first shard data obtained by dividing the target data is used as the benchmark bufferlist, and the subsequent shard data is appended to the end of the benchmark bufferlist. This process is repeated until the memory shard mapping table is traversed, so that the read data bufferlist of the target data can be reconstructed.

[0086] Based on the technical solution provided by the above steps S202-S208, when the target data is written to the Ceph storage system, the data is segmented to obtain multiple shard data, and each shard data is encapsulated into an object, and the object name of the object where the shard data is located and the offset of the shard data in the corresponding object are used as the key in the key-value pair, and the second verification information obtained by operating the shard data using a preset verification algorithm is used as the value in the key-value pair to form multiple key-value pairs, and the key-value pair table composed of the multiple key-value pairs and the target data are stored in multiple copies in the Ceph storage system. When reading target data from the Ceph storage system, in order to verify whether silent data damage occurs to the target data during the storage process, the target data can be segmented to obtain multiple shard data, and the first verification information obtained by operating each shard data using a preset verification algorithm is obtained; the corresponding reading processing operation is performed on the target data according to the first comparison result of the first verification information and the second verification information of each shard data, wherein, when the first comparison results corresponding to all shard data are the same, it means that the target data has not suffered silent data damage, and the target data to be read can be directly output at this time; and when the first comparison results of any shard data are different, it means that silent data loss has occurred to the shard data, and the shard data is marked as abnormal shard data, and the abnormal shard data in the target data can be repaired and then output. The above data reading processing process can check each shard data of the data to be read, and repair the abnormal shard data with silent data damage, avoiding the performance loss caused by full data inspection and full recovery, thereby solving the technical problem that the related technology has hysteresis when checking silent data damage, resulting in low efficiency in repairing silent damaged data.

[0087] Example 2

[0088] Based on Embodiment 1 of the present application, an embodiment of a data reading and processing device based on a Ceph storage system is also provided. When the device is running, the data reading and processing method based on a Ceph storage system of the above embodiment is executed. Figure 6 is a schematic diagram of a structure of an optional data reading and processing device based on a Ceph storage system according to an embodiment of the present application, such as Figure 6 As shown, the data reading processing device based on the Ceph storage system includes at least an acquisition module 62, a first determination module 64, a second determination module 66 and a data reading processing module 68, wherein:

[0089] The acquisition module 62 is used to respond to the data read request initiated by the client and obtain the target data corresponding to the data read request from the Ceph storage system;

[0090] The first determination module 64 is used to divide the target data into multiple pieces of data, and use a preset verification algorithm to calculate each piece of data to obtain first verification information corresponding to each piece of data;

[0091] The second determination module 66 is used to use the object name of the object where the shard data is located and the offset of the shard data in the corresponding object as a key to search for the second verification information corresponding to each shard data in a preset key-value pair table, wherein the key-value pair table stores multiple sets of key-value pairs, and the key-value pair uses the object name of the object where each shard data is located obtained by splitting the target data when writing the target data and the offset of each shard data in the corresponding object as the key in the key-value pair, and uses the second verification information obtained by calculating each shard data using a preset verification algorithm as the value in the key-value pair;

[0092] The data reading and processing module 68 is used to determine the first comparison result of the first verification information and the second verification information of each shard data, and perform corresponding reading and processing operations on the target data according to the first comparison result of each shard data, wherein the reading and processing operations include: directly outputting or repairing the abnormal shard data in the target data before outputting.

[0093] It should be noted that the various modules in the above-mentioned data reading and processing device based on the Ceph storage system can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0094] Example 3

[0095] According to an embodiment of the present application, a non-volatile storage medium is also provided, in which a program is stored, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the data reading and processing method based on the Ceph storage system in Example 1.

[0096] Optionally, the device where the non-volatile storage medium is located implements the following steps by running the program: in response to a data read request initiated by a client, obtain target data corresponding to the data read request from the Ceph storage system; divide the target data into multiple shard data, and use a preset verification algorithm to operate on each shard data to obtain first verification information corresponding to each shard data; use the object name of the object where the shard data is located and the offset of the shard data in the corresponding object as keys, and search for second verification information corresponding to each shard data in a preset key-value pair table, wherein the key-value pair table stores multiple groups of keys. A key-value pair is a key-value pair in which the object name of each shard data obtained by splitting the target data and the offset of each shard data in the corresponding object are used as the key in the key-value pair, and the second verification information obtained by operating each shard data by using a preset verification algorithm is used as the value in the key-value pair; a first comparison result between the first verification information and the second verification information of each shard data is determined, and a corresponding read processing operation is performed on the target data according to the first comparison result of each shard data, wherein the read processing operation includes: direct output or output after repairing the abnormal shard data in the target data.

[0097] According to an embodiment of the present application, a computer program product is also provided, which includes a stored computer program, wherein when the computer program is executed by a processor, the data reading and processing method based on the Ceph storage system in Example 1 is implemented.

[0098] Optionally, the computer program executes the following steps: in response to a data read request initiated by a client, obtaining target data corresponding to the data read request from the Ceph storage system; dividing the target data to obtain multiple shard data, and using a preset verification algorithm to operate on each shard data to obtain first verification information corresponding to each shard data; using the object name of the object where the shard data is located and the offset of the shard data in the corresponding object as keys, searching for second verification information corresponding to each shard data in a preset key-value pair table, wherein the key-value pair table stores multiple groups of key-value pairs, and the key-value pairs are the object names of the objects where the shard data is located when the target data is written, and the offsets of each shard data in the corresponding object as keys in the key-value pairs, and the second verification information obtained by operating each shard data using a preset verification algorithm as values ​​in the key-value pairs; determining a first comparison result of the first verification information and the second verification information of each shard data, and performing corresponding read processing operations on the target data based on the first comparison results of each shard data, wherein the read processing operations include: directly outputting or repairing abnormal shard data in the target data before outputting.

[0099] According to an embodiment of the present application, a processor is also provided, which is used to run a program, wherein the data reading processing method based on the Ceph storage system in Example 1 is executed when the program is running.

[0100] Optionally, the following steps are executed when the program is running: in response to a data read request initiated by a client, target data corresponding to the data read request is obtained from the Ceph storage system; the target data is segmented to obtain multiple shard data, and each shard data is operated by a preset verification algorithm to obtain first verification information corresponding to each shard data; the object name of the object where the shard data is located and the offset of the shard data in the corresponding object are used as keys to search for second verification information corresponding to each shard data in a preset key-value pair table, wherein the key-value pair table stores multiple groups of key-value pairs, and the key-value pairs are the object names of the objects where each shard data is located when the target data is segmented and the offset of each shard data in the corresponding object as the keys in the key-value pairs, and the second verification information obtained by operating each shard data by using a preset verification algorithm is used as the value in the key-value pairs; a first comparison result of the first verification information and the second verification information of each shard data is determined, and a corresponding read processing operation is performed on the target data according to the first comparison result of each shard data, wherein the read processing operation includes: directly outputting or repairing the abnormal shard data in the target data before outputting.

[0101] According to an embodiment of the present application, an electronic device is also provided, wherein: Figure 7 is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application, such as Figure 7 As shown, the electronic device includes one or more processors; a memory for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors are implemented to run the programs, wherein the programs are configured to execute the data reading processing method based on the Ceph storage system in the above-mentioned embodiment 1 when running.

[0102] Optionally, the processor is configured to implement the following steps through a computer program: in response to a data read request initiated by a client, obtain target data corresponding to the data read request from the Ceph storage system; divide the target data into multiple shard data, and use a preset verification algorithm to operate on each shard data to obtain first verification information corresponding to each shard data; use the object name of the object where the shard data is located and the offset of the shard data in the corresponding object as a key, and search for second verification information corresponding to each shard data in a preset key-value pair table, wherein the key-value pair table stores multiple sets of key-value pairs. The key-value pair uses the object name of the object where each shard data obtained by splitting the target data and the offset of each shard data in the corresponding object as the key in the key-value pair when writing the target data, and uses the second verification information obtained by operating each shard data using a preset verification algorithm as the value in the key-value pair; determines a first comparison result between the first verification information and the second verification information of each shard data, and performs corresponding read processing operations on the target data based on the first comparison result of each shard data, wherein the read processing operation includes: direct output or output after repairing the abnormal shard data in the target data.

[0103] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0104] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0105] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0106] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0107] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0108] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0109] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data reading and processing method based on a Ceph storage system, characterized in that: include: In response to a data read request initiated by a client, obtaining target data corresponding to the data read request from the Ceph storage system; The target data is segmented to obtain a plurality of shard data, and each shard data is operated by using a preset verification algorithm to obtain first verification information corresponding to each shard data; Using the object name of the object where the shard data is located and the offset of the shard data in the corresponding object as a key, searching for the second verification information corresponding to each shard data in a preset key-value pair table, wherein the key-value pair table stores a plurality of key-value pairs, and the key-value pairs are the object name of the object where each shard data is located obtained by splitting the target data when writing the target data and the offset of each shard data in the corresponding object as the key in the key-value pair, and the second verification information obtained by calculating each shard data by using a preset verification algorithm is used as the value in the key-value pair; Determine a first comparison result of the first verification information and the second verification information of each of the shard data, and perform a corresponding read processing operation on the target data according to the first comparison result of each of the shard data, wherein the read processing operation includes: directly outputting or repairing the abnormal shard data in the target data before outputting.

2. The method according to claim 1, characterized in that Before obtaining the target data corresponding to the data read request from the Ceph storage system, the method further includes: Responding to a data write request initiated by a client, wherein the data write request carries at least the target data to be written; Starting from the starting position of the target data, the target data is segmented according to a preset alignment granularity to obtain a plurality of shard data, and each of the shard data is respectively encapsulated into a plurality of objects, and the object names of the objects where the plurality of shard data are located and the offsets of the shard data in the corresponding objects are determined; Using the preset verification algorithm to operate on each of the slice data to obtain second verification information of each of the slice data; Taking the object name of the object where each of the shard data is located and the offset of each of the shard data in the corresponding object as a key, and taking the second verification information of each of the shard data as a value, a plurality of key-value pairs are formed, and the key-value pair table is formed by the plurality of key-value pairs; The target data and the key-value pair table are stored in multiple copies in the Ceph storage system in a multi-copy manner, wherein the types of the copies include: primary copies and secondary copies.

3. The method according to claim 2, characterized in that After obtaining the plurality of shard data, the method further includes: Storing the plurality of slice data in a temporary data cache list respectively; Determine the position of each of the slice data in the target data; The position of each of the shard data in the target data and the corresponding relationship between each of the shard data and the temporary data cache list are recorded in a memory shard mapping table.

4. The method according to claim 3, characterized in that Determining a first comparison result between the first verification information and the second verification information of each of the shard data, and performing a corresponding read processing operation on the target data according to the first comparison result of each of the shard data, including: According to the position of each of the shard data recorded in the memory shard mapping table in the target data, the first verification information and the second verification information of each of the shard data are sequentially compared to obtain a corresponding first comparison result; When the first comparison results of the slice data are the same, determining that the slice data of the target data are in a normal state, and directly outputting the target data; When the first comparison result of any shard data is different, the shard data is determined to be abnormal shard data, the abnormal shard data is repaired using the secondary copy storing the target data, the target data is updated according to the repaired abnormal shard data, and the updated target data is output.

5. The method according to claim 4, characterized in that Directly output the target data, including: Taking the temporary data cache list corresponding to the first shard data of the target data as the benchmark cache list, according to the position of each shard data recorded in the memory shard mapping table in the target data, the temporary data cache list corresponding to each other shard data is appended to the end of the benchmark cache list in sequence to obtain the target data cache list, and the target data is processed through the target data cache list, wherein the other shard data are the shard data other than the first shard data among the multiple shard data obtained by splitting the target data.

6. The method according to claim 4, characterized in that Repairing the abnormal shard data by using the secondary copy storing the target data, and updating the target data according to the repaired abnormal shard data, including: Traverse each of the secondary copies storing the target data and perform the following steps: According to the data length of the abnormal shard data, the object name of the object where the abnormal shard data is located, and the offset of the abnormal shard data in the corresponding object, the interval data corresponding to the abnormal shard data is obtained from the secondary copy, and the preset verification algorithm is used to operate the interval data to obtain the corresponding third verification information; Compare the third verification information with the first target verification information of the abnormal shard data to obtain a second comparison result; When the second comparison result is the same, the data recorded in the temporary data cache list corresponding to the abnormal shard data recorded in the memory shard mapping table is updated to the interval data to update the target data; When the second comparison result is different, continue to repair the abnormal shard data using other secondary copies storing the target data.

7. The method according to claim 1, characterized in that The preset check algorithm includes at least one of the following: a cyclic redundancy check algorithm, a Hamming code, and a parity check algorithm.

8. A data reading and processing device based on a Ceph storage system, characterized in that: include: The acquisition module is used to respond to the data read request initiated by the client and obtain the target data corresponding to the data read request from the Ceph storage system; A first determination module is used to divide the target data into multiple slice data, and use a preset verification algorithm to calculate each slice data to obtain first verification information corresponding to each slice data; A second determination module is used to use the object name of the object where the shard data is located and the offset of the shard data in the corresponding object as a key to search for the second verification information corresponding to each shard data in a preset key-value pair table, wherein the key-value pair table stores multiple sets of key-value pairs, and the key-value pairs are the object name of the object where each shard data is located obtained by splitting the target data when writing the target data and the offset of each shard data in the corresponding object as the key in the key-value pair, and the second verification information obtained by calculating each shard data using a preset verification algorithm is used as the value in the key-value pair; A data reading and processing module is used to determine a first comparison result of the first verification information and the second verification information of each of the shard data, and perform a corresponding reading and processing operation on the target data according to the first comparison result of each of the shard data, wherein the reading and processing operation includes: directly outputting or repairing the abnormal shard data in the target data before outputting.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, wherein the device where the non-volatile storage medium is located executes the data reading and processing method based on the Ceph storage system as described in any one of claims 1 to 7 by running the computer program.

10. A computer program product, characterized in that include: A computer program, wherein when the computer program is executed by a processor, the data reading and processing method based on the Ceph storage system according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Method and device for repairing data of distributed storage system

    CN105550229A

  • Copy storage system-based data processing method and related apparatus

    CN107807792A

  • Data verification method of distributed storage system, and related device

    CN111176885A

  • Method, device, and system for playing and checking streaming media file

    WO2017035786A1

Cited By

  • Land resource survey result storage method and system based on artificial intelligence

    CN122018780A