A data processing method, device, program product, and storage medium

Abstract

The application provides a data processing method, device, program product and storage medium. The method comprises: receiving a first service request, wherein the first service request comprises a read request and / or a write request, the read request is used for reading data of a first data block, and the write request is used for writing data to the first data block; obtaining a first state code of the first data block, wherein the first state code represents state information of the first data block; and determining a data processing strategy corresponding to the first service request according to the first state code and the first service request.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, program product and storage medium. Background Technology

[0002] In today's wave of big data and cloud computing, distributed storage systems, as the core infrastructure supporting massive data processing and efficient access, directly impact the continuous operation of business systems and data security. Distributed storage, by distributing data across multiple independent nodes, achieves redundant backup and load balancing, greatly enhancing system scalability and fault tolerance. However, despite the widespread adoption of various strategies to ensure high availability and fault tolerance, distributed storage systems inevitably trigger data recovery mechanisms when faced with node or disk failures. Current data recovery mechanisms still suffer from challenges such as difficulty in guaranteeing data consistency, business latency caused by fault recovery, and significant network overhead during the recovery process. Summary of the Invention

[0003] To address the aforementioned technical problems, embodiments of this application provide a data processing method, apparatus, program product, and storage medium, which can improve data processing efficiency.

[0004] The data processing method provided in this application includes:

[0005] Receive a first service request, which includes a read request and / or a write request. The read request is used to read data from the first data block, and the write request is used to write data to the first data block.

[0006] Obtain the first status code of the first data block, which represents the status information of the first data block;

[0007] The data processing strategy corresponding to the first service request is determined based on the first status code and the first service request.

[0008] The data processing apparatus provided in this application includes a processor and a memory, wherein the memory is used to store computer programs, and the processor is used to call and run the computer programs stored in the memory to perform the above-described data processing method.

[0009] This application provides a computer program product, comprising: a computer program that, when executed by a processor, implements the above-described data processing method.

[0010] The computer-readable storage medium provided in this application is used to store a computer program that causes a computer to perform the above-described data processing method.

[0011] In the technical solution of this application, a first service request is received, which includes a read request and / or a write request. The read request is used to read data from a first data block, and the write request is used to write data to the first data block. A first status code of the first data block is obtained, which represents the status information of the first data block. A data processing strategy corresponding to the first service request is determined based on the first status code and the first service request. In this way, the status information of multiple data slices in the first data block is represented by the first status code. The data slices that need to be processed are determined according to the first service request. Furthermore, based on the status information of multiple data slices and the data slices related to the first service request, the data processing strategy corresponding to the first service request can be accurately determined, improving data processing efficiency. At the same time, after determining the status information of each data slice in the first data block according to the first status code, the data slices that need to be repaired are identified, improving the repair efficiency of the first data block. Attached Figure Description

[0012] The accompanying drawings, which are provided to further illustrate this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application.

[0013] Figure 1 This is a flowchart illustrating the data processing method provided in the embodiments of this application. Figure 1 ;

[0014] Figure 2 This is a flowchart illustrating the data processing method provided in the embodiments of this application. Figure 2 ;

[0015] Figure 3 This is a schematic diagram of the data slice status codes provided in an embodiment of this application;

[0016] Figure 4 This is a schematic diagram of the data repair process provided in the embodiments of this application;

[0017] Figure 5 This is a schematic diagram of the structural composition of the data processing apparatus provided in the embodiments of this application;

[0018] Figure 6 This is a schematic structural diagram of a data processing device provided in an embodiment of this application;

[0019] Figure 7 This is a schematic structural diagram of the chip according to an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0021] In the following description, the term "some embodiments" refers to a subset of all possible embodiments. However, it is understood that "some embodiments" can be the same or different subsets of all possible embodiments and can be combined with each other without conflict. It should also be noted that the terms "first," "second," and "third" used in the embodiments of this application are only used to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first," "second," and "third" can be interchanged in a specific order or sequence where permissible, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein. Furthermore, the terms "system" and "network" are often used interchangeably herein. The term "and / or" in this document is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects are in an "or" relationship. It should also be understood that the "instruction" mentioned in the embodiments of this application can be a direct instruction, an indirect instruction, or an indication of an association relationship. For example, A instructing B can mean that A directly instructs B, for example, B can obtain information through A; it can also mean that A indirectly instructs B, for example, A instructs C, and B can obtain information through C; or it can mean that there is an association between A and B. It should also be understood that the term "correspondence" mentioned in the embodiments of this application can indicate a direct or indirect correspondence between the two, or an association between them, or a relationship of instruction and being instructed, configuration and being configured, etc. It should also be understood that the "predefined" or "predefined rules" mentioned in the embodiments of this application can be implemented by pre-saving corresponding codes, tables, or other methods that can be used to instruct relevant information in the device, and this application does not limit the specific implementation method. For example, a predefined rule can refer to one defined in a protocol.

[0022] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and they all fall within the protection scope of the embodiments of this application.

[0023] In today's wave of big data and cloud computing, distributed storage systems, as the core infrastructure supporting massive data processing and efficient access, directly impact the continuous operation of business systems and data security. Distributed storage, by distributing data across multiple independent nodes, achieves redundant backup and load balancing, greatly enhancing system scalability and fault tolerance. However, no technological system is perfect. Despite the widespread adoption of various strategies to ensure high availability and fault tolerance, distributed storage systems inevitably trigger data recovery mechanisms when faced with node or disk failures.

[0024] For example, in database systems, to ensure stable business traffic, some solutions choose to roll back data, sacrificing consistency. In weak consistency scenarios, some solutions choose to repair data blocks; however, the underlying data block size is constant, and a small IO (e.g., 8KB) write could trigger the repair of an entire data block, causing read / write amplification and leading to drastic fluctuations in business data. In strong consistency scenarios, if a business request happens to access a data block being repaired during data repair, the system will temporarily suspend the business to ensure consistency, waiting for the repaired data block to complete before providing normal service. However, due to the size of the entire data block, it takes longer to write to disk than the business data. In this scenario, business processes may be blocked or even interrupted, resulting in drastic fluctuations in business data. How to reduce business data fluctuations during repair scenarios is precisely the problem this application aims to solve.

[0025] Significant technical flaws still exist in current repair solutions. Traditional repair solutions have the following drawbacks:

[0026] 1) Data consistency is difficult to guarantee: When a node goes offline or a disk fails in the system, the traditional approach is to roll back to the old data to ensure business continuity. While this approach can reduce the risk of service interruption to some extent, it sacrifices data consistency. That is, the system may provide users with inconsistent data views at different points in time, and the number of nodes that can carry valid data becomes smaller. This is unacceptable for application scenarios that require strong consistency guarantees (such as financial transactions, database systems, etc.).

[0027] 2) Business latency caused by fault recovery: To restore data consistency, existing systems typically need to repair data on failed nodes or disks. If the accessed data happens to be located in the faulty area, user requests will have to wait for the underlying data to be repaired, introducing significant business latency. This latency not only degrades the user experience but can also trigger a chain reaction due to delayed data access responses, further impacting the overall performance and stability of the distributed storage system. Especially in high-concurrency access scenarios, if multiple data blocks are simultaneously under repair, the waiting time for business requests will increase significantly, potentially leading to large fluctuations in the response time for reading and writing business data, affecting user experience or business continuity.

[0028] 3) Significant network overhead during the repair process: In a distributed storage architecture, underlying data is stored in the form of data blocks. While this design improves the flexibility of data management, it can lead to significant network overhead when data repair is required. Specifically, when a data block needs to be recovered due to a node failure, the system needs to read the entire data from other healthy nodes. This process of transmitting large amounts of data across nodes undoubtedly consumes valuable network bandwidth resources, thus putting pressure on the overall system performance, resulting in slower response times, and impacting user experience and business continuity.

[0029] Therefore, how to maximize data processing efficiency and minimize resource consumption becomes a key consideration in data processing. To this end, the following technical solutions based on embodiments of this application are proposed.

[0030] Figure 1 This is a flowchart illustrating the data processing method provided in the embodiments of this application. Figure 1 ,like Figure 1 As shown, the data processing method includes the following steps:

[0031] Step 101: Receive the first service request, which includes a read request and / or a write request. The read request is used to read data from the first data block, and the write request is used to write data to the first data block.

[0032] Here, the technical solution of this application embodiment is applied to a data processing system, which can be a distributed storage database. The data processing system includes multiple nodes for storing data. It can be understood that the nodes here can also be called data nodes.

[0033] In some implementations, the data processing system receives a first service request, which may be a read request and / or a write request. The read request is used to read data from a first data block, and the write request is used to write data to the first data block.

[0034] In some implementations, when the data processing system is a distributed storage database, the write request is used not only to write data to the first data block, but also to write the same data to the second data block corresponding to the first data block. Here, the number of second data blocks can be one or more, which can be determined according to the actual situation, and this application does not make a specific limitation on this.

[0035] In some implementations, the first data block is a data block in the first node.

[0036] It should be noted that the embodiments of this application are only illustrated with the first service request and the first data block as examples. Specifically, the technical solutions of the embodiments of this application can be applied to other data blocks and other service requests. It should also be understood that the technical solutions of the embodiments of this application can be applied to multiple service requests at the same time.

[0037] It should also be noted that, in the embodiments of this application, a faulty node can be referred to as an abnormal node or an unhealthy node. A non-faulty node can be referred to as a healthy node or a normal node. A data slice can also be referred to as a slice.

[0038] It should also be noted that, in this embodiment of the application, when the cluster in the data processing system malfunctions, the data processing method of this embodiment of the application will be executed.

[0039] In some implementations, the data processing system determines the first data block corresponding to the first service request based on the content information of the first service request. That is, when the first service request is a read request, the data required by the first service request is read from the first data block; when the first service request is a write request, the data of the first service request is written to the first data block; when the first service request is both a read request and a write request, the system determines whether to perform the read request or the write request first, thereby realizing the content corresponding to the first service request.

[0040] It is understood that when the first business request includes both write and read requests, different requests can correspond to different data blocks or the same data block. For example, data can be written to the first data block and then read from the first data block, or data can be read from the first data block and then written to the first data block, or data can be read from the third data block and then written to the fourth data block, etc. In this case, the data processing strategy for reading the data from the third data block is the same as the data processing strategy for reading the data from the first data block, and the data processing strategy for writing data to the fourth data block is the same as the data processing strategy for writing data to the first data block. This application embodiment uses reading and writing the same data block as an example for illustration.

[0041] Step 102: Obtain the first status code of the first data block. The first status code represents the status information of the first data block.

[0042] In some implementations, when the data processing system receives a first service request, it determines the first data block corresponding to the first service request, obtains a first status code representing the status information of the first data block, and then determines how to implement the first service request based on the first status code.

[0043] It can be understood that obtaining the first status code of the first data block here is essentially obtaining the status code of the data block corresponding to the first business request.

[0044] In some implementations, obtaining the first status code of the first data block includes: dividing the first data block into multiple data slices; obtaining status information of each data slice in the multiple data slices; and determining the first status code of the first data block based on the status information of each data slice in the multiple data slices.

[0045] In some implementations, upon receiving a first service request, the first node initiates the slicing of the first data block, divides the first data block into multiple data slices, obtains the status information of each data slice, and determines the first status code of the first data block based on the status information of each slice.

[0046] In some implementations, the state information of each data slice in the first data block is described using binary numbers. Specifically, the first data block is subdivided into multiple independent parts, i.e., data slices, and the state of each data slice is represented by 0 and 1.

[0047] For example, a 4MB data block is divided into 64 data slices, and a set of 64 binary bits is used to accurately record the state of each slice. Thus, the uint64 data type, an unsigned 64-bit integer value, is used to carry these 64 state information values; this uint64 data is the state code. Each binary bit occupies a fixed position in the uint64 value, and its value (0 or 1) directly maps to the specific state of the corresponding data slice. In the representation method of this application embodiment, 0 represents that the slice has not been modified or is in a certain "inactive" state, while 1 represents that the slice has been updated or is in an "active" state. This binary encoding method not only achieves compact storage of state information but also facilitates fast bit operations, thereby improving the efficiency of data management and querying. Furthermore, this mechanism also has good scalability and flexibility. As the data block size increases or decreases, the number of binary bits can be adjusted accordingly to adapt to new needs.

[0048] In some implementations, a data slice has a value of 0 when it is not written and a value of 1 when it is written. Here, status codes can be used to indicate the modification status of data slices when a node fails.

[0049] It's understandable that read requests do not change the state of the data slice.

[0050] In some implementations, if some data slices in the current first data block have been written before, and the current first service request is a write request, then the data corresponding to the write request can be written to the remaining data slices that have not been written.

[0051] Step 103: Determine the data processing strategy corresponding to the first service request based on the first status code and the first service request.

[0052] Here, after obtaining the first status code corresponding to the first data block, the status information of each data slice in the first data block can be determined. Therefore, the data processing strategy corresponding to the first business request can be determined based on the first business request. It can be understood that the data processing strategy here is the strategy for processing the first business request.

[0053] In some implementations, determining the data processing strategy corresponding to the first service request based on the first status code and the first service request includes: obtaining fault information of the first node where the first data block is located; and determining the data processing strategy corresponding to the first service request based on the fault information of the first node, the first status code, and the first service request.

[0054] In some implementations, the fault information of the first node is whether the first node is a faulty node. In this way, the data processing strategy corresponding to the first service request can be accurately determined by combining whether the first node is a faulty node, the first status code of the first data block, and the first service request.

[0055] It is understandable that whether the first node is a faulty node also affects the data processing strategy corresponding to the first business request.

[0056] In some implementations, determining the data processing strategy corresponding to the first service request based on the fault information of the first node, the first status code, and the first service request includes: when the fault information indicates that the first node is a non-faulty node and the first service request is a write request, executing a first data processing strategy, which includes writing the data corresponding to the first service request into a first data block and updating the first status code of the first data block; when the fault information indicates that the first node is a faulty node and the first service request is a write request, executing a second data processing strategy, which includes updating the first status code of the first data block; and when the fault information indicates that the first node is a faulty node and the first service request is a read request, executing a third data processing strategy, which includes reading the data corresponding to the first service request from the first data block and the second data block corresponding to the first data block, wherein the multiple nodes where the multiple second data blocks are located are non-faulty nodes.

[0057] Here, when the data processing system is a distributed storage database, when faced with a write request, the write request may be written to more than one data node, meaning the write request corresponds to more than one node. The first and second data processing strategies are implemented when the nodes corresponding to the write request include both faulty and non-faulty nodes. Specifically, when the node corresponding to the first data block of the write request is a non-faulty node, the first data processing strategy is executed to complete the write request when writing data to that non-faulty node. At this time, when writing data to another faulty node, the second data processing strategy is executed to complete the write request. When the node corresponding to the first data block of the write request is a faulty node, the second data processing strategy is executed to complete the write request when writing data to that faulty node. At this time, when writing data to another non-faulty node, the first data processing strategy is executed to complete the write request. When the first business request is a read request, if the first node to which the first data block of the read request belongs is a faulty node, the third data processing strategy is executed to complete the read request.

[0058] It is understandable that when the nodes corresponding to a write request include both faulty and non-faulty nodes, the process of writing to the faulty node and the process of writing to the non-faulty node are performed synchronously. That is, for the same write request, the process of writing to different nodes is synchronized.

[0059] In some implementations, executing a first data processing strategy includes: obtaining a second status code for the data corresponding to the first service request; updating the first status code of the first data block according to the second status code; and writing the data corresponding to the first service request into the first data block. Here, when writing data to a non-faulty node, it is necessary to write the data into the corresponding data block and update the status code corresponding to the data block to ensure the validity of the data written to disk.

[0060] In some implementations, after obtaining the first service request, the method further includes: when the first service request is a write request, determining the data block corresponding to the write request as the first data block based on the data corresponding to the first service request. Here, the data to be written to the first data block is determined based on the data of the write request.

[0061] In some implementations, the data corresponding to the first service request includes data length information and data offset information. Determining the data block corresponding to the write request as the first data block based on the data corresponding to the first service request includes: determining the data block corresponding to the write request as the first data block based on the data length information and data offset information.

[0062] In some implementations, the method further includes: determining the number of data slices corresponding to the first data block according to a first strategy configuration. The first strategy configuration is used to determine the number of data slices to divide the data block.

[0063] Here, the number of data slices to be divided into data blocks can be determined according to the first strategy configuration, that is, the number of bits or numbers of status codes corresponding to the first data block can be determined. The number of data slices to be divided can be determined according to the parameter configuration of the cluster where the first node is located. For example, it can be divided into 64 data slices or 128 data slices, etc. The specific number of divisions can be determined according to the actual situation, and this application does not make a specific limitation on this.

[0064] In some implementations, updating the first status code of the first data block according to the second status code includes: performing an OR operation on the second status code and the first status code to obtain a fifth status code, and using the fifth status code as the updated status code of the first data block.

[0065] Specifically, non-faulty nodes obtain the business I / O at the business layer, i.e., the data corresponding to the first business request. Based on the length and offset of this business I / O, the data block to which the business I / O should fall is calculated. The granularity of the data block slice during this first business request is obtained through policy configuration (i.e., the first policy configuration). Based on the granularity of the data block slice, the length of the business I / O, and the aforementioned encoding method for determining the status code, the status code V1 (i.e., the second status code) corresponding to this business I / O is calculated. The business data is successfully written to disk. After successful writing to disk, the status code V1 of the I / O is ORed with the status code V of the data block, and the new data block status code V=V1|V is saved, i.e., the updated first status code. The new data block status code is only stored in memory and is not persisted. Upon receiving cluster scheduling information, all abnormal nodes have completed repairs and the status codes are cleared.

[0066] Here, the status code corresponding to the data in the first business request is 1, that is, the status code of the data slice that the first business request needs to write is 1.

[0067] It is understandable that the node corresponding to the first business request here includes both faulty and non-faulty nodes. Therefore, a data repair process needs to be implemented for the faulty node corresponding to the first business request. Thus, policy configuration can not only determine the granularity of data block slicing in the business request, but also the granularity of data block slicing in the subsequent data repair process.

[0068] In some implementations, a second data processing strategy is executed, including: obtaining the third status code of the data corresponding to the first service request; and updating the first status code of the first data block according to the third status code. Here, for faulty data blocks in faulty nodes, only the status code V of the corresponding data block and the checksum of the corresponding slice are updated to mark the data status, but the data is not directly persisted.

[0069] For example, the checksum of a data slice can be the timestamp of the data slice. It's understood that the timestamp of the data block is not updated here; therefore, the timestamp of the data block can be used to determine whether the data block needs to be recovered.

[0070] In some implementations, after determining that the data block corresponding to the write request is the first data block, a consistency check is performed to determine whether the first data block needs repair. If repair is required, a second data processing strategy is executed. Here, consistency checking can be implemented through a reliability query. The reliability query determines whether the data block has a problem, and thus whether it needs recovery. If repair is not required, this step is not necessary; if repair is required, the data block has a problem and the process needs to be executed. For example, the timestamp of the data block can be used to determine whether data recovery is needed. If the timestamps are inconsistent, the data block needs recovery. The specific consistency check method can be determined according to the actual situation, and this application does not impose specific limitations on it. The consistency check here is to determine whether the timestamp of the data block is consistent with the timestamp of the normal data block when the cause of the failure is unclear. If they are inconsistent, the second data processing strategy needs to be executed; if they are consistent, the data block can be updated normally, and the second data processing strategy does not need to be executed. For example, failure causes include disk replacement, power outages, or short-term network fluctuations. That is, before the failure data block is repaired, the timestamp is inconsistent with the timestamp of the normal data block.

[0071] It is understandable that each node includes multiple data blocks. This refers to the faulty data block in the faulty node. If it is not a faulty data block, the normal write process is the same.

[0072] In some implementations, the method further includes: determining the number of data slices corresponding to the first data block according to a second strategy configuration. The second strategy configuration is used to determine the number of data slices to divide the data block.

[0073] Here, the number of data slices to be divided into data blocks can be determined according to the second strategy configuration, that is, the number of bits or number of status codes corresponding to the first data block can be determined. The number of data slices to be divided can be determined according to the parameter configuration of the cluster where the first node is located. For example, it can be divided into 64 data slices or 128 data slices, etc. The specific number of divisions can be determined according to the actual situation, and this application does not make a specific limitation on this.

[0074] Here, the first strategy configuration is used by non-faulty nodes to determine the number of data slices to divide a data block, and the second strategy configuration is used by faulty nodes to determine the number of data slices to divide a data block. The second strategy configuration is the same as the first strategy configuration.

[0075] In some implementations, updating the first status code of the first data block according to the third status code includes: performing an OR operation on the third status code and the first status code to obtain a sixth status code, and using the sixth status code as the updated status code of the first data block.

[0076] Specifically, the faulty node receives a business I / O at the business layer. Based on the length and offset of this business I / O, it calculates which data block the business I / O falls on on the faulty node. A consistency check is performed to determine if the corresponding data block needs recovery. If no repair is needed, the normal write process begins. If repair is required, the following process continues: The granularity of the data block slice during this repair process is determined through policy configuration (i.e., the second policy configuration). Based on the granularity, the length of the business I / O, and the previously determined status code encoding method, the status code V2 (i.e., the third status code) corresponding to this business I / O is calculated. The I / O status code V2 is ORed with the data block status code V, resulting in the new data block status code V = V2|V, which is the updated first status code. The write operation is locked in the critical section. The new data block status code is stored in memory without persistence. Upon receiving a message indicating that the data block repair is complete, the status code is cleared, and the normal write operation process begins, meaning the data node is now a normal node.

[0077] Here, the status code corresponding to the data in the first business request is 1, that is, the status code of the data slice that the first business request needs to write is 1.

[0078] In some implementations, a third data processing strategy is executed, including: determining the change status information of multiple data slices corresponding to the first data block based on a first status code; determining the unchanged data slices among the multiple data slices corresponding to the first data block based on the change status information, and obtaining a first data set from the unchanged data slices; determining the changed data slices among the multiple data slices corresponding to the first data block based on the change status information, and obtaining a second data set from the data slices in the second data block that correspond to the changed data slices; and constructing the data corresponding to the first business request based on the first data set and the second data set. Here, if the local node corresponding to the read request is a faulty node, the unchanged data slices are quickly filtered based on the first status code, read from the first data block, and then the changed data request is routed to other corresponding healthy nodes, i.e., the node to which the second data block belongs, and the complete business data is combined and returned.

[0079] In some implementations, after obtaining the first service request, the method further includes: when the first service request is a read request, determining the data block corresponding to the read request as the first data block based on the data corresponding to the first service request.

[0080] Specifically, the faulty node obtains business I / O at the business layer, calculates the data block requested by the business I / O based on its length and offset, i.e., the data block that needs to be read; acquires the critical section resource - the status code of the data block - through locking; quickly parses the read status code to identify changes in the local node's data block slice; based on the parsing results, for unchanged slices, read directly from the local node to reduce network bandwidth consumption, and for changed slices, obtain from the healthy replica (i.e., the second data block) to ensure data consistency; and combines the data read locally and the data obtained from the healthy replica to form the complete business data.

[0081] In some implementations, the method further includes: when the first node containing the first data block is a faulty node, executing a data repair strategy based on the first status code of the first data block, the data repair strategy being used to repair the first data block. Here, when service data is written to or has been written to a faulty node, data repair of the faulty node needs to be implemented, i.e., the data repair strategy is executed. It can be understood that the first status code of the first data block at this time is the latest status code before the first data block was repaired.

[0082] In some implementations, the method further includes: using a first thread to verify all data blocks in the first node, obtaining the verification result of each data block, and determining the data blocks that need to be repaired based on the verification result of each data block.

[0083] In some implementations, data blocks can be verified based on their MD5 values. The specific verification method can be determined according to the actual situation, and this application does not impose any specific limitations on it.

[0084] Specifically, when the faulty node experiences disk-related problems (disk failure) or its logical clock (epoch) falls below that of the cluster leader, it will notify the cluster scheduler, mark itself as a faulty node, and start another thread (i.e., the first thread) to traverse and check the status of local data blocks. If the data block verification is consistent, the data block will provide services normally. If the verification fails, it will enter the data repair (Recovery) process (i.e., the data repair strategy).

[0085] In some implementations, the timing for executing the data repair strategy may be the end of the current first business request, or it may be based on a predefined time. This application does not specifically limit the timing of the repair.

[0086] In some implementations, executing a data repair strategy includes: obtaining status codes of multiple second data blocks corresponding to a first data block, wherein the multiple second data blocks correspond to multiple second nodes; determining a fourth status code based on the status codes of the multiple second data blocks and a first status code of the first data block, wherein the fourth status code represents the status information of the first data block and the multiple second data blocks; obtaining a first data slice set required to repair the first data block from the multiple second data blocks based on the fourth status code; and repairing the first data block based on the first data slice set.

[0087] In some implementations, the second node is a healthy node.

[0088] In some implementations, the first data slice set required to repair the first data block is obtained from any of the second data blocks based on the fourth status code. Here, when there are multiple second data blocks, the first data slice set can be obtained from one of the second data blocks.

[0089] In some implementations, the position of the data slice marked as 1 in each of the second and first data blocks is determined based on the status codes of the multiple second data blocks and the first status code of the first data block, and the position of each data slice marked as 1 in the fourth status code is marked as 1. Here, the number of data slices divided by the multiple second data blocks and the first block is the same, and the data slices marked as 1 constitute the first data slice set.

[0090] In some implementations, the status code of the first data block is cleared to zero after the fourth status code is generated. Thus, updating the status code of the first data block during the repair process can determine the data slice to be repaired.

[0091] In some implementations, for the second node to which the second data block belongs, the second data block is encoded according to the fourth status code. That is, the status code of the second data block is determined according to the fourth status code, and the first data slice set to be transmitted, i.e., the data slice marked 1, is determined from it, and the first data slice set is encoded. Here, the data of the data slice to be acquired can be determined according to the fourth status code.

[0092] In some implementations, when the first data block receives the encoded first data slice set, it is decoded using the fourth status code to recover the original data, and the first data block is repaired based on the recovered data.

[0093] In some implementations, the first status code of the first data block is updated based on the repair result. The status code after repair is compared with the fourth status code. If they match, the data block is repaired. If they do not match, the status code of the first data block before repair is used as the fourth status code, and the data repair strategy is re-executed. This process continues until the data block is repaired.

[0094] In some implementations, once all data blocks of the faulty nodes have been repaired, the status codes of all data blocks are cleared to zero, the faulty nodes are restored to non-faulty nodes, and the cluster provides normal services.

[0095] Specifically, the status codes of multiple data block replicas corresponding to the first data block are obtained, i.e., the status codes of multiple second data blocks. These status codes are then integrated with the first status code to generate a fourth status code. This fourth status code represents the modification information of all data slices in the first data block and the multiple second data blocks. Based on the integrated full status code V3 (i.e., the fourth status code), the faulty node calculates the total length of the data slices to be repaired. Then, it sends a precise data request to any healthy replica node to obtain the required data slices. Before receiving the data, the faulty node clears the status code V (i.e., the first status code of the first data block) of its local critical section and protects this area through a locking mechanism. Subsequent changes to the critical section status code V indicate incremental updates of the data blocks during the repair period. Upon receiving the data request, the healthy node encodes the locally stored data blocks according to status code V3 (i.e., encodes the first data slice set) to optimize data transmission efficiency. The encoded data is then transmitted to the faulty node for subsequent decoding and repair operations. Upon receiving the encoded data, the faulty node performs a decoding operation based on status code V3 to recover the original data group data[N]. Using the "io_submit" system call, it submits the decoded data group data[N] in batches to the kernel for processing and reads the processing results all at once. The repaired data blocks are then verified again to ensure data consistency and integrity. If the verification results are consistent, the data block is transferred to the normal service process; if the verification results are inconsistent, the original value of critical section V is used to replace the value of V3, and the repair process is re-executed starting from the step of clearing the critical section status code. After the repair thread has traversed all data blocks, it reports that the cluster nodes have returned to normal. Once all nodes have returned to normal, the cluster status changes from sub-healthy to healthy, and all nodes clear the status codes of the critical section data blocks, allowing the cluster to provide normal service.

[0096] Here, each faulty node corresponds to one thread. It should also be understood that after the repair is complete, the timestamps of the faulty data blocks will be consistent with the timestamps of the normal data blocks.

[0097] The technical solution of this application embodiment receives a first service request, which includes a read request and / or a write request. The read request is used to read data from a first data block, and the write request is used to write data to the first data block. A first status code of the first data block is obtained, representing the status information of the first data block. A data processing strategy corresponding to the first service request is determined based on the first status code and the first service request. Thus, by using the first status code to represent the status information of multiple data slices in the first data block, the data slices to be processed are determined according to the first service request. Furthermore, based on the status information of multiple data slices and the data slices related to the first service request, the data processing strategy corresponding to the first service request can be accurately determined, improving data processing efficiency. Simultaneously, after determining the status information of each data slice in the first data block based on the first status code, the data slices that need repair are identified, improving the repair efficiency of the first data block.

[0098] Based on the foregoing embodiments, the data processing method provided by the embodiments of this application will be further described.

[0099] This application proposes an innovative method for distributed storage data consistency and efficient fault data repair, which can improve system reliability, reduce business data fluctuations caused by repair, and improve resource utilization efficiency. Specifically, it ensures data consistency and redundancy by reading redundant data from other healthy nodes, recalculating and verifying it, and then writing it to a new node or disk. This reduces the risk of data inconsistency caused by rolling back old data, while still handling traffic pressure externally, achieving load balancing and ensuring that node redundancy is not affected by the repair process. It optimizes the fault recovery process and reduces business latency by designing an efficient fault recovery strategy that can quickly repair data on faulty nodes or disks without affecting normal business access. Through data slicing and other technologies, it shortens data repair time, reduces user request latency, and improves system response speed. It reduces network overhead during the repair process by using data slicing encoding to reduce the amount of data used for repair, thereby reducing the amount of healthy data that needs to be read from other nodes and reducing network bandwidth overhead caused by data transmission.

[0100] The technical solution of this application mainly consists of three parts: generating status code V through slice encoding, data repair process for abnormal nodes based on status code V, and updating status code in the read / write business process. Figure 2 This is a schematic flowchart of the data processing method provided in the embodiments of this application. Figure 2 ,like Figure 2As shown, the system requests business data from the faulty node. This business data arrives at the data center. For write requests to healthy nodes (i.e., the aforementioned non-faulty nodes), the system obtains the corresponding IO status code V', writes it to the corresponding data block on the healthy node, and updates the corresponding status code V. For write requests from faulty nodes, only the corresponding status code V is updated; no data is written. For read requests from faulty nodes, the system first reads the status code V of the faulty node's data block, obtains the unupdated data slice, then obtains the data corresponding to the changed data slice from the healthy node, integrates the data, and returns it to the data center. The repair process involves obtaining the corresponding data from the healthy node, writing it to the local data block on the faulty node, updating the corresponding status code, and then comparing it with the status code of the healthy node. If they are inconsistent, the repair process is re-implemented.

[0101] Specifically, the system monitors the cluster status to promptly detect node anomalies and trigger a data repair process. First, when business operations are active, data block slicing is initiated within the cluster nodes. A status code V is generated based on the slice status, and modifications to data blocks during the business operation are tracked and recorded. The status code V is updated along with the writing of business data. In other words, the status code change and business operations occur simultaneously.

[0102] In terms of business services: A refined data operation strategy is implemented to address differences in node states. For write requests, healthy nodes directly write data and update the status code V to ensure the validity of the data persisted to disk. Abnormal nodes employ a conservative strategy, only updating the status code V of the corresponding data block and the checksum of the corresponding data slice to mark the data block status (0 for no update, 1 for a change), but without directly persisting the data. For read requests, if the local node is faulty, it first quickly filters out unchanged data slices based on the status code V and reads them from the local machine. Then, requests for modified data are routed to other healthy nodes, combined to form complete business data, and returned. Here, only write requests update the status code; read requests do not. For unchanged data slices in faulty nodes, read requests can be executed normally. For data blocks in faulty nodes, no data is written; only the status code is updated. The checksum of the data slice is a parameter of the data slice, which can be, for example, the timestamp of the data slice. It should also be understood that although read requests do not change the status code, when the current read request arrives, the current data block may have previously contained a write request, resulting in a modified data slice.

[0103] For faulty node repair: A separate repair thread is started on the faulty node, allowing it to focus on restoring the underlying data blocks. During the repair process, the system first checks whether the data blocks have been modified by verifying attributes. If not, the repair is skipped; if modified, the required data slices are accurately obtained from the healthy node and updated to the faulty node. To ensure data consistency, after repair, the status code of the faulty node is compared again with the data slice status code V of the healthy node to handle any potential incremental updates (i.e., data slices marked 1), ensuring that only the modified parts are updated, reducing unnecessary data transmission and repair time.

[0104] Here, data slicing and business operations are intertwined. Read and write requests only occur when there is business activity. At this point, node anomalies have been detected, causing the cluster to enter an unhealthy state. Data reads and writes at this stage are performed on slices. It should also be understood that for write requests, if a data block has not yet been repaired, data slices can be added sequentially. That is, data slices that have previously been written to, with a status code of 1, will not be written to again, but will instead be written to the next data slice. It should also be understood that the initial status code corresponding to a data slice is 0, and it is updated to 1 as data is written. After a failure occurs, this system, being a distributed storage system database, will know which node has a problem. The cluster flag will change, and when new read and write requests arrive, it will use this flag to determine if it belongs to the faulty cluster. If so, the technical solution of this application embodiment will be executed. Therefore, the repair process in this application embodiment is also a repair process for faulty nodes in a cluster with faulty nodes and business activity. When the repair process needs to be started, the status code will be retained until the repair is completed, at which point the status code will be cleared. In other words, the status code is used to record data changes in the abnormal node. It's understandable that this isn't a fix for every business transaction. The fix process can be implemented when needed, or when strong data consistency is required, data block fixes can be performed.

[0105] (a) Generation and function of data slice status code V:

[0106] In this application embodiment, an efficient mechanism based on binary numbers to describe the update state of data blocks is adopted, called status code V. This mechanism subdivides an underlying data block into multiple independent parts and uses 0 and 1 to represent the state. Figure 3 This is a schematic diagram of the data slice status codes provided in the embodiments of this application, such as... Figure 3As shown, there are N data blocks, namely data block 0, data block 1, ..., data block N. Data block 1 is encoded to obtain its status code V, which is 0101011...10110. Decoding this status code yields the data group data[N]. In other words, the data slice labeled 1 is extracted, reducing transmission resources and improving transmission efficiency. For example, a 4MB data block at the underlying level can be divided into 64 independent parts (or "slices"), and the status of each part can be precisely recorded using a set of 64 binary bits. Thus, the uint64 data type, an unsigned 64-bit integer value, is cleverly used to carry these 64 status information; this uint64 data is the status code v.

[0107] Specifically, each binary bit occupies a fixed position in the uint64 value, and its value (0 or 1) directly maps to a specific state of the corresponding data slice. In this representation, 0 may represent that the slice has not been modified or is in a "non-active" state, while 1 may represent that the slice has been updated or is in an "active" state. This binary encoding method not only achieves compact storage of state information but also facilitates fast bit operations, thereby improving the efficiency of data management and querying.

[0108] Furthermore, this mechanism offers excellent scalability and flexibility. As the data block size increases or decreases, the number of bits can be adjusted accordingly to accommodate new requirements.

[0109] In the data recovery and state reconstruction process, status codes serve the following roles: 1) Slice Location: In the data recovery process, status codes, as key information, are used to accurately encode and locate the state of a data block set at a specific point in time. By parsing status codes, the system can quickly identify which data slices are valid and which need to be recovered due to damage or loss. A 0 indicates valid slices, and a 1 indicates slices that need to be recovered. This process not only improves the accuracy of data recovery but also significantly reduces unnecessary inspection and verification steps. 2) Data Recovery: Based on status code analysis, the system can intelligently formulate recovery strategies, extracting only necessary slices from the backup for recovery, rather than performing a full disk recovery. This precise recovery mechanism effectively shortens recovery time and significantly reduces the consumption of network bandwidth and storage resources. 3) Data Block Reconstruction: Guided by status codes, the system can eliminate data slices that do not need repair and only recombine necessary slices into new data blocks. These optimized data blocks not only maintain consistency with the original data blocks in content but are also more compact in size, further improving the efficiency of data transmission and storage.

[0110] During data reading and disk writing, the status codes serve the following purposes: 1) Fast decoding: After an abnormal node receives a valid data block from a replica, by parsing the status codes, the system can quickly decode the complex data block into a series of ordered data groups (data[N]), where each data group consists of a variable number of slices, adjacent slices are continuous, and the offsets and lengths of these slices can be accurately calculated. 2) Data submission: Using the "submit_bio" method, the system can send (submit) the decoded data group to the kernel's general block layer at once, realizing multiple write batch processing of data, and then checking the result of each request at once using the "io_getevents" method. This process reduces the number of context switches and improves the throughput and efficiency of data processing.

[0111] (II) Read and write business process processing

[0112] When the cluster scheduler detects a fault such as a node anomaly requiring repair, it initiates a data slicing mechanism and handles read and write processes differently. This means that a single write request simultaneously writes to both faulty and non-faulty nodes, maintaining strong consistency. The specific process is as follows:

[0113] 1. Write the request processing flow:

[0114] The cluster obtains routing information from the data center and sends write requests to the nodes where the corresponding replicas exist. During this process, the cluster maintains the original number of replica nodes without change (abnormal nodes are not removed from the cluster). The write process differs between normal nodes and faulty nodes in the following ways;

[0115] The normal node process is as follows:

[0116] 1) When a normal node obtains a business I / O at the business layer, it calculates the data block to which the business I / O should fall based on the length and offset of the business I / O.

[0117] 2) The granularity of data block slicing during this repair process is obtained through policy configuration (number of status codes and number of bits). Here, the policy refers to the fact that a data block can be divided into multiple data slices. For example, it can be divided into 64 or 128 slices, depending on the parameter configuration given by the cluster. This application does not make specific limitations on this. It should also be understood that the granularity of the slices is consistent during the repair and business service processes.

[0118] 3) Calculate the status code V1 corresponding to this business IO based on the granularity and the length of the business IO, as well as the encoding method mentioned above. That is, mark the status code corresponding to the slice position to be written as 1, and mark the rest as 0.

[0119] 4) Business data is successfully written to disk, meaning the business data has been successfully written to the data block in the normal node;

[0120] 5) After successful disk write, OR the IO status code V1 with the data block status code V, and save the new data block status code V=V1|V.

[0121] 6) The new data block status code is only stored in memory and is not persisted.

[0122] 7) Upon receiving cluster scheduling information, all abnormal nodes have been repaired and the status code has been cleared. Here, cluster scheduling information refers to cluster actions, such as notifying nodes to rejoin the cluster.

[0123] The fault node process is as follows:

[0124] 1) The faulty node obtains the service IO at the service layer, and calculates which data block of the faulty node the service IO falls on based on the length and offset of the service IO;

[0125] 2) Check whether the corresponding data block needs to be recovered through consistency verification. If it does not need to be repaired, proceed to the normal write process. If it needs to be repaired, continue with the following process. Here, consistency verification is a reliability query before the distributed write request to determine whether the timestamps of the data slices in the data block are consistent. If they are consistent, the data block is considered not to need to be repaired. If they are inconsistent, the data block is considered to need to be repaired.

[0126] 3) The granularity of data block slicing during this repair process is obtained through the policy configuration (number of status codes and number of bits). The policy configuration here is the same as the aforementioned policy configuration content.

[0127] 4) Calculate the status code V2 corresponding to this business IO based on the granularity, the length of the business IO, and the encoding method mentioned in the first section above;

[0128] 5) Perform an OR operation between the I / O status code V2 and the data block status code V, resulting in the new data block status code V = V2|V. Lock the write operation in the critical section.

[0129] 6) The new data block status code is stored in memory and not persisted.

[0130] 7) Upon receiving a message indicating that the data block repair is complete, the status code is cleared, and the normal write operation process begins, meaning that the faulty node has been transformed into a non-faulty node.

[0131] 2. Read request processing flow:

[0132] Since read requests do not have the same amplification effect as write requests, when the data requested by the business is not on the faulty node, the read request process is the same as the normal read process. That is, the read request does not need to generate a status code. Therefore, the read request on the faulty node is explained here:

[0133] The fault node process is as follows:

[0134] 1) The faulty node obtains the business IO at the business layer, and calculates the data block requested by the business IO, i.e. the data block that needs to be read, based on the length and offset of the business IO.

[0135] 2) Obtain the status code of the critical section resource—the data block—through locking.

[0136] 3) Quickly parse the read status codes to identify changes in the local node data block slices;

[0137] 4) Based on the parsing results, unmodified slices are read directly from the local machine to reduce network bandwidth consumption, while modified slices are obtained from healthy replicas to ensure data consistency. Here, healthy replicas will also have corresponding status codes, and the classification method is the same as that of the status codes of the faulty node.

[0138] 5) Combine the two sets of data obtained above to form complete business data.

[0139] (III) Data Repair Process

[0140] When a node experiences disk-related issues (disk failure) or its logical clock (epoch) falls below that of the cluster leader, it notifies the cluster scheduler, marks itself as a failed node, and starts a separate thread to iterate through and check the status of its local data blocks. If the data block verification matches, it provides services normally; otherwise, it enters the data recovery process. Verification can be performed using MD5 values ​​or other methods. It should also be understood that each failed node has its own dedicated recovery thread.

[0141] Figure 4 This is a schematic diagram of the data repair process provided in the embodiments of this application, such as... Figure 4 As shown, the data recovery process includes the following steps:

[0142] Step 401: Verify location and determine repair requirements.

[0143] The data block that needs to be repaired is located by checking the results of the repair thread.

[0144] Step 402: Collect and integrate status codes in parallel.

[0145] The faulty node actively initiates a network request and simultaneously retrieves the critical section status codes from all healthy nodes that store copies of the corresponding data blocks. These collected status codes are then integrated with the local critical section status code V (i.e., the status code of the currently faulty data block) to generate a comprehensive and accurate full status code V3. Here, the network request is a command to request repair, which is perceptible to the entire cluster. This integration is to combine the modified data slices from each corresponding data block to prevent inconsistencies.

[0146] Step 403: Calculate and request data slices for the required data.

[0147] Based on the integrated full status code V3, the faulty node calculates the total length of the data slice that needs to be repaired, thus determining the bandwidth resources for data transmission. Then, it sends a precise data request to any healthy replica node to obtain the required data slice.

[0148] Step 404: Clear and lock critical section status codes.

[0149] Before receiving data, the faulty node clears the status code V of the local critical section and protects the area through a locking mechanism. Subsequent changes to the critical section status code V indicate incremental updates of data blocks during the repair period. Clearing here means clearing the data code to 0.

[0150] Step 405: Data encoding and transmission.

[0151] Upon receiving a data request, a normal node encodes the locally stored data block according to status code V3, thus determining the status code of the locally stored data block. It then encodes the data slice labeled 1 and transmits the data from the labeled 1 data slice to optimize data transmission efficiency. The encoded data is subsequently transmitted to the faulty node for subsequent decoding and repair operations. In other words, the encoded data here includes the data required by the faulty data block; the data required by the faulty data block is encoded and transmitted to the faulty node.

[0152] Step 406: Data decoding and disk writing.

[0153] After receiving the encoded data, the faulty node performs a decoding operation based on status code V3 to recover the original data group data[N]. The decoded data group data[N] is then submitted to the kernel in batches via the io_submit system call to achieve rapid disk write and read the processing results at once.

[0154] Step 407: Verify the repair results.

[0155] The repaired data block is validated again to ensure data consistency and integrity. If the validation results are consistent, the data block is transferred to the normal service process; if the validation results are inconsistent, the original value of critical section V is used to replace the value of V3, and the repair process is restarted from the step of clearing the critical section status code. Here, the validation is compared with the status code of the data block of the healthy node. If they are inconsistent, the validation fails, the repair is restarted, and step 404 is executed. If they are consistent, the validation succeeds, step 408 is executed, and the next request can be made.

[0156] Step 408: Cluster state recovery.

[0157] After the repair thread completes its traversal of all data blocks, it reports that the cluster nodes have returned to normal. Once all nodes have returned to normal, the cluster status changes from sub-healthy to healthy, and all nodes clear the status codes of the critical section data blocks, allowing the cluster to provide services normally.

[0158] The technical solution of this application proposes an encoding method that uses bits to identify the state of data slices, thereby achieving effective data pruning. It enables efficient storage; using bits to identify the state of data slices can significantly reduce storage space requirements because a bit is the smallest unit of data storage. This method is particularly suitable for scenarios with a large number of data slices and infrequent state changes. Fast retrieval: Since the state information is directly embedded in the data, data slices that need to be processed or ignored can be quickly filtered based on the state code, improving data processing efficiency. Flexible expansion: The bit encoding method is easily expandable; more state identifiers can be added as needed without changing the underlying data structure.

[0159] It also proposes decoupling the business read / write process from the repair process by adding and changing status codes. This achieves process decoupling: by decoupling the business read / write process and the repair process through status code changes, the two processes can run independently without interference. This helps improve system stability and reliability; reduces complexity: the decoupled system is easier to maintain and upgrade, allowing focus on optimizing individual processes without worrying about the impact on other processes; and improves response speed: while the repair process is in progress, the business read / write process can continue to run without interruption, thus improving the overall system response speed. Read / write process optimization points: Improved write efficiency: by differentiating the processing flow between normal nodes and faulty nodes, the impact of faulty nodes on normal write request traffic is reduced, improving overall write efficiency. Reduced critical section operations: when performing write operations on faulty nodes, only a small amount of data is locked at the memory level to ensure data consistency, reducing business lock waiting time. Enhanced system reliability: by clearing status codes in a timely manner through cluster scheduling information, the system can quickly resume normal write operation flow after the abnormal node is repaired, enhancing system reliability. Consistency Guarantee: By combining status code parsing and replica reading mechanisms, consistent and complete data is ensured even on failed nodes. Node Reliability: The cluster does not reduce the number of nodes providing read request services due to failed nodes, effectively distributing the cluster's load and efficiently returning data from failed nodes, thus improving system performance in sub-optimal states.

[0160] It also proposes incremental repair through control status codes. This enables precise repair: by using control status codes, the data slices requiring repair can be accurately identified, allowing for incremental repair and avoiding a full scan and processing of the entire dataset, thus improving repair efficiency; reduced resource consumption: incremental repair reduces unnecessary data processing, thereby reducing the consumption of system resources such as CPU, memory, and I / O; and rapid recovery: in the event of data corruption or loss, incremental repair can quickly recover data, reducing business waiting time. Repair process optimization points: the business process and the repair process are decoupled, reducing the impact on business traffic and ensuring that the repair process and business process are logically completely separated. Repair tasks are executed asynchronously in the background, without directly interfering with or blocking normal business processing. The method of slicing and then reassembling data reduces the amount of data that needs to be transmitted during the repair process, reducing network overhead; batch submission of multiple sets of data to disk in a batch processing manner allows for parallel computing at the kernel level, utilizing the parallel processing capabilities of multi-core processors to complete the disk write operation quickly. Reading the results of multiple submissions at once also reduces system context switching.

[0161] Batch disk writing is achieved through status code parsing, thus enabling efficient disk writing: By parsing status codes, data slices that need to be written to disk can be identified in batches and written to the storage device at once, improving the efficiency of data writing to disk and reducing I / O operations: Batch disk writing reduces the number of I / O operations because multiple data slices can be combined into a larger data block for writing, thereby reducing I / O latency and overhead. It also improves data consistency: Batch disk writing helps ensure data consistency when writing to the storage device, avoiding data inconsistency problems caused by multiple I / O operations.

[0162] In addition, while a 64-bit value is exemplarily used to represent the state of a data block in the technical solution of this application embodiment, this design can be adjusted according to actual needs to better adapt to diverse business scenarios. Specifically, a 32-bit value can be flexibly selected to reduce resource consumption, or the value can be increased to 64 bits or even more bits to support finer data processing. Furthermore, by introducing data slicing strategies of different granularities (such as 32-bit, 64-bit, 128-bit, etc.), flexible segmentation of data blocks can be achieved, thereby accurately matching the specific needs of various businesses and providing more customized service solutions. This design not only enhances the flexibility of the system but also significantly improves its adaptability to complex business scenarios.

[0163] Figure 5 This is a schematic diagram of the structural composition of the data processing device provided in the embodiments of this application, as shown below. Figure 5 As shown, the data processing device includes:

[0164] The receiving unit 501 is used to receive a first service request, which includes a read request and / or a write request. The read request is used to read data from the first data block, and the write request is used to write data to the first data block.

[0165] The acquisition unit 502 is used to acquire the first status code of the first data block, whereby the first status code represents the status information of the first data block.

[0166] The determining unit 503 is used to determine the data processing strategy corresponding to the first service request based on the first status code and the first service request.

[0167] In some embodiments, the acquisition unit 502 is used to divide the first data block into multiple data slices; acquire the status information of each data slice in the multiple data slices; and determine the first status code of the first data block based on the status information of each data slice in the multiple data slices.

[0168] In some implementations, the determining unit 503 is used to obtain fault information of the first node where the first data block is located; and to determine the data processing strategy corresponding to the first service request based on the fault information of the first node, the first status code and the first service request.

[0169] In some embodiments, the determining unit 503 is configured to: execute a first data processing strategy when fault information indicates that the first node is a non-faulty node and the first service request is a write request; the first data processing strategy includes writing the data corresponding to the first service request into a first data block and updating the first status code of the first data block; execute a second data processing strategy when fault information indicates that the first node is a faulty node and the first service request is a write request; and execute a third data processing strategy when fault information indicates that the first node is a faulty node and the first service request is a read request. The third data processing strategy includes reading the data corresponding to the first service request from the first data block and the second data block corresponding to the first data block, wherein the multiple nodes where the multiple second data blocks are located are non-faulty nodes.

[0170] In some implementations, the determining unit 503 is used to obtain the second status code of the data corresponding to the first service request; update the first status code of the first data block according to the second status code; and write the data corresponding to the first service request into the first data block.

[0171] In some implementations, the determining unit 503 is used to obtain the third status code of the data corresponding to the first service request; and update the first status code of the first data block according to the third status code.

[0172] In some implementations, the determining unit 503 is configured to: determine change information of multiple data slices corresponding to the first data block based on the first status code; determine the unchanged data slices among the multiple data slices corresponding to the first data block based on the change information, and obtain a first data set from the unchanged data slices; determine the changed data slices among the multiple data slices corresponding to the first data block based on the change information, and obtain a second data set from the data slices in the second data block that correspond to the changed data slices; and construct the data corresponding to the first service request based on the first data set and the second data set.

[0173] In some implementations, the determining unit 503 is used to execute a data repair strategy based on the first status code of the first data block when the first node where the first data block is located is a faulty node. The data repair strategy is used to repair the first data block.

[0174] In some implementations, the determining unit 503 is configured to: obtain status codes of multiple second data blocks corresponding to a first data block, wherein the multiple second data blocks correspond to multiple second nodes; determine a fourth status code based on the status codes of the multiple second data blocks and the first status code of the first data block, wherein the fourth status code represents the status information of the first data block and the multiple second data blocks; obtain a set of first data slices required to repair the first data block from the multiple second data blocks based on the fourth status code; and repair the first data block based on the set of first data slices.

[0175] Those skilled in the art should understand that Figure 5 The functions of each unit in the data processing device shown can be understood by referring to the relevant description of the aforementioned method. Figure 5 The functions of each unit in the data processing device shown can be implemented by a program running on a processor or by specific logic circuits.

[0176] Figure 6 This is a schematic structural diagram of a data processing device 600 provided in an embodiment of this application. Figure 6 The data processing device 600 shown includes a processor 610, which can call and run computer programs from memory to implement the methods in the embodiments of this application.

[0177] Optionally, such as Figure 6 As shown, the data processing device 600 may further include a memory 620. The processor 610 can retrieve and run computer programs from the memory 620 to implement the methods described in the embodiments of this application.

[0178] The memory 620 can be a separate device independent of the processor 610, or it can be integrated into the processor 610.

[0179] Optionally, such as Figure 6 As shown, the data processing device 600 may also include a transceiver 630, which the processor 610 can control to communicate with other devices. Specifically, it can send information or data to other devices or receive information or data sent by other devices.

[0180] The transceiver 630 may include a transmitter and a receiver. The transceiver 630 may further include antennas, and the number of antennas may be one or more.

[0181] Optionally, the data processing device 600 can implement the corresponding processes implemented by the data processing apparatus in the various methods of the embodiments of this application, which will not be described in detail here for the sake of brevity.

[0182] Figure 7 This is a schematic structural diagram of the chip according to an embodiment of this application. Figure 7 The chip 700 shown includes a processor 710, which can call and run computer programs from memory to implement the methods in the embodiments of this application.

[0183] Optionally, such as Figure 7 As shown, chip 700 may further include memory 720. Processor 710 can retrieve and run computer programs from memory 720 to implement the methods described in this embodiment.

[0184] The memory 720 can be a separate device independent of the processor 710, or it can be integrated into the processor 710.

[0185] Optionally, the chip 700 may also include an input interface 730. The processor 710 can control the input interface 730 to communicate with other devices or chips; specifically, it can acquire information or data sent by other devices or chips.

[0186] Optionally, the chip 700 may also include an output interface 740. The processor 710 can control the output interface 740 to communicate with other devices or chips, specifically, to output information or data to other devices or chips.

[0187] Optionally, the chip can implement the corresponding processes implemented by the data processing device in the various methods of the embodiments of this application, which will not be described in detail here for the sake of brevity.

[0188] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0189] It should be understood that the processor in the embodiments of this application may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0190] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0191] It should be understood that the above-described memory is exemplary and not a limiting description. For example, the memory in the embodiments of this application may also be static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DR RAM), etc. That is to say, the memory in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0192] This application also provides a computer program product, including a computer program.

[0193] Optionally, when the computer program is executed by the processor, it implements the corresponding processes implemented by the data processing device in the various methods of the embodiments of this application. For the sake of brevity, these will not be described in detail here.

[0194] This application also provides a computer-readable storage medium for storing computer programs.

[0195] Optionally, the computer program causes the computer to execute the corresponding processes implemented by the data processing apparatus in the various methods of the embodiments of this application, which will not be described in detail here for the sake of brevity.

[0196] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0197] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0198] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0199] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0200] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0201] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0202] The preferred embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this application, various simple modifications can be made to the technical solutions of this application, and these simple modifications all fall within the protection scope of this application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this application will not describe the various possible combinations separately. Furthermore, various different embodiments of this application can also be arbitrarily combined, as long as they do not violate the spirit of this application, they should also be considered as the content disclosed in this application. Furthermore, without conflict, the various embodiments and / or the technical features in the various embodiments described in this application can be arbitrarily combined with the prior art, and the resulting technical solutions should also fall within the protection scope of this application. It should be understood that in the various method embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, The method includes: Receive a first service request, the first service request including a read request and / or a write request, the read request being used to read data from a first data block, and the write request being used to write data to the first data block; Obtain the first status code of the first data block, where the first status code represents the status information of the first data block; The data processing strategy corresponding to the first service request is determined based on the first status code and the first service request. Obtaining the first status code of the first data block includes: The first data block is divided into multiple data slices; Obtain the status information of each data slice in the plurality of data slices; The first status code of the first data block is determined based on the status information of each of the plurality of data slices.

2. The method according to claim 1, characterized in that, The step of determining the data processing strategy corresponding to the first service request based on the first status code and the first service request includes: Obtain the fault information of the first node where the first data block is located; The data processing strategy corresponding to the first service request is determined based on the fault information of the first node, the first status code, and the first service request.

3. The method according to claim 2, characterized in that, The step of determining the data processing strategy corresponding to the first service request based on the fault information of the first node, the first status code, and the first service request includes: When the fault information indicates that the first node is a non-faulty node and the first service request is a write request, the first data processing strategy is executed. The first data processing strategy includes writing the data corresponding to the first service request into the first data block and updating the first status code of the first data block. When the fault information indicates that the first node is a faulty node and the first service request is a write request, the second data processing strategy is executed, and the second data processing strategy includes updating the first status code of the first data block. When the fault information indicates that the first node is a faulty node and the first service request is a read request, a third data processing strategy is executed. The third data processing strategy includes reading the data corresponding to the first service request from the first data block and the second data block corresponding to the first data block, where the multiple nodes where the multiple second data blocks are located are non-faulty nodes.

4. The method according to claim 3, characterized in that, The execution of the first data processing strategy includes: Obtain the second status code of the data corresponding to the first service request; Update the first status code of the first data block according to the second status code; Write the data corresponding to the first service request into the first data block; The execution of the second data processing strategy includes: Obtain the third status code of the data corresponding to the first service request; Update the first status code of the first data block according to the third status code; The execution of the third data processing strategy includes: Based on the first status code, determine the change information of multiple data slices corresponding to the first data block; Based on the change information, determine the unchanged data slices among the multiple data slices corresponding to the first data block, and obtain the first data set from the unchanged data slices; Based on the change information, determine the changed data slices among the multiple data slices corresponding to the first data block, and obtain the second data set from the data slices in the second data block that correspond to the changed data slices; The data corresponding to the first business request is constructed based on the first data set and the second data set.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: When the first node containing the first data block is a faulty node, a data repair strategy is executed according to the first status code of the first data block, and the data repair strategy is used to repair the first data block.

6. The method according to claim 5, characterized in that, The data repair strategy includes: Obtain the status codes of multiple second data blocks corresponding to the first data block, wherein the multiple second data blocks correspond to multiple second nodes; A fourth status code is determined based on the status codes of the plurality of second data blocks and the first status code of the first data block, wherein the fourth status code represents the status information of the first data block and the plurality of second data blocks; Based on the fourth status code, obtain the first data slice set required to repair the first data block from the plurality of second data blocks; Repair the first data block based on the first data slice set.

7. A data processing device, characterized in that, include: A processor and a memory, the memory for storing a computer program, the processor for calling and running the computer program stored in the memory to perform the data processing method as described in any one of claims 1 to 6.

8. A computer program product, characterized in that, include: A computer program that, when executed by a processor, implements the data processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the data processing method as described in any one of claims 1 to 6.

Images