Data processing method

Abstract

Embodiments of the present specification provide a data processing method, wherein the data processing method is applied to a first data node, and includes: receiving a data processing request sent by a second data node, and in a case where it is determined that the data processing request is a target data processing request, obtaining a current state of a request queue, wherein the target data processing request is a request for system data in a set data node, and the request queue is a queue in the first data node; in a case where it is determined that the current state meets request storage conditions, storing the target data processing request to the request queue; and in a case where it is determined that a current request processing state meets request processing conditions, processing the target data processing request in the request queue. Further avoiding overload of the target data processing request, and not affecting user data reading and writing, and avoiding the problem that a user data reading and writing request cannot be responded in time.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to a data processing method. Background Technology

[0002] With the development of computer and distributed storage technologies, many organizations use distributed storage systems for data storage. In distributed processing systems, the data nodes handle mostly user data read / write requests, with a smaller portion being internal system data processing requests. Therefore, existing data nodes only implement I / O flow control or overload protection measures for user data read / write requests, neglecting internal system data processing requests. This results in a lack of corresponding overload protection during system operation. In this situation, when the number of system data processing requests increases, it will preempt a large amount of disk I / O capacity, affecting user data read / write operations and causing delays in responding to user data read / write requests. Therefore, there is an urgent need to provide a solution to avoid system data processing request overload and prevent impact on user data read / write operations. Summary of the Invention

[0003] In view of the above, embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program, to address the technical deficiencies existing in the prior art.

[0004] According to a first aspect of the embodiments of this specification, a data processing method is provided, applied to a first data node, comprising:

[0005] Receive a data processing request sent by a second data node. If the data processing request is determined to be a target data processing request, obtain the current status of the request queue. The target data processing request is a request for system data in a set data node, and the request queue is a queue in the first data node.

[0006] If the current state is determined to meet the storage conditions, the target data processing request is stored in the request queue.

[0007] If the current request processing status is determined to meet the request processing conditions, the target data processing request in the request queue is processed.

[0008] According to a second aspect of the embodiments of this specification, a data processing apparatus is provided, applied to a first data node, comprising:

[0009] The receiving module is configured to receive a data processing request sent by a second data node, and when it is determined that the data processing request is a target data processing request, to obtain the current state of the request queue, wherein the target data processing request is a request for system data in a set data node, and the request queue is a queue in the first data node.

[0010] The storage module is configured to store the target data processing request into the request queue when it is determined that the current state meets the request storage conditions;

[0011] The processing module is configured to process the target data processing request in the request queue when it is determined that the current request processing status meets the request processing conditions.

[0012] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising:

[0013] Memory and processor;

[0014] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the data processing method.

[0015] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the data processing method.

[0016] According to a fifth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the data processing method.

[0017] The data processing method provided in this specification, applied to a first data node, includes: receiving a data processing request sent by a second data node; if the data processing request is determined to be a target data processing request, obtaining the current state of a request queue, wherein the target data processing request is a request for system data in a designated data node, and the request queue is a queue in the first data node; if the current state satisfies the request storage conditions, storing the target data processing request in the request queue; and if the current request processing state satisfies the request processing conditions, processing the target data processing request in the request queue.

[0018] The method provided in this specification, applied to the first data node, upon receiving a target data processing request from the second data node, determines to store the target data processing request if the current state of the request queue meets the request storage conditions. This avoids request overload and the problem of excessive resource occupation of the first data node due to an excessive number of target data processing requests. Furthermore, during the processing of the target data processing request, the request is only processed if the current request processing state of the first data node meets the request processing conditions. This further avoids target data processing request overload and does not affect user data read / write operations, preventing the first data node from being unable to respond to user data read / write requests in a timely manner due to request overload. Attached Figure Description

[0019] Figure 1 This is a schematic diagram illustrating an application scenario of a data processing method provided in one embodiment of this specification;

[0020] Figure 2 This is a flowchart illustrating a data processing method provided in one embodiment of this specification;

[0021] Figure 3 This is a schematic diagram of the scheduler in a data processing method provided in one embodiment of this specification;

[0022] Figure 4 This is a schematic diagram illustrating a request for scheduling conditions in a data processing method provided in one embodiment of this specification;

[0023] Figure 5 This is a flowchart illustrating the processing procedure of a data processing method provided in one embodiment of this specification.

[0024] Figure 6 This is a flowchart of the scheduler rescheduling process in a data processing method provided in one embodiment of this specification;

[0025] Figure 7 This is a schematic diagram of background I / O throughput in a data processing method provided in one embodiment of this specification;

[0026] Figure 8 This is a schematic diagram of background I / O latency in a data processing method provided in one embodiment of this specification;

[0027] Figure 9 This is a schematic diagram illustrating the changes in MetaIOOPS in a data processing method provided in one embodiment of this specification;

[0028] Figure 10This is a schematic diagram of the state of a disk management tool in a data processing method provided in one embodiment of this specification;

[0029] Figure 11 This is a schematic diagram of the structure of a data processing apparatus provided in one embodiment of this specification;

[0030] Figure 12 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0031] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0032] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0033] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0034] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0035] I / O: Short for Input / Output.

[0036] Layout: A user data structure used to store user data.

[0037] ChunkHeader: This is a structure present in all types of blocks, primarily used to describe the structural information of a chunk.

[0038] Chunk: A data block.

[0039] SealChunk: This means that after a Chunk is sealed, it becomes read-only and cannot be written to.

[0040] Buffer: cache.

[0041] DataSync: refers to the persistence of Chunk data and metadata.

[0042] Create request: Data creation processing request.

[0043] Seal request: A request to process data encapsulation.

[0044] OPS: Short for Operation Per Second, refers to the number of specific types of requests a data node processes per second.

[0045] Data node: The process in a distributed storage system that actually stores user data and interacts with system client processes to provide data read and write capabilities.

[0046] Overload protection: Because storage systems have a capacity limit for I / O or request processing, excessive pressure from a certain type of request can preempt system resources (such as I / O, memory, or CPU), causing other types of requests to perform poorly. Therefore, to cope with this extreme scenario and ensure system availability, corresponding overload protection mechanisms need to be implemented based on different request types.

[0047] MetaIO: MetaIO within a data node refers to IO requests relative to user data IO. User data IO refers to read and write requests for user data chunks. MetaIO mainly refers to additional IO introduced by internal system mechanisms. For example, MetaIO includes, but is not limited to, the ChunkHeader added to the layout for organizing user data within the data node, the metadata update brought by SealChunk after the chunk is written, and DataSync to ensure that buffer write updates are written to disk.

[0048] With the development of computer and distributed storage technologies, many organizations use distributed storage systems for data storage. In distributed processing systems, the data nodes handle mostly user data read / write requests for I / O, with a smaller portion being internal system data processing requests. MetaIO is an example. In other words, the data nodes within this distributed system handle both actual I / O and operations such as Create / DataSync / Seal for data blocks.

[0049] Therefore, the IO flow control / overload protection measures in data nodes mainly target user data read and write operations. For example, data nodes have overload protection mechanisms such as flow control / priority scheduling for user data IO requests (read / write). However, this overload protection scheme lacks corresponding flow control / overload protection mechanisms for MetaIO-dominated Create / Seal / DataSync requests. This leads to MetaIO preempting a large amount of disk IO capacity in extreme scenarios. For instance, in extreme scenarios, high-OPS Create / Seal / DataSync requests consume significant disk IO capacity, resulting in a decrease in user data IO service capabilities and impacting user data read and write operations. Therefore, there is an urgent need to provide an overload protection scheme for data node MetaIO.

[0050] Based on this, a data processing method is provided in this specification. This specification also relates to a data processing method, a computing device, a computer-readable storage medium, and a computer program, which will be described in detail in the following embodiments.

[0051] Figure 1 This diagram illustrates an application scenario of a data processing method according to an embodiment of this specification. Figure 1 It is understood that the second data node sends a MetaIO request to the first data node. Upon receiving the MetaIO request, the first data node needs to determine if its own request queue has reached its limit. If so, it rejects the request; otherwise, it stores the MetaIO request in the request queue. Then, the first data node checks if the number of MetaIO requests currently being sent exceeds the preset maximum number. If so, it stops sending MetaIO requests; otherwise, it sends the MetaIO requests to the storage engine for processing. This overload protection strategy prevents MetaIO from monopolizing a large amount of disk I / O, thus avoiding a decrease in user data I / O service capabilities and impacting user data read / write operations.

[0052] Figure 2 A flowchart of a data processing method according to an embodiment of this specification is shown. The method is applied to a first data node and specifically includes the following steps.

[0053] Step 202: Receive a data processing request sent by the second data node. If the data processing request is determined to be a target data processing request, obtain the current status of the request queue. The target data processing request is a request for system data in a set data node, and the request queue is a queue in the first data node.

[0054] In this context, the first data node can be understood as a data node in a distributed system, which can be a server or a client. The designated data node can be understood as the data node to which the target data processing request needs to be sent. The system data within this designated data node can be processed through the target data processing request. This designated data node can be the first data node. The second data node can be understood as any other data node in the distributed system besides the first data node, or it can be understood as a client. The data processing request can be understood as a request that needs to be processed by the first data node. This data processing request may include user data requests and requests related to target data metadata of the first data node, i.e., the target data processing request described below; the user data request can be understood as a request for user data, such as a user data read / write request. The target data processing request can be understood as a processing request for system data within the designated data node's system, including but not limited to operations such as Create / DataSync / Seal for data blocks. That is to say, the target data processing request can be MetaIO. This system data can be understood as the data required by the system within the first data node during its operation, including but not limited to user data organization layout, metadata, and data information related to ensuring buffer write updates to disk. It should be noted that this system data can be data that is pre-defined in the data node as needing processing. For example, metadata can be set as system data that needs processing, and the target data processing request can process this defined system data.

[0055] In one embodiment provided in this specification, the target data processing request is a processing request for a user data structure in the first data node; and / or

[0056] The target data processing request is a processing request for the metadata in the first data node.

[0057] The processing request for the user data structure in the first data node can be understood as a request to process the user data structure used to store user data in the first data node, such as a request for the ChunkHeader added to the layout for organizing user data inside the data node.

[0058] The metadata processing request in the first data node can be understood as a request to update the metadata in the first data node, such as the metadata update request brought by SealChunk after the Chunk write is completed.

[0059] It should be noted that the data processing method provided in this specification can be applied to the first data node or to the scheduler in the first data node; the scheduler can be understood as a scheduler that schedules the target data processing requests in the first data node, that is, the scheduler is a MetaIO scheduler, and the scheduler can be a software module.

[0060] The request queue can be understood as a queue in the first data node that stores the target data processing request. In practical applications, the data processing method provided in this specification is designed as an overload protection mechanism for MetaIO, which mainly comes from Create / Seal / Datasync requests. In practice, it has been found that the difficulty of the MetaIO overload protection mechanism lies in ensuring that the success rate of MetaIO requests in extreme scenarios is not too low, while also controlling the preemption of IO resources by issued MetaIO requests. Therefore, this solution firstly needs to provide a certain queuing capability for MetaIO requests; that is, it needs to provide a certain queuing capability for each type of MetaIO request to be processed. Therefore, in the data processing method provided in this specification, the first data node contains a request queue, which is used to store the target data processing request.

[0061] Based on the above, it can be seen that target data processing requests can include various types of requests, including but not limited to Create / Seal / Datasync, etc. Therefore, the scheduler in the data processing method provided in this specification creates a corresponding request queue for each type of request to facilitate scheduling of different types of requests. When the target data processing request is received, the scheduler determines the corresponding request queue based on the request type and determines whether the request queue can currently store the target data processing request. Specifically, when it is determined that the data processing request is a target data processing request, obtaining the current state of the request queue includes:

[0062] If the data processing request is determined to be a target data processing request, the request type of the target data processing request is determined;

[0063] Determine the target request queue corresponding to the request type from at least two of the request queues, and obtain the current state of the target request queue.

[0064] The request type refers to information used to characterize a type of target data processing request, including but not limited to the request source address, the name of the request, etc., without specific restrictions.

[0065] When the target data processing request includes requests such as Create / Seal / Datasync, the request queues created for it can be Create queue, Seal queue, and Datasync queue, used to store different types of requests.

[0066] The current state of the target request queue can be understood as representing the information of the currently stored requests in the queue; for example, the current state can be an identifier representing the current number of requests, with a value of "1" indicating idle and a value of "2" indicating that the queue is full. Alternatively, the current state can be a request count (PendingCount); different types of queues can have request counts such as PendingCreateCount, PendingSealCount, and PendingDatasyncCount.

[0067] For example, when the scheduler of the first data node determines that the received data processing request is a MetaIO request, it will determine the request type of the MetaIO request. If the request type is Create, it will obtain the current request count (PendingCreateCount) of the Create queue. Subsequently, it will use the current request count to determine whether to enqueue the Create request, thereby ensuring the success rate of MetaIO requests under the pressure of a large number of requests.

[0068] Step 204: If the current state satisfies the request storage conditions, store the target data processing request into the request queue.

[0069] Based on the above, the data processing method provided in this specification creates different types of request queues for different types of requests, thereby facilitating the scheduling of target data processing requests. Therefore, the step of storing the target data processing request into the request queue when the current state meets the request storage conditions includes:

[0070] If the current state of the target request queue satisfies the request storage conditions, the target data processing request is stored in the target request queue.

[0071] Regarding the difficulties of the aforementioned overload protection mechanism, and secondly, the solution needs to control the number of requests for MetaIO. Therefore, the data processing method provided in this specification, specifically the scheduler of the first data node, needs to determine whether to store the target data processing request in the request queue based on the current number of requests in the request queue, thereby controlling the number of requests processed by the first data processing node and avoiding overload. Specifically, determining that the current state of the target request queue meets the request storage conditions includes:

[0072] Determine the number of requests to be stored in the target request queue, and if the number of requests to be stored is less than a preset storage threshold, determine that the current state of the target request queue satisfies the request storage condition.

[0073] The request storage quantity can be understood as the number of requests in the target request queue, such as the request count (PendingCount). This preset storage quantity threshold can be set according to the actual application scenario, such as 100, 1000, etc.

[0074] Using the previous example, the preset storage threshold is 100 requests. Based on this, when the request count (PendingCreateCount) of the Create queue is obtained, it is necessary to determine whether the request count (e.g., 50 requests) is less than the preset storage threshold (100 requests). Based on this, the Create request is stored in the Create queue.

[0075] In one embodiment provided in this specification, the number of requests queued in the corresponding request queue is controlled by a corresponding configured PendingCount for each type of request. However, if a request in the queue cannot be processed for a long time, in order to ensure the timeliness of request processing and avoid data errors, the timed-out unprocessed requests will be deleted. That is, in order to implement the overload protection mechanism for MetaIO, this specification provides the ability to cancel requests after queue timeout for MetaIO. Specifically, after storing the target data processing request in the request queue, the process further includes:

[0076] Determine the request storage time of the target data processing request in the request queue;

[0077] If the requested storage time exceeds a preset time threshold, the target data processing request is deleted from the request queue.

[0078] The request storage time can be understood as the time the target data processing request is stored in the request queue, such as 1 second, 5 seconds, etc.

[0079] The preset time threshold can be set according to the actual application scenario, such as 1 second, 10 seconds, etc.

[0080] Based on this, continuing with the previous example, the scheduler will monitor the storage time of Create requests stored in the Create queue. If it is determined that the storage time (e.g., 5 seconds) is greater than the preset time threshold (e.g., 1 second), it is determined that the Create request has not been processed for a long time, and therefore it is deleted from the Create queue.

[0081] Step 206: If the current request processing status meets the request processing conditions, process the target data processing request in the request queue.

[0082] To address the challenges of the aforementioned overload protection mechanism, and to ensure that MetaIO does not excessively preempt lower-level resources, it is necessary to control the number of MetaIOs issued simultaneously for each type, thereby controlling the consumption of lower-level resources by MetaIOs. Specifically, the process involves processing the target data processing request in the request queue when the current request processing status meets the request processing conditions, including:

[0083] Obtain the current number of requests processed for the target data processing request;

[0084] If the current number of requests to be processed is less than a preset processing threshold, the target data processing requests in the target request queue are processed.

[0085] The current request processing count can be understood as the current request count (InflightCount) being processed by the first data node. As described above, the data processing method provided in this specification controls the number of requests queued in the request queue for each type of request through the corresponding configured PendingCount. Furthermore, to avoid MetaIO consuming lower-level resources, the number of requests sent for each type is controlled through the corresponding configured request delivery count (InflightCount). InflightCount refers to the number of requests that have been dequeued but not yet completed. For example, the request delivery counts for different queue types can be InflightCreateCount, InflightSealCount, and InflightDatasyncCount. It should be noted that the scheduler provided in this application will detect the request delivery count for each type of request through a round-robin method, thereby scheduling the various types of requests accordingly.

[0086] The preset processing quantity threshold can be understood as the maximum number of simultaneous requests of a certain type that is set in advance. The preset processing quantity threshold can be represented by MaxInflight. Different types of requests correspond to different preset processing quantity thresholds, including but not limited to MaxInflightCreate, MaxInflightSeal, and MaxInflightDatasync.

[0087] Based on this, continuing the previous example, the scheduler schedules various requests using a round-robin approach. If it determines that the request delivery count (InflightCount) for a Create request is 10, which is less than the corresponding preset processing threshold (MaxInflightCreate) of 50, it determines that the Create request can be processed. After scheduling the Create request is complete, the scheduler will schedule the Seal and Datasync requests using a round-robin approach. See [link to details] for further information. Figure 3 , Figure 3 This is a schematic diagram of the scheduler in a data processing method provided in one embodiment of this specification, based on... Figure 3 It can be seen that... Figure 3 The logical structure of the scheduler handling Create / Seal / DataSync requests is given. First, the scheduler places each type of request into its respective queue. Second, it calculates the corresponding queue size, PendingXXXCount, for each type of request; PendingXXXCount can be understood as PendingCreateCount / PendingCreateCount / PendingCreateCount. Finally, the scheduler determines the number of tasks that can be dispatched simultaneously for each type of request, MaxInflightXXX, which can be understood as MaxInflightCreate / MaxInflightSeal / MaxInflightDatasync. The scheduler's decision-making conditions include three factors, see [link to relevant documentation]. Figure 4 , Figure 4It is a schematic diagram of request scheduling conditions in a data processing method provided by an embodiment of this specification. When the scheduler needs to determine whether a request can be dispatched, the following 3 conditions need to be judged; in Condition 1, the number of requests in the Create queue > 0 means && the number of dispatched Create requests < the maximum number of dispatched Create requests, indicating that there are pending CreateTasks in the current Create queue and InflightCreateTask does not exceed the limit; in Condition 2, the number of requests in the Seal queue > 0 && the number of dispatched Seal requests < the maximum number of dispatched Seal requests, indicating that there are pending SealTasks in the current Seal queue and InflightSealTask does not exceed the limit; in Condition 3, the number of requests in the Datasync queue > 0 && the number of dispatched Datasync requests < the maximum number of dispatched Datasync requests, indicating that there are pending DatasyncTasks in the current Datasync queue and InflightDatasyncTask does not exceed the limit; when any one of the 3 conditions is true, there must be a task in the MetaIO scheduler that meets the dispatchable conditions; when all 3 conditions are false, there is no task in the MetaIO scheduler that meets the dispatchable conditions.

[0088] In an embodiment provided by this specification, the scheduler processes the target data processing request, which can be understood as sending the request to the corresponding processing unit for processing, so as to achieve load balancing. The specific process is that the target data processing request is a metadata processing request;

[0089] Correspondingly, processing the target data processing request in the request queue includes:

[0090] According to the request type of the metadata processing request, determine the request processing unit corresponding to the metadata processing request, where the request processing unit is the processing unit in the first data node that processes the metadata processing request.

[0091] Process the metadata processing request through the request processing unit.

[0092] Among them, the metadata processing request can be understood as a request for processing metadata. Processing metadata can be understood as performing processing operations such as input, output, deletion, modification, update, reading, and writing on metadata. Therefore, the metadata processing request includes but is not limited to metadata read / write requests, metadata input / output requests (i.e., metadata IO), metadata deletion requests, metadata update requests, metadata modification requests, etc., and no specific restrictions are made here.

[0093] The request processing unit can be understood as a unit that processes the metadata processing request, such as a storage engine or disk in the first data node.

[0094] Continuing with the previous example, if it is determined that there are Create requests (CreateTasks) waiting to be issued in the current Create queue, and the InflightCreateTask limit has not been exceeded, then the Create request is sent to the storage engine. The storage engine then processes the Create request, which can be a creation request for metadata.

[0095] In one embodiment provided in this specification, the overload protection scheme for MetaIO is implemented to ensure that user data read and write operations are not affected. Therefore, while implementing this overload protection scheme, if the first data node receives a user data IO request instead of a target data processing request, it is necessary to process the user data IO request to ensure timely response to the user request. Specifically, the method further includes:

[0096] If the data processing request is determined to be a user data processing request, the user data processing request is processed.

[0097] In this context, the user data processing request can be understood as a request to process user data, such as a user data read / write request.

[0098] Based on this, in the data processing method provided in this specification, when the first data node receives a user data read / write request, but not a MetaIO request, it will also process the user data read / write request. For example, it can send the user data read / write request to the storage engine; or it can process it through an overload protection scheme for the user data read / write request, etc. This specification will not elaborate on this further.

[0099] The method applied to the first data node provided in this specification, upon receiving a target data processing request sent by the second data node, determines to store the target data processing request if the current state of the request queue meets the request storage conditions. This avoids request overload and the problem of excessive resource occupation of the first data node due to an excessive number of target data processing requests. Furthermore, during the processing of the target data processing request, the request is only processed if the current request processing state of the first data node meets the request processing conditions. This further avoids target data processing request overload and does not affect user data read / write operations, preventing the first data node from being unable to respond to user data read / write requests in a timely manner due to request overload.

[0100] The following is in conjunction with the appendix Figure 5 Taking the application of the data processing method provided in this specification in an overload protection scenario for the MetaIO data node of a distributed storage system as an example, the data processing method will be further explained. Figure 5 A flowchart illustrating the processing procedure of a data processing method according to an embodiment of this specification is shown, specifically including the following steps.

[0101] Step 502: Receive request.

[0102] Specifically, data nodes receive Create / Seal / DataSync requests.

[0103] Step 504: Has the current PendingCreate / Seal / DataSyncCount reached its limit? If yes, proceed to step 506; otherwise, proceed to step 508.

[0104] Step 506: Reject the new request.

[0105] Step 508: Accept the new Create / Seal / DataSync request.

[0106] Specifically, when a new request enters the scheduler, the new Create / Seal / DataSync request is stored in the corresponding type of queue, and the corresponding PendingXXXCnt count is incremented.

[0107] Step 510: Can the task be scheduled? If yes, proceed to step 514; otherwise, proceed to step 512.

[0108] Step 512: Do not proceed with subsequent logic.

[0109] Step 514: Poll the Create / Seal / DataSync queue and select the Task to be sent.

[0110] Step 516: After selecting the tasks to be sent, adjust the corresponding InflightXXXCnt count and PendingXXXCnt count.

[0111] Specifically, after selecting the tasks to be dispatched, the corresponding InflightXXXCnt count is increased, and the corresponding PendingXXXCnt count is decreased.

[0112] Step 518: Send the request.

[0113] In another embodiment provided in this specification, after the Task processing is completed, the scheduler needs to re-determine whether to issue the request. See details below. Figure 6 , Figure 6 This is a flowchart illustrating the scheduler rescheduling process in a data processing method provided in one embodiment of this specification. Figure 6 This illustrates the scheduler logic on the return path after MetaIOTask execution; the specific steps are as follows:

[0114] Step 602: Confirm that the Task processing is complete.

[0115] Step 604: For different types of Tasks, reduce the corresponding InflightXXXCnt count.

[0116] Step 606: Check if subsequent tasks can be issued. If yes, proceed to step 610; otherwise, proceed to step 608.

[0117] Step 608: Do not proceed with subsequent logic.

[0118] Step 610: Poll the queue and select the Task to be dispatched.

[0119] Specifically, poll the Create / Seal / DataSync queues and select the Tasks to be dispatched.

[0120] Step 612: After selecting the tasks to be issued, adjust the corresponding InflightXXXCnt count and PendingXXXCnt count.

[0121] Specifically, after selecting the tasks to be dispatched, the corresponding InflightXXXCnt count is increased, and the corresponding PendingXXXCnt count is decreased.

[0122] Step 614: Send the request.

[0123] This specification provides an overload protection strategy for MetaIO in a data node. It can protect user I / O under extreme MetaIO scenarios, preventing excessive MetaIO from consuming I / O resources and affecting the data node's user I / O service capabilities. Furthermore, this solution has low complexity, thus reducing implementation costs, and effectively protects the data node's ability to serve user I / O.

[0124] The following one or more embodiments illustrate the data processing method provided in this specification and its testing process in a real-world scenario. Specifically, the background user IO traffic is: numjobs = 2, iodepth = 2, bs256KSyncWrite; the MetaIO traffic is: numjobs = 1, opdepth = 4, Create empty Chunk. By comparing the method where MetaIO traffic enters the MetaIO scheduler with the method where it does not enter the MetaIO scheduler, the impact of this solution on background user IO traffic is determined. The specific results are as follows:

[0125] See Figure 7 , Figure 7 Figure 7 This is a schematic diagram of background I / O throughput in a data processing method provided in one embodiment of this specification, based on... Figure 7 It can be seen that, Figure 7 On the left is MetaIO without entering the MetaIO scheduler: Before MetaIO traffic starts, user IO traffic throughput reaches 60M; after MetaIO traffic starts, user IO traffic throughput drops to 37M, and further to 30M.

[0126] Figure 7 On the right is the MetaIO scheduler: before MetaIO traffic starts, user IO traffic throughput reaches 60M; after MetaIO traffic starts, user IO traffic throughput drops to 42-41M.

[0127] See Figure 8 , Figure 8 This is a schematic diagram illustrating background I / O latency in a data processing method provided in one embodiment of this specification, based on... Figure 8 It can be seen that, Figure 8 On the left is the case where MetaIO is not included in the MetaIO scheduler: before MetaIO traffic starts, the user IO write latency is 30.7ms; after MetaIO traffic starts, the user IO write latency increases to 51ms.

[0128] Figure 8 On the right is the MetaIO scheduler: before MetaIO traffic starts, the user IO write latency is 31ms; after MetaIO traffic starts, the user IO write latency rises to 44ms, and the upward trend is relatively gentle.

[0129] See Figure 9 , Figure 9 This is a schematic diagram illustrating the changes in MetaIOOPS in a data processing method provided in one embodiment of this specification, based on... Figure 9 It can be seen that, Figure 9On the left is MetaIO not entering the MetaIO scheduler: CreateOPS is between 24 and 28, with some fluctuation; Figure 9 On the right is the MetaIO scheduler: CreateOPS is stable at 14, which is relatively stable; it should be noted that the final OPS spike is caused by the decrease in background IO traffic.

[0130] See Figure 10 , Figure 10 This is a schematic diagram of the state of a disk management tool in a data processing method provided in one embodiment of this specification; based on Figure 10 It can be seen that under the mixed stress of the comparative test, the DiskUtil (a disk management tool) of the data nodes is in a fully loaded state.

[0131] In summary, under the above-mentioned high-pressure scenarios of mixed MetaIO and background IO traffic, the MetaIO scheduler designed in this solution can reduce the impact of MetaIO on background IO traffic from the perspective of throughput / latency: for scenarios where MetaIO traffic enters the MetaIO scheduler, the impact on background IO traffic (decrease in throughput / increase in latency) is better than in scenarios where MetaIO traffic does not enter the MetaIO scheduler.

[0132] Corresponding to the above method embodiments, this specification also provides 11 embodiments. Figure 11 A schematic diagram of the structure of a data processing apparatus according to one embodiment of this specification is shown. Figure 11 As shown, the device includes:

[0133] The receiving module 1102 is configured to receive a data processing request sent by a second data node, and when it is determined that the data processing request is a target data processing request, to obtain the current state of the request queue, wherein the target data processing request is a request for system data in a set data node, and the request queue is a queue in the first data node.

[0134] Storage module 1104 is configured to store the target data processing request to the request queue when it is determined that the current state meets the request storage conditions;

[0135] The processing module 1106 is configured to process the target data processing request in the request queue when it is determined that the current request processing status meets the request processing conditions.

[0136] Optionally, the receiving module 1102 is further configured to:

[0137] If the data processing request is determined to be a target data processing request, the request type of the target data processing request is determined;

[0138] Determine the target request queue corresponding to the request type from at least two of the request queues, and obtain the current state of the target request queue.

[0139] Optionally, the storage module 1104 is further configured to:

[0140] If the current state of the target request queue satisfies the request storage conditions, the target data processing request is stored in the target request queue.

[0141] Optionally, the processing module 1106 is further configured to:

[0142] Obtain the current number of requests processed for the target data processing request;

[0143] If the current number of requests to be processed is less than a preset processing threshold, the target data processing requests in the target request queue are processed.

[0144] Optionally, the storage module 1104 is further configured to:

[0145] Determine the number of requests to be stored in the target request queue, and if the number of requests to be stored is less than a preset storage threshold, determine that the current state of the target request queue satisfies the request storage condition.

[0146] Optionally, the data processing apparatus further includes a request processing module, configured to:

[0147] If the data processing request is determined to be a user data processing request, the user data processing request is processed.

[0148] Optionally, the target data processing request is a metadata processing request;

[0149] Accordingly, the processing module 1106 is further configured to:

[0150] Based on the request type of the metadata processing request, the request processing unit corresponding to the metadata processing request is determined, wherein the request processing unit is the processing unit in the first data node that processes the metadata processing request.

[0151] The request processing unit processes the metadata processing request.

[0152] Optionally, the target data processing request is a processing request for the user data structure in the first data node; and / or

[0153] The target data processing request is a processing request for the metadata in the first data node.

[0154] Optionally, the data processing apparatus further includes a request deletion module, configured to:

[0155] Determine the request storage time of the target data processing request in the request queue;

[0156] If the requested storage time exceeds a preset time threshold, the target data processing request is deleted from the request queue.

[0157] The data processing device for the first data node provided in this specification, upon receiving a target data processing request sent by the second data node, determines to store the target data processing request if the current state of the request queue meets the request storage conditions. This avoids request overload and the problem of excessive resource occupation of the first data node due to an excessive number of target data processing requests. Furthermore, during the processing of the target data processing request, the request is only processed if the current request processing state of the first data node meets the request processing conditions. This further avoids target data processing request overload and does not affect user data read / write operations, preventing the first data node from being unable to respond to user data read / write requests in a timely manner due to request overload.

[0158] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.

[0159] Figure 12 A structural block diagram of a computing device 1200 according to an embodiment of this specification is shown. The components of the computing device 1200 include, but are not limited to, a memory 1210 and a processor 1220. The processor 1220 is connected to the memory 1210 via a bus 1230, and a database 1250 is used to store data.

[0160] The computing device 1200 also includes an access device 1240, which enables the computing device 1200 to communicate via one or more networks 1260. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 1240 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0161] In one embodiment of this specification, the aforementioned components of the computing device 1200 and Figure 12 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 12 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0162] The computing device 1200 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 1200 can also be a mobile or stationary server.

[0163] The processor 1220 is configured to execute the following computer-executable instructions, which, when executed by the processor 1220, implement the steps of the above-described data processing method.

[0164] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the data processing method described above.

[0165] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0166] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the data processing method described above.

[0167] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described data processing method.

[0168] The above is an illustrative example of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the data processing method described above belong to the same concept. Details not described in detail in the technical solution of the computer program can be found in the description of the technical solution of the data processing method described above.

[0169] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0170] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0171] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0172] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0173] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method, applied to a first data node, comprising: The system receives a data processing request sent by a second data node. If the data processing request is determined to be a target data processing request, the system determines the request type of the target data processing request, determines the target request queue corresponding to the request type from at least two request queues, and obtains the current state of the target request queue. The target data processing request is a request for system data in the first data node, and the target data processing request includes a Create request, a Seal request, or a Datasync request. The request queue is a queue in the first data node, and the target request queue includes a Create queue, a Seal queue, or a Datasync queue. The current state of the target request queue indicates that the target request queue currently stores request information. If the current state satisfies the storage request condition, the target data processing request is stored in the target request queue. The storage condition includes that the number of requests to be stored in the target request queue is less than a preset storage quantity threshold. The number of requests to be stored represents the number of target data processing requests stored in the target request queue. If the current request processing status meets the request processing conditions, the target data processing request in the target request queue is processed. The request processing conditions include that the current number of requests to the target data processing request is less than a preset processing number threshold. Processing the target data processing request in the target request queue includes: the target data processing request is a metadata processing request; based on the request type of the metadata processing request, a request processing unit corresponding to the metadata processing request is determined; and the metadata processing request is processed by the request processing unit. The metadata processing request includes a Create request, a Seal request, or a Datasync request. The request processing unit includes the storage engine or disk of the first data node. If the data processing request is determined to be a user data processing request, the user data processing request is processed, wherein processing the user data processing request includes sending the user data processing request to the storage engine.

2. The data processing method according to claim 1, wherein the target data processing request is a processing request for the user data structure in the first data node.

3. The data processing method according to claim 1, further comprising, after storing the target data processing request to the request queue: Determine the request storage time of the target data processing request in the request queue; If the requested storage time exceeds a preset time threshold, the target data processing request is deleted from the request queue.

4. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the data processing method according to any one of claims 1 to 3.

5. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 3.

Images