Method, device, equipment and storage medium for tracking storage IO execution status

Abstract

A method, device, equipment and storage medium for tracking storage IO execution, relating to the technical field of data storage, wherein the method for tracking storage IO execution includes: when a node receives a remote procedure call (rpc) message, a tracker is assigned, and the rpc id, message type and start timestamp corresponding to the rpc message are recorded in the tracker; a timestamp and a key process name are inserted into the rpc key process; the reply timestamp of the node rpc reply is recorded, and whether the rpc processing time has timed out is determined based on the start timestamp and message type; when the rpc processing timestamp is timed out, the rpc key process timestamp is searched at the corresponding node based on the rpc id for auxiliary positioning. The present application can effectively monitor the IO processing in distributed storage, retain problem positioning information, and increase the locatability of the distributed storage system.

Description

Technical Field

[0001] The present application relates to the field of data storage technology, and in particular to a method, device, equipment and storage medium for tracking storage IO execution status. Background Art

[0002] As the number of servers in a distributed storage system increases, the failure rate also increases. In order to ensure that the system is still available when a server fails, the general practice is to divide a piece of data into multiple copies and store them in different servers, using different redundancy strategies according to needs to ensure data reliability and fault tolerance.

[0003] Taking three-copy write as an example, the data is replicated in three copies and stored on three different nodes, with one master copy and two slave copies. The client sends an RPC (Remote Procedure Call) request (IO) to the master copy node of the server, which is then distributed to the other two slave copies via the network.

[0004] In order to assist in locating bugs such as OP exceptions and timeouts, it is necessary to monitor the latency on the IO (Input / Output) path. Existing IO latency monitoring methods mostly monitor the average latency of a certain process, which has defects such as not being able to accurately determine the specific IO and not being able to monitor across boundaries. If the average latency is as expected but a certain IO execution time times out, the cause of the timeout cannot be accurately located. Summary of the invention

[0005] The present application provides a method, apparatus, device and storage medium for tracking storage IO execution status, which can effectively monitor IO processing status in distributed storage, retain problem location information, and increase the locatability of distributed storage systems.

[0006] In a first aspect, an embodiment of the present application provides a method for tracking storage IO execution status, the method for tracking storage IO execution status comprising:

[0007] When a node receives a remote procedure call (rpc) message, it allocates a tracker and records the rpc id, message type, and start timestamp corresponding to the rpc message in the tracker.

[0008] Insert timestamp and key process name into the key process of rpc;

[0009] Record the reply timestamp of the node's rpc reply, and determine whether the rpc processing time has timed out based on the start timestamp and message type;

[0010] When the RPC processing times out, the RPC key process timestamp is searched at the corresponding node according to the RPC ID for auxiliary positioning.

[0011] In combination with the first aspect, in one implementation, recording the reply timestamp of the node rpc reply, and calculating whether the rpc processing time has timed out according to the start timestamp and the message type, includes:

[0012] Determine whether the difference between the reply timestamp and the start timestamp exceeds the time threshold set based on the message type;

[0013] If the set time threshold is exceeded, the RPC processing time is determined to have timed out.

[0014] In combination with the first aspect, in one implementation, different time thresholds are set according to different message types.

[0015] In combination with the first aspect, in one implementation, when allocating a tracker, the tracker is taken out from the tracker pool and added to the inflight queue.

[0016] In combination with the first aspect, in one implementation, if the tracker pool is empty, a tracker is applied for through the alloc function, and the applied tracker is placed in the tracker pool when released.

[0017] In combination with the first aspect, in one implementation, if the RPC falls into an infinite loop or actively switches out and does not continue processing, a watchdog is used to detect the start timestamp of the RPC in the inflight queue at regular intervals. If the threshold is exceeded and not processed, the information is output to the log and a fault recovery alarm is issued.

[0018] In combination with the first aspect, in one implementation, when the RPC processing times out, the recorded RPC key process timestamp is output to the log and an alarm is issued.

[0019] In a second aspect, an embodiment of the present application provides a device for tracking storage IO execution status, wherein the device for tracking storage IO execution status includes:

[0020] An allocation module is used to allocate a tracker when a node receives a remote procedure call (rpc) message, and record the rpc id, message type and start timestamp corresponding to the rpc message into the tracker;

[0021] Insert module, which is used to insert timestamp and key process name in the key process of rpc;

[0022] The judgment module is used to record the reply timestamp of the node rpc reply and judge whether the rpc processing time has timed out according to the start timestamp and message type;

[0023] The positioning module is used to find the RPC key process timestamp in the corresponding node according to the RPC ID when the RPC processing times out, so as to assist in positioning.

[0024] In a third aspect, an embodiment of the present application provides a device for tracking storage IO execution status, wherein the device for tracking storage IO execution status comprises a processor, a memory, and a program for tracking storage IO execution status stored in the memory and executable by the processor, wherein when the program for tracking storage IO execution status is executed by the processor, the steps of the method for tracking storage IO execution status as described in any one of the above are implemented.

[0025] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which is stored a program for tracking storage IO execution status, wherein when the program for tracking storage IO execution status is executed by a processor, the steps of the method for tracking storage IO execution status as described in any one of the above are implemented.

[0026] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0027] The method for tracking storage IO execution status in the present application allocates a tracker when a node receives a remote procedure call (rpc) message, and records the rpc id, message type and start timestamp corresponding to the rpc message in the tracker; inserts the timestamp and key process name into the rpc key process; records the reply timestamp of the node rpc reply, and determines whether the rpc processing time has timed out based on the start timestamp and message type; when the rpc processing times out, searches for the rpc key process timestamp at the corresponding node based on the rpc id for auxiliary positioning.

[0028] The solution in this application maintains a tracker memory pool on each thread to store tracker instances, which can reduce memory application release and reduce CPU and memory consumption. Using rpc id as a unique identifier, different threads use different trackers to track, achieving lock-free; in addition, using rpc id as a unique identifier, latency can also be tracked across nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A flowchart of an embodiment of a method for tracking storage IO execution status of the present application;

[0030] Figure 2 Flowchart of tracker assignment for this application;

[0031] Figure 3 This is the RPC processing flow chart for this application node;

[0032] Figure 4 A functional module diagram of an embodiment of a device for tracking storage IO execution status of the present application;

[0033] Figure 5 This is a schematic diagram of the hardware structure of a device for tracking storage IO execution status involved in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0035] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit "first", "second" and "third" to different types.

[0036] In the description of the embodiments of the present application, "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a specific way.

[0037] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0038] In some processes described in the embodiments of the present application, multiple operations or steps that appear in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or in parallel, and the sequence number of the operation is only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.

[0039] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0040] In a first aspect, an embodiment of the present application provides a method for tracking storage IO execution status.

[0041] In one embodiment, referring to Figure 1 , Figure 1 This is a flow chart of an embodiment of a method for tracking storage IO execution status of the present application. Figure 1 As shown, the method for tracking storage IO execution includes:

[0042] S1. When a node receives a remote procedure call (rpc) message, it allocates a tracker and records the rpc id, message type (rpc type) and start timestamp corresponding to the rpc message into the tracker.

[0043] See also Figure 2 As shown, in this embodiment, when allocating a tracker, the method used is to take out the tracker from the tracker pool and add the tracker to the inflight queue. If the tracker pool is empty, the tracker is applied for through the alloc function, and the applied tracker is put into the tracker pool when released, so as to reduce the number of memory application and release times.

[0044] See also Figure 3As shown, the nodes in this embodiment include a master node and a slave node. The master node receives rpc1 sent by the Client Write Master op, and assigns tracker1 to record rpc1 id to tracker1. Subsequently, the master node distributes the slave replica write op and sends rpc2. At this time, the slave node receives rpc2 written from the slave replica, and assigns tracker2 to record rpc2 id to tracker2. It is worth noting that the master node's tracker1 will also record rpc2 id to ensure docking with the slave node.

[0045] It is understandable that for the case of multiple slave nodes, such as the case of three copies of writing, there are two slave nodes, and the processing method of the two slave nodes is similar, refer to Figure 3 The above description can be used for processing, and the present invention will not be elaborated here.

[0046] S2. Insert timestamp and key process name into the key process of rpc.

[0047] The key process referred to in this embodiment refers to the process that is often experienced during the RPC processing process, which is also the focus of this application. There are many key processes (several dozen of them are subdivided). Figure 3 The following are three main processes as examples. After the distributed storage receives a write operation:

[0048] 1. You need to prepare the op context information (context, which stores the object where the op is located, its address and offset, master-slave shard distribution, etc.);

[0049] 2. Data needs to be written to the physical hard disk;

[0050] 3. The metadata information of the object needs to be updated, including the location on the hard disk corresponding to this op, the transaction, etc.

[0051] There will be timestamps before and after each key process. If the operation times out, the process that is stuck can be calculated based on these timestamps.

[0052] In addition, timestamps can use integer compression algorithms to reduce memory. Key process names are represented by indexes when nodes are initialized, and compression dictionary algorithm fields are used to reduce memory.

[0053] S3. Record the reply timestamp of the node's rpc reply, and determine whether the rpc processing time has timed out based on the start timestamp and message type.

[0054] Specifically, step S3 includes:

[0055] S31, determining whether the difference between the reply timestamp and the start timestamp exceeds a time threshold set based on the message type;

[0056] S33. If the set time threshold is exceeded, it is determined that the RPC processing time has timed out.

[0057] by Figure 3 For example, the master node corresponds to timestamps 1, 2, 3, 9, 10, and 11, and the slave node corresponds to timestamps 4, 5, 6, 7, and 8. The master node and the slave node set the time thresholds timeout1 and timeout2 respectively.

[0058] For the slave node, if stamp8-stamp4>timeout2, it is determined that the rpc processing time has timed out.

[0059] For the master node, if stamp11-stamp1>timeout1, it is determined that the RPC processing time has timed out.

[0060] It is understandable that tracker1 will record the ID of rpc2 of the slave replica. Through this ID, the timeout information can be found on the node where the slave replica is located. If not, it means that the process of writing to the slave replica has not timed out, and the problem lies in other processes of the master op.

[0061] It is worth noting that in some embodiments, different types of messages can customize their own timeouts, that is, different time thresholds can be set according to different message types. If the rpc type is update, the message needs to be processed by multiple nodes before returning.

[0062] S4. When the RPC processing times out, the RPC key process timestamp is searched at the corresponding node according to the RPC ID for auxiliary positioning.

[0063] In this embodiment, when the RPC processing times out, the recorded RPC key process timestamp will be output to the log and an alarm will be issued.

[0064] For example, when tracking, if a timeout alarm is generated, the cause of the timeout is that the processing time from the replica is too long. By going from the replica's unique identifier rpc id to its node, you can find the information printed from the replica tracker, and then track the flow of messages processed across nodes to assist in locating the problem.

[0065] In addition, if there are bugs or system failures, and the RPC falls into an infinite loop or actively switches out and stops processing at a certain stage, the watchdog can be used to detect the start timestamp of the RPC in the tracker inflight queue at regular intervals. If the threshold is exceeded and not processed, its information can be output to the log and fault recovery alarm and other processing can be performed.

[0066] Specifically, after the watchdog is turned on, the tracker inflight queue will be traversed, and then the tracker start timestamp will be used to determine whether it has timed out. If it has not timed out, the watchdog will go into sleep mode. If it has timed out, the tracker location information will be dumped and an alarm will be reported.

[0067] To summarize, the method for tracking storage IO execution in the present application allocates a tracker when a node receives a remote procedure call rpc message, and records the rpc id, message type and start timestamp corresponding to the rpc message into the tracker; inserts the timestamp and key process name into the key process of the rpc; records the reply timestamp of the node rpc reply, and determines whether the rpc processing time has timed out based on the start timestamp and message type; when the rpc processing times out, searches for the rpc key process timestamp at the corresponding node based on the rpc id for auxiliary positioning.

[0068] The solution in this application maintains a tracker memory pool on each thread to store tracker instances, which can reduce memory application and release, and reduce CPU and memory consumption. Using rpc id as the unique identifier, different threads use different trackers to track and achieve lock-free; in addition, using rpc id as the unique identifier, latency can also be tracked across nodes. In this way, the latency of key nodes on each IO execution path can be recorded in memory, and automatically dumped when IO is abnormal to assist in positioning. In addition, timestamps, key point information, point names and meanings are compressed using compression algorithms to reduce memory consumption.

[0069] In a second aspect, an embodiment of the present application also provides a device for tracking storage IO execution status.

[0070] In one embodiment, referring to Figure 4 , Figure 4 This is a functional module diagram of an embodiment of a device for tracking storage IO execution status of the present application. Figure 4 As shown, the device for tracking storage IO execution status includes:

[0071] An allocation module is used to allocate a tracker when a node receives a remote procedure call (rpc) message, and record the rpc id, message type and start timestamp corresponding to the rpc message into the tracker;

[0072] Insert module, which is used to insert timestamp and key process name in the key process of rpc;

[0073] The judgment module is used to record the reply timestamp of the node rpc reply and judge whether the rpc processing time has timed out according to the start timestamp and message type;

[0074] The positioning module is used to find the RPC key process timestamp in the corresponding node according to the RPC ID when the RPC processing times out, so as to assist in positioning.

[0075] Further, in one embodiment, the allocation module records the reply timestamp of the node RPC reply, and calculates whether the RPC processing time has timed out according to the start timestamp and the message type, including:

[0076] Determine whether the difference between the reply timestamp and the start timestamp exceeds the time threshold set based on the message type;

[0077] If the set time threshold is exceeded, the RPC processing time is determined to have timed out.

[0078] Furthermore, in one embodiment, the allocation module sets different time thresholds according to different message types.

[0079] Furthermore, in one embodiment, when the allocation module allocates a tracker, the allocation module takes the tracker from the tracker pool and adds the tracker to the inflight queue.

[0080] Furthermore, in one embodiment, the allocation module is also used for:

[0081] If the tracker pool is empty, a tracker is applied for through the alloc function, and the applied tracker is placed in the tracker pool when it is released.

[0082] Furthermore, in one embodiment, the positioning module is also used for:

[0083] If the RPC falls into an infinite loop or is actively cut out and does not continue processing, the watchdog is used to detect the start timestamp of the RPC in the inflight queue at regular intervals. If it exceeds the threshold and is not processed, the information is output to the log and a fault recovery alarm is issued.

[0084] Furthermore, in one embodiment, the positioning module is also used for:

[0085] When the RPC processing times out, the recorded RPC key process timestamp is output to the log and an alarm is issued.

[0086] Among them, the functional implementation of each module in the above-mentioned device for tracking storage IO execution status corresponds to each step in the above-mentioned method embodiment for tracking storage IO execution status, and its functions and implementation processes are no longer repeated here.

[0087] In summary, the device for tracking storage IO execution in the present application includes an allocation module, an insertion module, a judgment module and a positioning module. The allocation module is used to allocate a tracker when a node receives a remote procedure call rpc message, and record the rpc id, message type and start timestamp corresponding to the rpc message into the tracker; the insertion module is used to insert the timestamp and key process name into the key process of the rpc; the judgment module is used to record the reply timestamp of the node rpc reply, and judge whether the rpc processing time has timed out according to the start timestamp and message type; the positioning module is used to search for the rpc key process timestamp in the corresponding node according to the rpc id when the rpc processing times out, so as to assist in positioning.

[0088] The solution in this application maintains a tracker memory pool on each thread to store tracker instances, which can reduce memory application and release, and reduce CPU and memory consumption. Using rpc id as the unique identifier, different threads use different trackers to track and achieve lock-free; in addition, using rpc id as the unique identifier, latency can also be tracked across nodes. In this way, the latency of key nodes on each IO execution path can be recorded in memory, and automatically dumped when IO is abnormal to assist in positioning. In addition, timestamps, key point information, point names and meanings are compressed using compression algorithms to reduce memory consumption.

[0089] In a third aspect, an embodiment of the present application provides a device for tracking storage IO execution status. The device for tracking storage IO execution status may be a personal computer (PC), a laptop computer, a server, or other device with data processing capabilities.

[0090] Reference Figure 5 , Figure 5 The hardware structure diagram of the device for tracking storage IO execution status involved in the embodiment of the present application is shown in FIG. In the embodiment of the present application, the device for tracking storage IO execution status may include a processor, a memory, a communication interface, and a communication bus.

[0091] The communication bus may be of any type and is used to interconnect the processor, the memory, and the communication interface.

[0092] The communication interface includes an input / output (I / O) interface, a physical interface, and a logical interface, which are used to interconnect the internal devices of the device for tracking the storage IO execution status, and an interface for interconnecting the device for tracking the storage IO execution status with other devices (such as other computing devices or user devices). The physical interface can be an Ethernet interface, a fiber optic interface, an ATM interface, etc.; the user device can be a display, a keyboard, etc.

[0093] The memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0094] The processor may be a general-purpose processor, and the general-purpose processor may call the program for tracking the storage IO execution status stored in the memory, and execute the method for tracking the storage IO execution status provided in the embodiment of the present application. For example, the general-purpose processor may be a central processing unit (CPU). The method executed when the program for tracking the storage IO execution status is called may refer to the various embodiments of the method for tracking the storage IO execution status of the present application, and will not be repeated here.

[0095] Those skilled in the art will understand that Figure 5 The hardware structure shown in the figure does not constitute a limitation on the present application, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0096] In a fourth aspect, an embodiment of the present application also provides a computer-readable storage medium.

[0097] The computer-readable storage medium of the present application stores a program for tracking storage IO execution status, wherein when the program for tracking storage IO execution status is executed by a processor, the steps of the method for tracking storage IO execution status as described above are implemented.

[0098] Among them, the method implemented when the program for tracking the storage IO execution status is executed can refer to the various embodiments of the method for tracking the storage IO execution status of this application, and will not be repeated here.

[0099] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0100] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present application.

[0101] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for tracking storage IO execution, characterized in that: The method for tracking storage IO execution includes: When a node receives a remote procedure call (rpc) message, it allocates a tracker and records the rpcid, message type and start timestamp corresponding to the rpc message into the tracker. A tracker memory pool is maintained on each thread to store tracker instances, and different threads use different trackers for tracking. When a tracker is allocated, the tracker is taken out of the tracker pool and added to the tracker inflight queue. Insert timestamp and key process name into the key process of rpc; Record the reply timestamp of the node's rpc reply, and determine whether the rpc processing time has timed out based on the start timestamp and message type; When the RPC processing times out, the RPC key process timestamp is searched at the corresponding node according to the RPC ID for auxiliary positioning.

2. The method for tracking storage IO execution as claimed in claim 1, characterized in that: The reply timestamp of the recording node rpcreply, and calculating whether the rpc processing time has timed out according to the start timestamp and the message type, include: Determine whether the difference between the reply timestamp and the start timestamp exceeds the time threshold set based on the message type; If the set time threshold is exceeded, the RPC processing time is determined to have timed out.

3. The method for tracking storage IO execution as claimed in claim 2, characterized in that: Set different time thresholds according to different message types.

4. The method for tracking storage IO execution as claimed in claim 1, characterized in that: If the tracker pool is empty, a tracker is applied for through the alloc function, and the applied tracker is placed in the tracker pool when it is released.

5. The method for tracking storage IO execution as claimed in claim 1, characterized in that: If the RPC falls into an infinite loop or is actively cut out and does not continue processing, the watchdog is used to detect the start timestamp of the RPC in the trackerinflight queue at regular intervals. If it exceeds the threshold and is not processed, the information is output to the log and a fault recovery alarm is issued.

6. The method for tracking storage IO execution as claimed in claim 1, characterized in that: When the RPC processing times out, the recorded RPC key process timestamp is output to the log and an alarm is issued.

7. A device for tracking storage IO execution, characterized in that: The device for tracking storage IO execution status includes: The allocation module is used to allocate a tracker when a node receives a remote procedure call (rpc) message, and record the rpc id, message type and start timestamp corresponding to the rpc message into the tracker. A tracker memory pool is maintained on each thread to store tracker instances, and different threads use different trackers for tracking. When allocating a tracker, the tracker is taken out from the tracker pool and added to the tracker inflight queue. Insert module, which is used to insert timestamp and key process name in the key process of rpc; The judgment module is used to record the reply timestamp of the node rpc reply and judge whether the rpc processing time has timed out according to the start timestamp and message type; The positioning module is used to find the RPC key process timestamp in the corresponding node according to the RPC ID when the RPC processing times out, so as to assist in positioning.

8. A device for tracking storage IO execution, characterized in that: The device for tracking storage IO execution status includes a processor, a memory, and a program for tracking storage IO execution status stored in the memory and executable by the processor, wherein when the program for tracking storage IO execution status is executed by the processor, the steps of the method for tracking storage IO execution status as described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program for tracking storage IO execution status, wherein when the program for tracking storage IO execution status is executed by a processor, the steps of the method for tracking storage IO execution status according to any one of claims 1 to 6 are implemented.

Images