A Storage I / O Multipath Splitting Method and System Based on Cgroup Source Tracing

Abstract

This invention relates to a storage I / O multi-path routing method and system based on Cgroup tracing, belonging to the field of data storage technology. It solves the problem of performance degradation caused by the inability of existing multi-path routing to perceive business priorities and isolate abnormal traffic. The method includes: obtaining the Cgroup identity information to which the I / O request belongs, and then determining the path channel type corresponding to the I / O request; path channel types include: dedicated channels, isolated channels, and general channels; determining the physical path set corresponding to the path channel type from multiple pre-divided physical path sets, where different path channel types correspond to different physical path sets; selecting a physical path from the physical path set as the target path, and sending the I / O request to the storage device through the target path. This achieves business-aware, accurate multi-path routing, improving business performance.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and in particular to a storage I / O multipath routing method and system based on Cgroup tracing. Background Technology

[0002] With the rapid development of cloud computing, containerization, and software-defined storage technologies, shared storage environments have become the mainstream deployment model in data centers. In such environments, multiple tenants, applications, or container instances (such as cloud database instances or Kubernetes Pods) typically share the same backend storage cluster and rely on multipath I / O (MPIO) technology to achieve high availability and bandwidth aggregation. Multipath technology abstracts physical storage paths into a single logical device, enabling automatic switching in case of path failure and distributing input / output (I / O) requests across multiple paths.

[0003] Currently, the standard multipath framework in the Linux operating system is DM-Multipath (Device MapperMultipath), whose path selection strategy is mainly based on physical layer state information. Existing mainstream path selection strategies include round-robin, service-time minimization, and queue-length minimization. These strategies allocate I / O based solely on real-time performance metrics (latency, queue depth) or a simple round-robin mechanism. In simple shared scenarios with relatively uniform business flow and no sudden abnormal traffic, these technologies can effectively improve overall throughput and availability.

[0004] However, in complex multi-tenant production environments, when a low-priority business or abnormal process experiences a sudden surge in I / O (such as batch log writing, backup tasks, or runaway processes), traditional multipathing treats it as ordinary traffic and distributes it indiscriminately across all available paths. This causes I / O requests from critical business processes (such as database transaction processing) to compete with these sudden surges in traffic on the same physical links, triggering a "noisy neighbor" effect. This results in latency jitter and decreased throughput for critical business processes, and in severe cases, impacts service level agreements (SLAs). Summary of the Invention

[0005] Based on the above analysis, the embodiments of the present invention aim to provide a storage I / O multi-path routing method and system based on Cgroup tracing, in order to solve the problem that existing multi-path routing cannot perceive business priorities and cannot isolate abnormal traffic, resulting in a decline in business performance.

[0006] On one hand, embodiments of the present invention provide a storage I / O multipath routing method based on Cgroup tracing, including the following steps: Obtain the Cgroup identity information to which the IO request belongs, and then determine the path channel type corresponding to the IO request; the path channel types include: dedicated channel, isolated channel and general channel; From a pre-defined set of multiple physical paths, determine the set of physical paths corresponding to the path channel type, where different path channel types correspond to different sets of physical paths; Select a physical path from the set of physical paths as the target path, and send the IO request to the storage device through the target path.

[0007] Based on the further improvement of the above method, the identity information of the Cgroup to which the IO request belongs is obtained, including: Obtain the request object corresponding to the IO request. The request object is encapsulated in kernel space as a data structure containing a complete business context. Based on the pointer chain in the data structure of the request object, trace back to obtain the Cgroup identity information to which the process that initiated the IO request belongs.

[0008] Based on further improvements to the above method, the path channel type corresponding to the IO request is determined, including: Based on the Cgroup identity information, locate the kernel control block of the Cgroup and read the policy identifier value from the extended policy identifier field of the kernel control block; Query the mapping relationship between policy identifier values ​​and path channel types maintained in the kernel private data area, and determine the path channel type corresponding to the IO request based on the policy identifier value.

[0009] Based on the further improvement of the above method, the policy identifier value is written into the policy identifier field in the Cgroup kernel control block through the following steps: Add a new virtual file mapping to the corresponding policy identifier field in each Cgroup directory of the Cgroup virtual file system; By writing the policy identifier value to the virtual file, a kernel callback function is triggered to store the policy identifier value in the policy identifier field.

[0010] Based on the further improvements of the above method, the mapping relationship between policy identifier values ​​and path channel types, as well as the pre-divided sets of multiple physical paths, are all obtained by the user-mode daemon process parsing the user-mode configuration file and stored in the kernel private data area.

[0011] Based on the further improvement of the above method, the user-space configuration file includes: defining the corresponding path channel type, physical path set, rate limiting threshold and path selection algorithm for each policy identifier value; the physical path set includes the global identifier of one or more storage devices or the master / slave device number of the physical path.

[0012] Based on further improvements to the above method, a physical path is selected as the target path from the set of physical paths, including: After removing faulty physical paths from the physical path set, obtain the static priority weight and the number of real-time in-transit IO requests for each normal physical path; calculate the comprehensive score of each normal physical path based on the static priority weight and the number of real-time in-transit IO requests, and select the normal physical path with the highest comprehensive score as the target path.

[0013] Based on the further improvement of the above method, when the path channel type corresponding to the IO request is an isolated channel, before selecting a physical path as the target path from the physical path set, the method further includes: obtaining the real-time IO throughput of the Cgroup to which the IO request belongs; comparing the real-time IO throughput with a preset rate limiting threshold; if the real-time IO throughput exceeds the rate limiting threshold, suspending the IO request until the real-time IO throughput drops below the rate limiting threshold.

[0014] Based on the above method, the method further includes: when the path channel type corresponding to the IO request is a general channel, real-time monitoring of the real-time IO latency and utilization of the Cgroup to which the IO request belongs; when the real-time IO latency and utilization of the Cgroup are both detected to exceed the preset corresponding abnormal threshold, the policy identifier value in the kernel control block of the Cgroup is modified to the value corresponding to the isolated channel.

[0015] On the other hand, embodiments of the present invention provide a storage I / O multipath routing system based on Cgroup tracing, comprising: The channel type determination module is used to obtain the Cgroup identity information to which the IO request belongs, and then determine the path channel type corresponding to the IO request; the path channel types include: dedicated channel, isolated channel and general channel; The path set matching module is used to determine the physical path set corresponding to the path channel type from multiple pre-divided physical path sets, where different path channel types correspond to different physical path sets; The path selection and distribution module is used to select a physical path from the set of physical paths as the target path and distribute the IO request to the storage device through the target path.

[0016] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects: 1. By obtaining the Cgroup identity information to which the IO request belongs, the corresponding path channel type (dedicated / isolated / general) is determined, and the target path is selected from the pre-divided set of physical paths for delivery. This method introduces the upper-layer Cgroup business label into the kernel multi-path layer for the first time, realizing a complete closed loop from "business identity recognition" to "channel type matching", then to "physical path set determination" and "optimal path selection". This enables business IO of different priorities to be automatically guided to different physical path resources, realizing accurate IO path diversion based on business identity, and fundamentally solving the problem of critical business and ordinary business competing for path resources.

[0017] 2. Flexible configuration of different selection strategies for different channel types ensures dedicated resources for critical services while achieving efficient utilization of overall storage bandwidth: For dedicated channels, static priority weights and real-time load are combined to select the target path, achieving dynamic balancing within dedicated resources. This ensures performance isolation while avoiding overload on a single path. For isolated channels, rate limiting is added to prevent low-priority or abnormal services from consuming unlimited bandwidth even when redirected to isolated channels, thus achieving fair resource allocation in a shared storage environment. For general channels, a lightweight monitor using kernel statistics is used to monitor the IO latency and device utilization of each Cgroup in real time. When the latency and utilization of a Cgroup both exceed the abnormal threshold, the policy flag value in its kernel control block is automatically modified, instantly switching subsequent IO requests to the isolated channel. This process requires no user-space intervention, with a switching latency in the millisecond range, and can proactively suppress the impact of sudden traffic from "noisy neighbors" on other normal services in the general channel.

[0018] 3. Different policies can be defined through user-space configuration files and dynamically synchronized to the kernel private data area through user-space daemons. This allows for flexible adjustment of the binding relationship between services and path resources without modifying kernel code, restarting the system, or rebuilding multipath devices. At the same time, Cgroup service tags can be written at any time through the Cgroup virtual file system interface, supporting dynamic scheduling in containerized environments and perfectly adapting to the elastic requirements of cloud-native platforms.

[0019] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0020] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. Figure 1 This is a flowchart of a storage I / O multipath routing method based on Cgroup tracing in Embodiment 1 of the present invention. Detailed Implementation

[0021] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0022] Example 1 A specific embodiment of the present invention discloses a storage I / O multipath routing method based on Cgroup tracing, such as... Figure 1 As shown, it includes the following steps: S1. Obtain the Cgroup identity information to which the IO request belongs, and then determine the path channel type corresponding to the IO request; the path channel types include: dedicated channel, isolated channel and general channel; S2. Determine the set of physical paths corresponding to the path channel type from multiple pre-divided sets of physical paths, where different path channel types correspond to different sets of physical paths; S3. Select a physical path from the set of physical paths as the target path, and send the IO request to the storage device through the target path.

[0023] In practice, the method in this embodiment runs on the Linux kernel's IO stack. It introduces the business identity information identified by the Linux kernel's Cgroup mechanism into the path selection decision process of the DM-Multipath framework. Through the above three steps, a complete closed loop is achieved from "business identity recognition" to "channel type matching", then to "physical path set determination" and "optimal path selection". This enables business IO of different priorities to be automatically diverted to different physical path channels, providing business awareness and intelligent diversion capabilities, thereby ensuring the performance of critical businesses and isolating abnormal traffic.

[0024] In this embodiment, when an upper-layer application (such as a cloud database instance or a container process in a Kubernetes Pod) initiates an IO request to read / write a storage device, the IO request is passed down the Linux kernel IO stack and eventually reaches the DM-Multipath layer. In order to distinguish IO requests with different business priorities during path selection, this solution first obtains the Cgroup identity information to which the IO request belongs in step S1.

[0025] It's important to note that Cgroups (Control Groups) are a mechanism provided by the Linux kernel for grouping processes and allocating and controlling system resources (such as CPU, memory, and block device I / O) within each group. Each Cgroup corresponds to a kernel control block (such as struct blkcg), which stores the group's resource limits and statistics. In containerized environments, each container or Pod is assigned an independent Cgroup, allowing for fine-grained management of its resource usage.

[0026] In the traditional Linux kernel DM-Multipath framework, the select_path function interface of the path selector only receives the length of the I / O request and cannot obtain other context information of the I / O request. This embodiment extends the interface to receive the complete request object.

[0027] Specifically, in the kernel I / O stack, each I / O request is encapsulated as a request object, which is then encapsulated in kernel mode as a data structure containing a complete business context. This data structure is a struct request structure, which is associated with one or more struct bio structures. The bio structure holds a pointer to the kernel control block (i.e., the struct blkcg structure) of the Cgroup to which the process that initiated the I / O request belongs.

[0028] Furthermore, based on the pointer chain in the request object's data structure, the system sequentially traverses the `bio` structure and the `blkcg` structure to retrieve the Cgroup identity information of the process initiating the I / O request. Specifically, it obtains the corresponding `bio` structure through the `bio` pointer in the request object, and then uses the pointer in the `bio` structure to the `blkcg` structure to trace back to the kernel control block corresponding to the Cgroup to which the process initiating the I / O request belongs, thereby obtaining the Cgroup identity information. The Cgroup identity information includes: a pointer to the Cgroup kernel control block, a Cgroup identifier, or a Cgroup pathname.

[0029] The above extensions break the limitation of traditional multipath technology in being unable to perceive business identity, enabling the path selector to accurately distinguish the business source of different IO requests, providing the necessary prerequisite for subsequent path diversion based on business priority.

[0030] Furthermore, based on the Cgroup identity information, the path channel type that the current IO request should use is determined, including: ① Locate the kernel control block of the Cgroup based on the Cgroup identity information, and read the policy identifier value from the extended policy identifier field of the kernel control block.

[0031] As mentioned earlier, each Cgroup corresponds to a kernel control block (i.e., a struct blkcg structure) in the kernel. This embodiment extends this kernel control block by adding a policy identifier field (e.g., multipath_policy_id) to store the policy identifier value (e.g., policy_id) bound to the Cgroup. This policy identifier value is an integer value, such as 0, 1, 2, etc., and each value represents a predefined path selection policy.

[0032] It should be noted that the policy identifier value is written to the policy identifier field in the Cgroup kernel control block through the following steps: Add a new virtual file, such as blkio.multipath_policy, to each Cgroup directory in the Cgroup virtual file system (cgroupfs); this virtual file is directly mapped to the policy identifier field of the corresponding Cgroup kernel control block; Users write policy identifier values ​​to a virtual file using user-space command-line tools, triggering a kernel callback function to store the policy identifier values ​​in the policy identifier field.

[0033] For example, in a shared storage environment, a high-priority database container (Cgroup name "database") and a low-priority backup task container (Cgroup name "backup") are running. Users can set the policy identifier using the following command: Set the policy flag value to 1 for the database container Cgroup: echo 1> / sys / fs / cgroup / blkio / database / blkio.multipath_policy; Set the policy flag value to 2 for the backup task container Cgroup: echo 2> / sys / fs / cgroup / blkio / backup / blkio.multipath_policy; When a user performs the above write operation, the Cgroup in the kernel triggers the corresponding callback function, saving the written integer value to the multipath_policy_id field of the Cgroup's kernel control block. Once written, this field is persisted in kernel memory, and all subsequent IO requests generated by this Cgroup can obtain the policy identifier by reading this field.

[0034] This extension reuses and enhances the existing Cgroup configuration framework in the Linux kernel, allowing business priority settings to be fully integrated into the standard Cgroup management process without the need for additional complex tools. Administrators can batch configure IO policies for different tenants and containers using scripts or automation platforms, achieving flexibility and scalability in policy configuration.

[0035] Furthermore, after obtaining the kernel control block of the Cgroup to which the IO request belongs through the IO request object, the extended policy identifier field in that kernel control block can be read to obtain the policy identifier value of the Cgroup to which the IO request belongs. For example, for an IO request from a database Cgroup, the read policy identifier value is 1; for an IO request from a backup task Cgroup, the read policy identifier value is 2; and for a Cgroup that is not specifically configured, the read value is 0 (the default value).

[0036] ② Query the mapping relationship between policy identifier values ​​and path channel types maintained in the kernel private data area, and determine the path channel type corresponding to the IO request based on the policy identifier value.

[0037] It should be noted that the kernel private data area refers to the kernel private data area of ​​the multipath device. The multipath device is a logical device created by the Linux kernel DM-Multipath framework, which abstracts multiple physical paths pointing to the same storage unit into a unified device. In this embodiment, a mapping structure is extended within the private data area of ​​the multipath device to maintain the mapping relationship between policy identifier values ​​and path channel types. For example, an integer array `policy_to_channel[]` is defined, where the array index is the policy identifier value, and the array element takes the corresponding channel type enumeration value. The channel type enumeration values ​​include at least three types: dedicated channel, isolated channel, and general channel.

[0038] Furthermore, the mapping between policy identifier values ​​and path channel types is not hard-coded into the code, but is flexibly defined through user-space configuration files (such as / etc / multipath.conf). When the user-space daemon (multipathd) starts or receives a configuration reload signal, it parses this configuration file and synchronizes the mapping to the kernel private data area of ​​the multipath device via a kernel interface (such as Netlink or Sysfs). This ensures both configuration flexibility and efficient policy lookup in kernel space.

[0039] Further, in step S2, based on the channel type, the set of physical paths corresponding to the channel type is found from the pre-divided set of multiple physical paths.

[0040] It should be noted that, similar to the mapping relationship between the policy identifier value and the path channel type mentioned above, the pre-divided set of multiple physical paths is also predefined in the user-space configuration file.

[0041] Specifically, the user-space configuration file includes: defining the corresponding path channel type, physical path set, rate limiting threshold, and path selection algorithm for each policy identifier value; the physical path set includes one or more global identifiers (WWID, World Wide Identifier) ​​of storage devices or the master / slave device number (major:minor) of physical paths.

[0042] An array of logical channel lists is extended within the kernel's private data area. This array contains three lists, indexed by the enumeration value corresponding to the path channel type: a dedicated channel list, an isolated channel list, and a general channel list. Each list stores pointers or identifiers of all physical paths belonging to that path channel type's physical path set. During system initialization or configuration reload, the user-space daemon multipathd parses the user-space configuration file, synchronizes the information in the user-space configuration file to the kernel's private data area, and adds the corresponding physical paths to the corresponding channel type's list based on the parsed information. For example, dedicated channels correspond to the high-performance physical path set, isolated channels correspond to the rate-limited or low-performance physical path set, and general channels correspond to the set of all available physical paths.

[0043] Once the path channel type of the IO request is determined (e.g., a dedicated channel), the corresponding head pointer of the linked list is obtained directly from the logical channel linked list array of the private data area of ​​the multipath device, using the enumerated value of the path channel type as an index, thereby obtaining the set of all physical paths bound to the channel type.

[0044] This pre-partitioning and linked list storage method achieves O(1) time complexity positioning of channel type to physical path set, avoiding complex condition judgments or traversal queries on critical IO paths. Simultaneously, placing the partitioning logic in the user-space configuration file facilitates flexible allocation of different performance levels of physical resources (e.g., dedicated high-speed SSDs, isolated low-speed HDDs) to different channels, thereby ensuring the performance of critical services at the hardware level and limiting abnormal traffic within isolated resources.

[0045] After determining the set of physical paths corresponding to the path channel type, in step S3, a specific physical path is selected from this set as the actual delivery channel for this IO request. To achieve load balancing while ensuring service priority, this embodiment employs a dynamic path selection algorithm that combines static priority weights and real-time load.

[0046] To ensure that the selected path is available and performs optimally, faulty physical paths in the physical path set are first removed.

[0047] It's important to note that each physical path in the kernel contains a status bit `is_active` and a static priority weight `priority`. The `is_active` status bit is maintained in real-time by the multipath device's kernel based on information such as the underlying link status and errors reported from the SCSI layer. When the physical path is functioning normally, `is_active` is 1 (active); when a path fails (e.g., link disconnection, device removal, response timeout), the kernel detection mechanism sets this status bit to 0 (inactive). The static priority weight `priority` is calculated automatically by the underlying storage driver or based on the path's hardware characteristics (e.g., bandwidth, latency, media type), or it can be set by user-space command-line tools (such as `multipathd`) through the kernel interface.

[0048] Specifically, the system first iterates through all paths in the physical path set, checking the `is_active` status bit of each path. If `is_active` is 0, the path is removed from the candidate set and does not participate in subsequent path selection. By using this status mask to eliminate faulty physical paths, it ensures that I / O requests are never sent to unavailable paths, thereby avoiding I / O errors or long timeouts caused by path failures and improving system reliability and I / O response speed.

[0049] Furthermore, for each normal physical path in the candidate set (is_active = 1), two key parameters are obtained: the static priority weight (priority) and the number of real-time I / O requests in transit (in_flight). The static priority weight (priority) is an inherent attribute of each physical path; the number of real-time I / O requests in transit (in_flight) represents the number of I / O requests currently being processed on that physical path but not yet completed. Each physical path's request queue maintains a counter, which is updated in real-time by the kernel when I / O requests are issued and completed. This directly reflects the current load of the path; a higher value indicates that the path is busier, and newly issued I / O requests may need to queue.

[0050] Based on the two parameters mentioned above, a comprehensive score is calculated for each active physical path. The comprehensive score = static priority weight / (1 + number of real-time IO requests in transit). Paths with higher static priority weights have higher comprehensive scores; while the more real-time IO requests in transit, the busier the path is, and the greater the attenuation effect on the comprehensive score.

[0051] Choose the path with the highest score as the target path. If multiple paths have the same score, any one of them can be chosen (e.g., take the first path in the linked list order).

[0052] Preferably, different processing is performed on different path channel types, which not only ensures dedicated resources for critical services, but also achieves efficient utilization of overall storage bandwidth.

[0053] Specifically, it includes: ① When the path channel type corresponding to the IO request is a dedicated channel, the optimal path is selected directly from the dedicated path set by calculating the comprehensive score. Since the dedicated path set usually contains multiple high-performance paths (such as multiple SSD paths), it is convenient to achieve load-aware balanced distribution among these paths, avoiding overload of one dedicated path while other dedicated paths are idle.

[0054] ② When the path channel type corresponding to the IO request is an isolated channel, a physical path is also selected from the physical path set as the target path by calculating the comprehensive score. However, a rate limiting judgment is also included before this to prevent low-priority services or abnormal traffic from consuming too many resources on the isolated channel. Even if they are guided to the isolated channel, their resource consumption is strictly constrained, thereby ensuring that the bandwidth of the isolated channel itself is reasonably allocated.

[0055] Rate limiting determination includes: obtaining the real-time IO throughput of the Cgroup to which the IO request belongs; comparing the real-time IO throughput with the preset rate limiting threshold; if the real-time IO throughput exceeds the rate limiting threshold, suspending the IO request until the real-time IO throughput drops below the rate limiting threshold.

[0056] It should be noted that IO throughput refers to the total amount of data read and written per second by all processes within a Cgroup (in bytes per second, i.e., bandwidth). The blkio subsystem (Block I / O Controller, a subsystem in the Linux kernel Cgroup framework used to control and manage block device I / O resources) in the Linux kernel maintains IO statistics for each Cgroup, including the cumulative number of bytes read and written. This embodiment calculates the current real-time IO throughput by reading these statistics and using a sliding window algorithm (e.g., the increment over the past second). In practice, the real-time IO throughput is obtained by directly accessing the statistical fields in the Cgroup kernel control block through kernel functions.

[0057] The preset rate limiting threshold is configured in the user-space configuration file and synchronized to the kernel's private data area, and is associated with the Cgroup's policy identifier. When rate limiting is determined, the corresponding rate limiting threshold is retrieved based on the policy identifier value of the Cgroup to which the current IO request belongs, and then the real-time IO throughput of that Cgroup is compared with the rate limiting threshold.

[0058] If the real-time I / O throughput is greater than or equal to the rate-limiting threshold, it means that the Cgroup has already fully utilized the isolation channel bandwidth allocated to it. In this case, the current I / O request is suspended (blocked) and not sent to the physical path. The suspension operation typically places the I / O request in a waiting queue and sets a timer or relies on a subsequent I / O completion event to wake it up. When the real-time I / O throughput of the Cgroup drops below the rate-limiting threshold (e.g., due to a rate decrease caused by the completion of other I / O requests), the suspended I / O request is then woken up, allowing it to continue path selection and delivery.

[0059] If the real-time IO throughput does not exceed the rate limit threshold, the comprehensive score of each normal physical path in the set of physical paths of the isolated channel is calculated, the normal physical path with the highest comprehensive score is selected as the target path and the IO request is issued.

[0060] ③ When the path channel type corresponding to the IO request is a general channel, the path selection algorithm can use either the method described above for calculating the comprehensive score, or the traditional round-robin or service-time minimization algorithm to select the target path, in order to maintain compatibility with the existing system. Simultaneously, when any Cgroup belonging to the general channel is detected to have abnormally high load, it is automatically switched to an isolated channel to avoid impacting other general channel services.

[0061] Specifically, the system monitors the real-time IO latency and utilization of the Cgroup to which the IO requests belong. When both the real-time IO latency and utilization of the Cgroup exceed the preset abnormal threshold, the path channel type corresponding to the Cgroup is switched from a general channel to an isolated channel. This monitoring mechanism facilitates the automatic identification of services that should belong to normal priority but suddenly experience abnormally high loads, and instantly downgrades them to isolated channels, thereby avoiding the "noisy neighbor" effect affecting other services sharing the general channel.

[0062] Real-time IO latency refers to the average time (in milliseconds) from the issuance to the completion of an IO request within the Cgroup. This metric reflects the current storage service performance experienced by the Cgroup; high latency usually indicates that the storage path or device is overloaded. IO utilization refers to the percentage of time (per unit of time) that the Cgroup actually uses the storage device. This metric reflects the intensity of the Cgroup's use of storage resources. For example, if a Cgroup consistently performs high IOPS or high throughput reads and writes, its utilization will approach 100%.

[0063] This embodiment obtains real-time IO latency and utilization by reading statistical data from the Cgroup kernel control block. IO latency anomaly thresholds and IO utilization anomaly thresholds can be defined in the user-space configuration file and used as global thresholds for general channels. When the real-time IO latency of a Cgroup exceeds the latency anomaly threshold and the real-time IO utilization exceeds the utilization anomaly threshold, it is determined that the Cgroup is currently generating abnormal traffic, and the path channel type of the Cgroup is dynamically switched from a general channel to an isolated channel.

[0064] It should be noted that dynamic switching is achieved by directly modifying the policy identifier value stored in the kernel control block of the Cgroup in kernel mode. After the switch, subsequent IO requests generated by the Cgroup are automatically routed to the physical path set corresponding to the isolation channel.

[0065] Specifically, based on the Cgroup identity information to which the current IO request belongs, the kernel control block of that Cgroup is located, and the policy identifier value within it is modified to the value corresponding to the isolated channel, thus switching the path channel type of that Cgroup from a general channel to an isolated channel. This modification is completed instantaneously, without user-space intervention, nor does it require restarting any processes or remounting the device.

[0066] After the changes take effect, all new IO requests generated by this Cgroup will be mapped to isolated channels based on the new policy identifier value when obtaining the corresponding path channel type. Ultimately, path selection will be performed from the physical path set of the isolated channels. IO requests already issued will remain unaffected and will continue to be completed along the original path.

[0067] Finally, the IO request is submitted to the request queue corresponding to the selected target path, and the underlying driver and hardware send the data to the storage device.

[0068] Compared with existing technologies, this embodiment provides a storage I / O multi-path routing method based on Cgroup tracing. It introduces upper-layer Cgroup service tags into the kernel multi-path layer, achieving a leap from "indiscriminate distribution" to "service-aware routing," and realizing precise I / O path routing based on service identity. Different selection strategies can be flexibly configured for different channel types, ensuring dedicated resources for critical services while achieving efficient utilization of overall storage bandwidth. Different strategies are defined through user-space configuration files and dynamically synchronized to the kernel's private data area via user-space daemons. This allows for flexible adjustment of the binding relationship between services and path resources without modifying kernel code, restarting the system, or rebuilding the multi-path device. Simultaneously, Cgroup service tags can be written at any time through the Cgroup virtual file system interface, supporting dynamic scheduling in containerized environments and perfectly adapting to the elastic requirements of cloud-native platforms.

[0069] Example 2 Another embodiment of the present invention discloses a storage I / O multipath routing system based on Cgroup tracing, thereby implementing the storage I / O multipath routing system method based on Cgroup tracing in Embodiment 1. The specific implementation of each module is described in the corresponding description in Embodiment 1. The system includes: The channel type determination module is used to obtain the Cgroup identity information to which the IO request belongs, and then determine the path channel type corresponding to the IO request; The path set matching module is used to determine the physical path set corresponding to the path channel type from multiple pre-divided physical path sets, where different path channel types correspond to different physical path sets; The path selection and distribution module is used to select a physical path from the set of physical paths as the target path and distribute the IO request to the storage device through the target path.

[0070] Preferably, the system in this embodiment further includes: The rate limiting judgment module is used to compare the real-time IO throughput of the Cgroup with the preset rate limiting threshold when the path channel type is an isolated channel. If the real-time IO throughput exceeds the rate limiting threshold, the IO request is suspended until the real-time IO throughput drops below the rate limiting threshold.

[0071] The channel switching module is used to monitor the IO latency and utilization of a Cgroup in real time when the channel type is a general channel. If both exceed the preset corresponding abnormal threshold, the policy flag value in the kernel control block of the Cgroup is modified to the value corresponding to the isolated channel.

[0072] It should be noted that the system in this embodiment, acting as a business-aware path selector, is implemented as a kernel module. This kernel module is dynamically loaded via the `insmod` command during Linux system runtime, without requiring a Linux operating system reboot or reconfiguration / rebuilding of existing multipath devices. When Linux loads the kernel module, it automatically registers the path selector of this embodiment with the kernel's DeviceMapper framework, enabling existing and newly created multipath devices to specify the use of this path selector through configuration files. Correspondingly, when it is necessary to uninstall the path selector of this embodiment (e.g., for software upgrades or feature removal), simply ensure it is not being used by any multipath device, then execute the `rmmod` command to complete the uninstallation; the kernel will automatically unregister the path selector of this embodiment. This hot-loading mechanism ensures that the system in this embodiment can be seamlessly integrated into existing production environments without affecting running services, greatly improving deployment flexibility and ease of maintenance.

[0073] Since the storage I / O multipath routing system based on Cgroup tracing in this embodiment and the aforementioned storage I / O multipath routing method based on Cgroup tracing can be mutually referenced, this is a repetition and will not be repeated here. Because this system embodiment shares the same principle as the above method embodiment, it also possesses the corresponding technical effects of the above method embodiment.

[0074] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0075] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A storage IO multi-path load distribution method based on Cgroup tracing, characterized in that, The method comprises the following steps: obtaining Cgroup identity information to which an IO request belongs, and determining a path channel type corresponding to the IO request; the path channel type comprises: an exclusive channel, an isolated channel and a general channel; determining a physical path set corresponding to the path channel type from a plurality of pre-divided physical path sets, wherein different path channel types correspond to different physical path sets; selecting a physical path from the physical path set as a target path, and issuing the IO request to a storage device through the target path.

2. The method of claim 1, wherein, The method comprises the following steps: obtaining a request object corresponding to the IO request, wherein the request object is encapsulated into a data structure containing complete business context in a kernel state; and obtaining Cgroup identity information to which a process initiating the IO request belongs according to a pointer chain in the data structure of the request object.

3. The method of claim 1, wherein, The method comprises the following steps: locating to a kernel control block of the Cgroup according to the Cgroup identity information, and reading a policy identification value from a policy identification field extended from the kernel control block; querying a mapping relationship between the policy identification value and the path channel type maintained in a kernel private data area, and determining the path channel type corresponding to the IO request according to the policy identification value.

4. The method of claim 3, wherein, The policy identification value is written into the policy identification field in the Cgroup kernel control block through the following steps: adding a virtual file corresponding to the policy identification field under each Cgroup directory of a Cgroup virtual file system; writing the policy identification value into the virtual file to trigger a kernel callback function to store the policy identification value into the policy identification field.

5. The method of claim 3, wherein, The mapping relationship between the policy identification value and the path channel type, and the plurality of pre-divided physical path sets are obtained by a user state daemon process analyzing a user state configuration file, and are stored in a kernel private data area.

6. The method of claim 5, wherein, The user state configuration file comprises: defining a corresponding path channel type, a physical path set, a speed limit threshold and a path selection algorithm for each policy identification value; and the physical path set comprises one or more global identifiers of storage devices or master and slave device numbers of physical paths.

7. The Cgroup-based provenance-aware storage IO multipath offloading method of claim 1, wherein, The method comprises the following steps: after eliminating a fault physical path in the physical path set, obtaining a static priority weight and a real-time in-transit IO request number of each normal physical path; calculating a comprehensive score of each normal physical path according to the static priority weight and the real-time in-transit IO request number; and selecting a normal physical path with the highest comprehensive score as the target path.

8. The Cgroup-based provenance-aware storage IO multipath offloading method of claim 1, wherein, When the path channel type corresponding to the IO request is the isolation channel, before selecting a physical path from the physical path set as the target path, the method further comprises: obtaining the real-time IO throughput of the Cgroup to which the IO request belongs; comparing the real-time IO throughput with a preset speed limit threshold; if the real-time IO throughput exceeds the speed limit threshold, suspending the IO request until the real-time IO throughput falls below the speed limit threshold.

9. The method of claim 3, wherein the Cgroup-based provenance-aware storage IO multi-path load splitting is performed by a storage controller. The method further comprises: when the path channel type corresponding to the IO request is the general channel, monitoring the real-time IO delay and usage of the Cgroup to which the IO request belongs in real time; when it is monitored that the real-time IO delay and usage of the Cgroup both exceed the preset corresponding abnormal threshold, modifying the policy identification value in the kernel control block of the Cgroup to the value corresponding to the isolation channel.

10. A storage IO multi-path load distribution system based on Cgroup tracing, characterized in that, The method comprises: a channel type determination module configured to obtain the Cgroup identity information to which the IO request belongs, and then determine the path channel type corresponding to the IO request; the path channel type comprises: the exclusive channel, the isolation channel and the general channel; a path set matching module configured to determine the physical path set corresponding to the path channel type from a plurality of pre-divided physical path sets, wherein different path channel types correspond to different physical path sets; a path selection and issuing module configured to select a physical path from the physical path set as the target path, and issue the IO request to the storage device through the target path.

Images