Data processing method and device of storage system, electronic equipment and storage medium

Abstract

The invention discloses a data processing method and device of a storage system, electronic equipment and a storage medium, and relates to the technical field of computers.According to the data processing method and device of the storage system, due to the fact that when a fault domain is changed, a communication suppression state is activated firstly to avoid disordered data interaction, and communication cost is accurately calculated in combination with a storage node topological relation; data updating pulling and scheduling reconstruction tasks are initiated according to needs, the limitation of undifferentiated PG broadcast is eliminated, the cross-cabinet communication flow during fault domain change of the distributed storage system is reduced, the communication efficiency of the data reconstruction tasks is optimized, network resource occupation is reduced, and the reliability of the distributed storage system is improved. And therefore, the technical effect of stable and efficient operation of the distributed storage system in scenes such as a data center and a hybrid cloud can be ensured.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to data processing methods, apparatus, electronic devices and storage media for storage systems. Background Technology

[0002] Distributed storage systems, as the core infrastructure for cloud computing and big data processing, are widely used in scenarios such as data centers, hybrid clouds, and multi-active architectures.

[0003] The existing distributed storage communication mechanism directly adopts the protocol layer design of PG broadcasting notify messages, without considering the difference in communication costs when the fault domain changes, which may lead to an exponential increase in cross-rack traffic. Summary of the Invention

[0004] This application provides a data processing method, apparatus, electronic device, and storage medium for a storage system, to at least address the problem in the related art where differences in communication costs during fault domain changes lead to an exponential increase in cross-rack traffic.

[0005] This application provides a data processing method for a storage system, including: Acquire fault domain change events in the distributed storage system and activate the communication suppression state; Under the communication suppression state, monitor the data version difference information of each storage node; In response to the data version difference information meeting preset conditions, a data update pull request is initiated to the management node; The communication cost is calculated based on the topology of the storage nodes, and the data reconstruction task is scheduled based on the communication cost.

[0006] Optionally, obtaining the fault domain change event of the distributed storage system and activating the communication suppression state includes: Analyze the system storage topology rules to determine the type and scope of the fault domain; When the type of the fault domain is a target type, a communication suppression command is generated; According to the communication suppression instruction, stop broadcasting update notifications to the associated storage nodes.

[0007] Optionally, under the communication suppression state, monitoring the data version difference information of each storage node includes: Based on a preset period, the local data version number of the storage node and the target version number maintained by the management node are obtained, and a historical record of version differences for different preset periods is generated. A dynamic threshold for triggering the data update fetch request is determined based on the current network status and the historical records.

[0008] Optionally, the step of calculating the communication cost based on the topology of the storage nodes and scheduling the data reconstruction task based on the communication cost includes: Obtain the topology information of the physical location and network connection relationship of the storage nodes; Based on at least one of the factors in the topology information, namely distance, available bandwidth, and network path complexity, the data transmission cost between two storage nodes or a group of storage nodes is quantified, and a cost matrix is ​​generated.

[0009] Optionally, the step of calculating the communication cost based on the topology of the storage nodes and scheduling the data reconstruction task based on the communication cost includes: Prioritize scheduling data reconstruction tasks with communication costs below a first threshold; For batch data reconstruction tasks with communication costs exceeding the second threshold, batch scheduling is performed.

[0010] Optionally, the method further includes: Monitor network traffic across predetermined logical packets; When the network traffic exceeds the congestion threshold, traffic shaping control is initiated; wherein, the traffic shaping control includes adjusting transmission protocol parameters and introducing transmission delay.

[0011] This application also provides a data processing apparatus for a storage system, comprising: The acquisition unit is used to acquire fault domain change events of the distributed storage system and activate the communication suppression state. The monitoring unit is used to monitor the data version difference information of each storage node under the communication suppression state; The update unit is used to initiate a data update retrieval request to the management node in response to the data version difference information meeting preset conditions; The computing unit is used to calculate the communication cost based on the topology of the storage nodes and schedule data reconstruction tasks based on the communication cost.

[0012] Optionally, the acquisition unit is further configured to: Analyze the system storage topology rules to determine the type and scope of the fault domain; When the type of the fault domain is a target type, a communication suppression command is generated; According to the communication suppression instruction, stop broadcasting update notifications to the associated storage nodes.

[0013] Optionally, the monitoring unit is further configured to: Based on a preset period, the local data version number of the storage node and the target version number maintained by the management node are obtained, and a historical record of version differences for different preset periods is generated. A dynamic threshold for triggering the data update fetch request is determined based on the current network status and the historical records.

[0014] Optionally, the computing unit is further configured to: Obtain the topology information of the physical location and network connection relationship of the storage nodes; Based on at least one of the factors in the topology information, namely distance, available bandwidth, and network path complexity, the data transmission cost between two storage nodes or a group of storage nodes is quantified, and a cost matrix is ​​generated.

[0015] Optionally, the computing unit is further configured to: Prioritize scheduling data reconstruction tasks with communication costs below a first threshold; For batch data reconstruction tasks with communication costs exceeding the second threshold, batch scheduling is performed.

[0016] Optionally, the device further includes: The monitoring unit is used to monitor network traffic across predetermined logical packets; A control unit is configured to initiate traffic shaping control when the network traffic exceeds a congestion threshold; wherein the traffic shaping control includes adjusting transmission protocol parameters and introducing transmission delay.

[0017] This application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the data processing method of any of the above-described storage systems.

[0018] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the data processing method of any of the above-described storage systems.

[0019] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data processing method of any of the above-described storage systems.

[0020] This application addresses the limitations of indiscriminate PG broadcasting by activating a communication suppression state to prevent disordered data interaction during fault domain changes. It also enables precise calculation of communication costs based on storage node topology, and the on-demand initiation of data update retrieval and scheduling of reconstruction tasks. This overcomes the constraints of indiscriminate PG broadcasting. Therefore, it solves the technical problem of exponentially increasing cross-rack traffic caused by the use of PG broadcast notification message protocol layer design and failure to consider communication cost differences during fault domain changes in existing distributed storage communication mechanisms. This reduces cross-rack communication traffic during fault domain changes in distributed storage systems, optimizes communication efficiency for data reconstruction tasks, reduces network resource consumption, and ultimately ensures stable and efficient operation of distributed storage systems in data centers, hybrid clouds, and other scenarios. Attached Figure Description

[0021] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A schematic flowchart illustrating a data processing method for a storage system provided in an embodiment of this application; Figure 2 This is a schematic diagram of an overall process provided for an embodiment of this application; Figure 3 A schematic diagram of a workflow provided for an embodiment of this application; Figure 4 This is a schematic diagram of a hierarchical control architecture provided in an embodiment of this application; Figure 5 This is a schematic diagram of a three-level scheduling system architecture provided in an embodiment of this application; Figure 6 A schematic diagram of the structure of a data processing device for a storage system provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of a data processing device for another storage system provided in an embodiment of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0024] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0025] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] The embodiments of this application provide a data processing method for a storage system. The method is described in detail below in conjunction with the execution flow of the data processing method for the storage system. Figure 1 This is a schematic flowchart illustrating a data processing method for a storage system provided in an embodiment of this application.

[0027] like Figure 1 As shown, the method includes the following steps: Step 101: Obtain the fault domain change event of the distributed storage system and activate the communication suppression state; In distributed storage systems, accurate capture of fault domain changes and activation of communication suppression mechanisms ensure network stability during topology changes. In distributed storage systems, a fault domain refers to a logical group of devices that may experience synchronous failures, commonly including host racks and server rooms. Changes to these domains directly impact the data storage topology and communication patterns. The system uses a built-in monitoring module to monitor topology changes in the cluster in real time, focusing on capturing Crush Map change events. The Crush Map, a core component of the distributed storage data distribution control algorithm, records the topological relationships of all devices in the cluster. Upon capturing a change event, the monitoring module identifies the change type, confirming whether it is a preset target change type, such as a rack-level change. If determined to be a valid fault domain change event, a domain change message containing the new topology fingerprint is broadcast to the entire distributed storage cluster, ensuring that all relevant components within the cluster are synchronously aware of the topology change.

[0028] The activation of the communication suppression state is performed on Placement Groups (PGs). As logical units for data shard management, PGs in traditional architectures actively broadcast notify messages when the topology changes. These notify messages are data update notifications proactively sent by PGs to the Object Storage Devices (OSDs). Excessive broadcasting can trigger network storms. Upon receiving a domain change message, all PGs immediately set a communication suppression flag, modify their message processing callback functions, and skip the notification message sending logic. This prevents unnecessary broadcast communication at the source, avoids a surge of invalid messages at the protocol layer, and lays a stable network foundation for subsequent data synchronization and reconstruction operations.

[0029] Step 102: Under the communication suppression state, monitor the data version difference information of each storage node; Accurately monitoring the data synchronization status of each storage node under communication suppression provides a reliable basis for subsequent on-demand data retrieval. Storage nodes, or Object Storage Devices (OSDs), are the core units responsible for data storage and read / write operations in a distributed storage system. The consistency of their data versions directly impacts the overall system availability. Under communication suppression, placement groups (PGs) stop broadcasting notify messages, making version information synchronization impossible via traditional push mechanisms. Therefore, each OSD must proactively initiate a version monitoring mechanism. Each OSD runs its own dedicated monitoring thread, continuously tracking version changes in its stored data and maintaining a sliding window to record version difference history. This window continuously collects version data from multiple sampling points to ensure comprehensive capture of version change trends.

[0030] During monitoring, the OSD acquires the current data version number in real time and compares it with the previously recorded version number to calculate the version increment. This update then updates the version difference records within the sliding window. Simultaneously, it calculates key metrics such as the maximum and average version difference within the window, providing data support for subsequent difference determination. To ensure the accuracy of monitoring data in multi-threaded concurrent scenarios, the OSD protects shared data structures with thread-safe locks to prevent version record corruption.

[0031] This monitoring mechanism can capture the version differences of each storage node in real time and continuously, effectively making up for the deficiency that version information cannot be actively pushed under the communication suppression state, ensuring that the system can promptly perceive the data synchronization needs, laying a solid foundation for subsequent targeted pull operations based on differences, avoiding invalid communication and ensuring data consistency.

[0032] Step 103: In response to the data version difference information meeting the preset conditions, a data update pull request is initiated to the management node; Storage nodes, or Object Storage Devices (OSDs), are the core units responsible for data storage and read / write operations in a distributed storage system. The consistency of their data versions directly impacts the overall system availability. In communication-suppressed mode, placement groups (PGs) stop broadcasting notify messages, making version information synchronization impossible via traditional push mechanisms. Therefore, each OSD must proactively initiate a version monitoring mechanism. Each OSD independently runs its own dedicated monitoring thread to continuously track version changes in its stored data. Simultaneously, it maintains a sliding window to record historical version differences. This window continuously collects version data from multiple sampling points to ensure comprehensive capture of version change trends.

[0033] During monitoring, the OSD acquires the current data version number in real time and compares it with the previously recorded version number to calculate the version increment. This update then updates the version difference records within the sliding window. Simultaneously, it calculates key metrics such as the maximum and average version difference within the window, providing data support for subsequent difference determination. To ensure the accuracy of monitoring data in multi-threaded concurrent scenarios, the OSD protects shared data structures with thread-safe locks to prevent version record corruption. This monitoring mechanism can capture the dynamic version differences of each storage node in real time and continuously, effectively compensating for the inability to proactively push version information under communication suppression conditions. It ensures the system promptly detects data synchronization needs, laying a solid foundation for subsequent difference-based targeted fetch operations, thus avoiding invalid communication and guaranteeing data consistency.

[0034] Step 104: Calculate the communication cost based on the topology of the storage nodes, and schedule the data reconstruction task based on the communication cost.

[0035] By optimizing communication resource allocation through the topology of storage nodes, efficient scheduling of data reconstruction tasks is achieved, ensuring the performance stability of the distributed storage system after fault domain changes. The topology of storage nodes encompasses key information such as the rack and room location distribution of each object storage device (OSD) in the cluster, the interconnection bandwidth between devices, and the number of switch hops. This information is the basis for accurately calculating communication costs. The system obtains the physical location coordinates of the racks by parsing the CRUSH Map, collects the interconnection bandwidth between racks in real time through the LLDP protocol, and obtains the number of switch hops required for data transmission through the OSPF protocol, thus comprehensively constructing a topology information database.

[0036] The calculation of communication costs aims to quantify the cost of data transmission between storage nodes, comprehensively considering multiple dimensions such as distance, bandwidth, hop count, packet loss rate, and latency. A dynamically generated cost matrix enables accurate assessment of the transmission cost between any two nodes or racks. This cost matrix is ​​updated periodically or triggered by events based on network status changes, ensuring the real-time nature and accuracy of communication cost calculations. Based on the calculated communication costs, the system initiates a data reconstruction task scheduling mechanism, prioritizing the source node or rack with the lowest communication cost to initiate data transmission. Reconstruction tasks within the same rack are given the highest priority and achieve localized transmission via dedicated links, avoiding core switch congestion. Cross-rack reconstruction tasks employ a batch scheduling strategy, controlling the number of OSDs involved in each batch to avoid network pressure caused by concentrated transmission.

[0037] This communication cost-based scheduling method can maximize the use of network resources, reduce invalid transmissions and redundant overhead, and effectively control the cross-rack traffic ratio while ensuring rapid data synchronization, avoiding network congestion and service interruption, and achieving a balance between data reconstruction efficiency and system stability.

[0038] In some embodiments, obtaining the fault domain change event of the distributed storage system and activating the communication suppression state includes: Analyze the system storage topology rules to determine the type and scope of the fault domain; When the type of the fault domain is a target type, a communication suppression command is generated; According to the communication suppression instruction, stop broadcasting update notifications to the associated storage nodes.

[0039] This is used to precisely trigger communication suppression states, ensuring that invalid broadcasts are blocked only in specific fault domain change scenarios. The storage topology rules of the distributed storage system are defined by the CRUSH algorithm, which includes the logical grouping criteria and component relationships of fault domains. Parsing this rule requires extracting the type identifier and scope boundary of the fault domain, clarifying whether the fault domain is a logical grouping at the host rack or data center level, and the set of object storage devices covered by the fault domain.

[0040] The target type is a pre-defined fault domain type that requires communication suppression, typically a rack-level fault domain type that easily triggers a large amount of cross-domain communication. The system compares the parsed fault domain type with the target type to determine if the suppression triggering conditions are met. When the result is a match, the system generates a communication suppression command containing key information such as the topology change fingerprint fault domain range, ensuring that the relocation group can accurately identify the suppression requirement.

[0041] The associated storage node refers to the object storage device that has a data association with the fault domain change. The update notification is the notify message that the relocation group actively sends to the object storage device. After receiving the communication suppression instruction, the relocation group immediately adjusts the message processing flow and stops broadcasting the notify message to the associated object storage device. This blocks invalid communication at the protocol layer from the source, ensuring network stability in specific fault domain change scenarios without affecting communication synchronization during other normal topology changes, thus achieving precision and targeting of suppression triggering.

[0042] In some embodiments, monitoring the data version difference information of each storage node under the communication suppression state includes: Based on a preset period, the local data version number of the storage node and the target version number maintained by the management node are obtained, and a historical record of version differences for different preset periods is generated. A dynamic threshold for triggering the data update fetch request is determined based on the current network status and the historical records.

[0043] This feature is used to accurately collect version difference data of storage nodes and dynamically adjust triggering conditions to ensure the rationality and timeliness of data update pull requests.

[0044] The preset period is the version information collection interval pre-configured by the system, which can balance the real-time monitoring and resource overhead. The storage node will continuously obtain two types of key version information according to this period. One type is the local data version number of its own stored data, which directly reflects the current data update status of the node. The other type is the target version number maintained by the management node. This version number is maintained by the management node based on the cluster data consistency requirements and represents the standard version that the node needs to synchronize to.

[0045] After each data collection, the storage node calculates the difference between the local data version number and the target version number, generating single-time version difference data. This data is then stored chronologically in a preset data structure, forming a historical record of version differences for different preset periods. This historical record comprehensively reflects the changing trend of node version differences, providing data support for subsequent threshold determination. During storage, key indicators such as the maximum version difference and average version difference in the historical record are simultaneously calculated to facilitate rapid assessment of version difference status. The determination of dynamic thresholds requires a comprehensive consideration of the current network status and the historical version difference record. The current network status mainly refers to core indicators such as cross-rack bandwidth utilization to determine whether the network is congested. The historical version difference record provides historical evidence for threshold setting, avoiding misjudgments caused by relying solely on single-time version differences.

[0046] The system first determines the basic threshold range based on historical records, and then dynamically adjusts it according to the current network status. When the network is congested, the threshold is appropriately increased to reduce the frequency of fetch requests and alleviate network pressure. When the network status is good, the threshold is reasonably reduced to speed up data synchronization. This ensures that the dynamic threshold can adapt to different network scenarios and version change trends, avoiding excessive requests or synchronization delays caused by fixed thresholds, while accurately matching data update needs and providing a scientific triggering basis for subsequent on-demand fetching.

[0047] In some embodiments, calculating the communication cost based on the topology of the storage nodes and scheduling the data reconstruction task based on the communication cost includes: Obtain the topology information of the physical location and network connection relationship of the storage nodes; Based on at least one of the factors in the topology information, namely distance, available bandwidth, and network path complexity, the data transmission cost between two storage nodes or a group of storage nodes is quantified, and a cost matrix is ​​generated.

[0048] This feature is used to accurately quantify the data transmission cost between storage nodes, providing a scientific basis for data reconstruction task scheduling. Topology information is the core foundation for calculating communication costs, encompassing two key dimensions: the physical location of storage nodes and their network connectivity. Physical location specifically includes a list of the racks within the data center to which the storage nodes reside, clearly defining the spatial distribution of nodes within the cluster. Network connectivity includes information such as interconnect bandwidth, link status, and switch hop count, reflecting the fundamental network conditions for data transmission. Obtaining this topology information requires a multi-pronged approach. Parsing the storage topology rules defined by the CRUSH algorithm extracts the physical location associations of nodes. Real-time acquisition of interconnect bandwidth data between racks or nodes is achieved through the LLDP protocol, and the number of switch hops required for data transmission is obtained using the OSPF protocol, comprehensively ensuring the integrity and accuracy of the topology information.

[0049] The quantification of communication costs must be based on key factors in the topology information. At least one of distance, available bandwidth, and network path complexity should be selected for calculation. The distance factor quantifies the spatial cost between nodes using the Manhattan distance model. The distance cost between nodes in different data centers will increase proportionally. Available bandwidth is converted into a transmission latency baseline value through normalization. The higher the bandwidth, the lower the transmission cost. Network path complexity is comprehensively reflected by indicators such as switch hop count, packet loss rate, and latency. The simpler the path and the fewer the hops, the lower the transmission cost. In the specific quantification process, an appropriate combination of factors is selected according to the actual needs of the system. Each factor is converted into a calculable value through a preset mathematical formula, and then the transmission cost between two storage nodes or a group of storage nodes is obtained. The transmission costs between all nodes or groups of nodes are stored in a matrix format to form a cost matrix.

[0050] This cost matrix clearly shows the communication cost difference between any two storage nodes or node groups within the cluster, providing intuitive and accurate data support for subsequent cost-based task scheduling. It ensures that data reconstruction tasks can prioritize the path and node with the lowest communication cost for transmission, effectively reducing network overhead and improving reconstruction efficiency.

[0051] In some embodiments, calculating the communication cost based on the topology of the storage nodes and scheduling the data reconstruction task based on the communication cost includes: Prioritize scheduling data reconstruction tasks with communication costs below a first threshold; For batch data reconstruction tasks with communication costs exceeding the second threshold, batch scheduling is performed.

[0052] This feature optimizes the execution order of data reconstruction tasks through differentiated scheduling strategies, balancing network load and reconstruction efficiency to ensure the stable operation of the distributed storage system after fault domain changes. Prioritizing data reconstruction tasks with communication costs below a first threshold is the core scheduling logic. This first threshold is a pre-set communication cost threshold based on the system's network capacity and reconstruction efficiency goals. Tasks with communication costs below this threshold typically correspond to scenarios where storage nodes are close together, bandwidth is sufficient, and network paths are simple. These tasks have low data transmission costs and low resource consumption; prioritizing their execution can quickly advance the overall reconstruction progress while preventing low-cost tasks from being delayed by waiting for high-cost tasks.

[0053] After obtaining the communication cost of each data reconstruction task, the system compares it with the first threshold and includes tasks below the threshold in the high-priority scheduling queue, giving priority to the allocation of network and computing resources. Among them, reconstruction tasks within the same rack are usually given priority due to their extremely low communication cost and are transmitted efficiently through dedicated links.

[0054] Batch scheduling is performed on batch data reconstruction tasks with communication costs exceeding the second threshold to avoid network congestion caused by the concentrated execution of a large number of high-cost tasks. The second threshold, higher than the first threshold, is used to define the scope of high-communication-cost tasks. These tasks are often batch tasks that cross racks, travel long distances, or have complex network conditions. Centralized scheduling would consume a large amount of cross-rack bandwidth, leading to a surge in network pressure. During batch scheduling, the system breaks down batch tasks into several smaller batches based on the current network status and resource availability. The number of storage nodes involved in each batch is strictly controlled within a preset range. Each batch of tasks is started sequentially to ensure that the network occupancy rate during the execution of each batch does not exceed the safety threshold.

[0055] This differentiated scheduling approach ensures the rapid progress of low-cost tasks while controlling the network consumption of high-cost tasks through batch scheduling. It achieves a dynamic balance between data reconstruction efficiency and network stability, effectively avoiding network storms or reconstruction delays caused by disordered task execution in traditional scheduling. This ensures that the system can smoothly and efficiently complete data reconstruction during topology changes.

[0056] In some embodiments, the method further includes: Monitor network traffic across predetermined logical packets; When the network traffic exceeds the congestion threshold, traffic shaping control is initiated; wherein, the traffic shaping control includes adjusting transmission protocol parameters and introducing transmission delay.

[0057] This feature is used to avoid network congestion across predetermined logical groups through traffic shaping control, ensuring network stability and business continuity of distributed storage systems during data reconstruction. A predetermined logical group refers to a set of devices divided based on fault domains, commonly in the form of server racks. Network traffic across this group is the data transmission traffic between different server racks. This type of traffic is prone to surges during reconstruction after a fault domain change, and is a key factor triggering network storms.

[0058] The system collects network traffic data across predetermined logical groups in real time through a dedicated monitoring layer, continuously calculating the proportion of traffic to total bandwidth to provide accurate data for congestion assessment. The congestion threshold is a preset critical value based on the system's network carrying capacity. This threshold is set in conjunction with historical reconstructed traffic data and service quality requirements to ensure that control measures are triggered promptly when traffic approaches the congestion threshold.

[0059] When network traffic across predetermined logical packets exceeds the congestion threshold, traffic shaping control is immediately initiated. This control mechanism comprises two core operations: first, adjusting transmission protocol parameters and using RoCEv2's DCQCN protocol for dynamic rate limiting. This reduces the amount of data transmitted per unit time by decreasing the data transmission window size, thus preventing link overload; second, introducing transmission delays. A preset delay is inserted during data transmission to smoothly distribute concentrated transmission requests and alleviate instantaneous traffic pressure. The implementation of traffic shaping control follows a precise adaptation principle. The adjusted parameters and the introduced delay duration can be dynamically optimized based on real-time network traffic changes. This effectively suppresses disorderly growth of cross-packet traffic, preventing network congestion and service interruptions, while avoiding excessive restriction on normal data synchronization speed. It achieves a balance between network load and reconstruction efficiency, ensuring stable system operation during topology changes.

[0060] Please see Figure 2 , Figure 2 This application provides an overall process diagram with detailed steps as described in the embodiments. 1. Fault Domain Change Detection The monitoring module captures crush map change events (change type = rack level); Broadcast a domain_change_event message (carrying the new topology fingerprint) to the cluster.

[0061] 2. PG communication inhibition activation After all PGs receive an event, set the notify_suppress=TRUE flag; Modify the message processing callback function to skip the notify sending logic.

[0062] 3. OSD version monitoring Each OSD starts an independent monitoring thread with a period of 200ms; Maintain a sliding window to record version difference history (window size = 10 sampling points).

[0063] 4. Difference Threshold Determination Dynamically calculate the threshold: Δ_threshold = K×(1+α×congestion_level) in: K = Base threshold (default 100 version difference); α = congestion sensitivity coefficient (adjustable from 0.2 to 0.8); congestion_level = Current cross-rack bandwidth utilization.

[0064] 5. Initiation of a targeted request Select the source rack with the lowest cost: Total cost = (distance factor × 0.1) + (100 / bandwidth) + (hop count × 5) + (packet loss rate × 10 + latency × 0.2); Send a PULL_REQUEST message (carrying the required version range).

[0065] 6. Flow shaping control Dynamic rate limiting using the DCQCN protocol of RoCEv2; When cross-rack traffic exceeds 40% of total bandwidth, traffic shaping is initiated. Reduce the sending window size (win_size -= 50%); Insert a 50ms delay.

[0066] 7. Accelerated Localization Restructuring Prioritize scheduling data reconstruction tasks within the same server rack; Data transmission within the rack is achieved via a dedicated 25Gbps backplane link, avoiding congestion of the core switch. Cross-rack tasks adopt a batch scheduling strategy (each batch ≤ 10 OSDs).

[0067] Key Algorithm 1. Version Difference Dynamic Threshold Algorithm The core of this algorithm lies in dynamically adapting to network conditions and reconstruction progress, avoiding the "premature / late triggering" problem caused by a fixed threshold. The specific implementation consists of three layers: a. Basic threshold calculation layer The initial threshold is set to a fixed value K (default K=100 version difference). This value is based on historical statistics: when the version difference is less than 100, the impact of data latency on business is negligible.

[0068] b. Network congestion adaptive layer A congestion sensitivity coefficient α (range 0.2~0.8, default 0.5) is introduced to collect the cross-rack link bandwidth utilization η (η = current traffic / total bandwidth) in real time. When η > 0.4, it is determined to be a congested state, and the threshold is increased to reduce the request frequency. congestion_factor = 1 + α * min(1.0, max(0, (η - 0.4) / 0.6)); dynamic_threshold = base_threshold * congestion_factor; When η=1.0 (complete congestion), congestion_factor=1+α, and the threshold is increased by a maximum of 80%.

[0069] c. Reconstruct the schedule compensation layer In the later stages of reconstruction (progress > 70%), to avoid delayed recovery of remaining fragmented data, a progress compensation factor β is introduced: if rebuild_progress>0.7: progress_factor = 1.5 - (rebuild_progress - 0.7)*1.5 # Linearly decreases from 1.5 to 1.0 dynamic_threshold *= progress_factor Algorithm advantages: Automatically reduce the request frequency when congestion occurs to alleviate network pressure; Restrictions were relaxed towards the end of the reconstruction phase to accelerate the recovery of remaining data; The parameter K / α can be dynamically adjusted through the management interface.

[0070] 2. Rack Cost Matrix Generation Algorithm The cost matrix is ​​used to quantify the data transmission cost between any two racks, enabling intelligent scheduling based on the physical topology. Matrix construction steps: 1. Topology Information Extraction Parsing the CRUSH Map to obtain: Physical location coordinates of the server rack (data center - row - column); Inter-rack interconnection bandwidth (collected in real time via LLDP protocol).

[0071] 2. Distance Cost Calculation Using the Manhattan distance model (suitable for data center cabling): distance = |x1-x2| + |y1-y2| + 10*|z1-z2| # z is the data center number, the cost increases exponentially with different data centers. 3. Bandwidth normalization Mapping bandwidth to a transmission delay baseline: bandwidth_delay = REF_BW / min(bw1, bw2) # REF_BW=100Gbps (baseline bandwidth).

[0072] 4. Comprehensive Cost Formula cost = distance * bandwidth_delay * hop_count - `hop_count`: Number of hops through switches (obtained via OSPF protocol).

[0073] Matrix update mechanism: Scheduled updates: Full recalculation every 5 minutes; Event-triggered update: Updates immediately when a link failure or bandwidth change >20% is detected; Incremental update: Only the rows and columns related to the cabinet are changed and recalculated.

[0074] Sliding window version monitor a. Data Structure Design Circular buffer: Contains 10 elements recording historical version differences; Timestamp array: Records the precise time (millisecond precision) of each sample; Statistical indicators: Maximum version difference and average version difference in the current window; Status records: the version number of the last record, and the pointer to the current position in the buffer; Thread-safe locks: ensure thread safety during concurrency.

[0075] Please see Figure 3 , Figure 3This is a schematic diagram of a workflow provided in an embodiment of this application, such as... Figure 3 As shown, it includes: The timer triggers a monitoring task every 200 milliseconds; Get the current data version number and calculate the version increment compared to the last record; Locking mutexes protects shared data structures; Write the new version difference to the current position in the circular buffer; Update the buffer pointer (circular queue mechanism); Recalculate the maximum and average version difference within the window; Update the last recorded version number and release the lock; Asynchronous decision: If the maximum version difference exceeds the dynamic threshold, a fetch request is triggered.

[0076] Please see Figure 4 , Figure 4 This is a schematic diagram of a hierarchical control architecture provided in an embodiment of this application, such as... Figure 4 As shown, it includes: Monitoring layer: Real-time collection of cross-rack bandwidth utilization; Decision-making level: Three-state machine transition logic; Network status: The trigger condition is a control action; Normal state: Bandwidth <40% maintains standard transmission parameters; Congestion status: 40% ≤ bandwidth < 70%, window size halved, latency increased by 20ms, mild congestion control initiated; Emergency state: Bandwidth ≥ 70%, window size reduced to 8, latency increased by 50ms, aggressive congestion control initiated.

[0077] Execution layer: Dynamically adjusts TCP window size and transmission delay.

[0078] Core state machine: c. Packet loss retransmission strategy: Smart retransmission trigger: Three consecutive missing ACKs or a hole was detected in the SACK; The receiving end actively sends a NACK (negative acknowledgment); Tiered retransmission mechanism: Packet loss rate: <5%; Strategy: Standard retransmission; Retransmission interval: 200ms; Packet loss rate: 5%-15%; Strategy: Fast retransmission + forward error correction; Retransmission interval: 50ms; Packet loss rate: >15%; Strategy: Batch retransmission + path switching; Retransmission interval: Immediate.

[0079] Memory backup mechanism: Data packets are sent and backed up simultaneously in RDMA memory; The receiver can directly read the lost data from the backup memory.

[0080] Please see Figure 5 , Figure 5 This is a schematic diagram of a three-level scheduling system architecture provided in an embodiment of this application, as shown below. Figure 5 As shown, it includes: Task classifier: Rack-in tasks: highest priority; Cross-rack task: Medium priority; Metadata synchronization: Low priority in the backend.

[0081] b. Dedicated processor Localization acceleration engine: processes tasks within the server rack; Intelligent batch processor: handles batch tasks across racks; Idle resource scheduler: handles metadata synchronization.

[0082] c. Transmission optimization technology RDMA zero-copy transfer: Memory registration: Creates a protected memory region; Data transfer: bypassing the operating system to directly read and write remote memory.

[0083] Pipeline Restructuring: Three-stage parallel processing: data reading → verification calculation → data writing; Pipeline operation: When verifying the Nth block, the N+1th block is read simultaneously.

[0084] Details of rack-mounted acceleration technology: a) RDMA zero-copy channel Memory registration: Direct memory connection; Zero-copy data transfer is achieved between OSDs in the same rack via RDMA memory registration; Bypassing the kernel protocol stack reduces latency to below 5μs.

[0085] b. Pipeline reconfiguration technology The reconstruction process is broken down into three stages: data reading → verification → writing. The three-stage parallel execution resulted in a measured throughput increase of up to 3 times.

[0086] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0087] Embodiments of this application also provide a data processing apparatus for a storage system, such as... Figure 6 As shown, it includes: Acquisition unit 21 is used to acquire fault domain change events of the distributed storage system and activate the communication suppression state; Monitoring unit 22 is used to monitor the data version difference information of each storage node under the communication suppression state; Update unit 23 is used to initiate a data update pull request to the management node in response to the data version difference information meeting preset conditions; The computing unit 24 is used to calculate the communication cost based on the topology of the storage nodes and schedule the data reconstruction task based on the communication cost.

[0088] Furthermore, in one possible implementation of this application embodiment, the acquisition unit 21 is further configured to: Analyze the system storage topology rules to determine the type and scope of the fault domain; When the type of the fault domain is a target type, a communication suppression command is generated; According to the communication suppression instruction, stop broadcasting update notifications to the associated storage nodes.

[0089] Furthermore, in one possible implementation of this application embodiment, the monitoring unit 22 is further configured to: Based on a preset period, the local data version number of the storage node and the target version number maintained by the management node are obtained, and a historical record of version differences for different preset periods is generated. A dynamic threshold for triggering the data update fetch request is determined based on the current network status and the historical records.

[0090] Furthermore, in one possible implementation of this application embodiment, the computing unit 24 is further configured to: Obtain the topology information of the physical location and network connection relationship of the storage nodes; Based on at least one of the factors in the topology information, namely distance, available bandwidth, and network path complexity, the data transmission cost between two storage nodes or a group of storage nodes is quantified, and a cost matrix is ​​generated.

[0091] Furthermore, in one possible implementation of this application embodiment, the computing unit 24 is further configured to: Prioritize scheduling data reconstruction tasks with communication costs below a first threshold; For batch data reconstruction tasks with communication costs exceeding the second threshold, batch scheduling is performed.

[0092] Furthermore, in one possible implementation of the embodiments of this application, such as Figure 7 As shown, the device further includes: Monitoring unit 25 is used to monitor network traffic across predetermined logical packets; Control unit 26 is used to initiate traffic shaping control when the network traffic exceeds the congestion threshold; wherein, the traffic shaping control includes adjusting transmission protocol parameters and introducing transmission delay.

[0093] For a description of the features in the embodiment corresponding to the data processing device of the storage system, please refer to the relevant description of the embodiment corresponding to the data processing method of the storage system, which will not be repeated here.

[0094] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the data processing method embodiments of the above-described storage system.

[0095] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described data processing method embodiments of the storage system when it is run.

[0096] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0097] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described data processing method embodiments of a storage system.

[0098] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described data processing method embodiments of the storage system.

[0099] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0100] The data processing method, apparatus, electronic device, and storage medium of the storage system provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A data processing method for a storage system, characterized in that, include: Acquire fault domain change events in the distributed storage system and activate the communication suppression state; Under the communication suppression state, monitor the data version difference information of each storage node; In response to the data version difference information meeting preset conditions, a data update pull request is initiated to the management node; The communication cost is calculated based on the topology of the storage nodes, and the data reconstruction task is scheduled based on the communication cost.

2. The method according to claim 1, characterized in that, The step of acquiring the fault domain change event of the distributed storage system and activating the communication suppression state includes: Analyze the system storage topology rules to determine the type and scope of the fault domain; When the type of the fault domain is a target type, a communication suppression command is generated; According to the communication suppression instruction, stop broadcasting update notifications to the associated storage nodes.

3. The method according to claim 1, characterized in that, The monitoring of data version difference information for each storage node under the communication suppression state includes: Based on a preset period, the local data version number of the storage node and the target version number maintained by the management node are obtained, and a historical record of version differences for different preset periods is generated. A dynamic threshold for triggering the data update fetch request is determined based on the current network status and the historical records.

4. The method according to claim 1, characterized in that, The step of calculating communication costs based on the topology of storage nodes and scheduling data reconstruction tasks based on the communication costs includes: Obtain the topology information of the physical location and network connection relationship of the storage nodes; Based on at least one of the factors in the topology information, namely distance, available bandwidth, and network path complexity, the data transmission cost between two storage nodes or a group of storage nodes is quantified, and a cost matrix is ​​generated.

5. The method according to claim 4, characterized in that, The step of calculating communication costs based on the topology of storage nodes and scheduling data reconstruction tasks based on the communication costs includes: Prioritize scheduling data reconstruction tasks with communication costs below a first threshold; For batch data reconstruction tasks with communication costs exceeding the second threshold, batch scheduling is performed.

6. The method according to claim 1, characterized in that, The method further includes: Monitor network traffic across predetermined logical packets; When the network traffic exceeds the congestion threshold, traffic shaping control is initiated; wherein, the traffic shaping control includes adjusting transmission protocol parameters and introducing transmission delay.

7. A data processing apparatus for a storage system, characterized in that, include: The acquisition unit is used to acquire fault domain change events of the distributed storage system and activate the communication suppression state. The monitoring unit is used to monitor the data version difference information of each storage node under the communication suppression state; The update unit is used to initiate a data update retrieval request to the management node in response to the data version difference information meeting preset conditions; The computing unit is used to calculate the communication cost based on the topology of the storage nodes and schedule data reconstruction tasks based on the communication cost.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the data processing method of the storage system as claimed in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the data processing method of the storage system as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the data processing method of the storage system as described in any one of claims 1 to 6.

Images