Adaptive read-write lock partition optimization method, apparatus, equipment and storage medium

Abstract

This invention relates to the field of data processing and discloses an adaptive read-write lock partition optimization method, apparatus, device, and storage medium. The method dynamically adapts to hot and cold data segment adjustments to ensure cross-segment write security. The method includes: continuously collecting real-time access metrics for each data segment protected by the read-write lock; dynamically analyzing and identifying hot and cold data segments based on the changing trends of the real-time access metrics over multiple consecutive collection periods; generating local structure adjustment strategies corresponding to the hot and cold data segments respectively; adjusting the hot and cold data segments based on the local structure adjustment strategies; and during the adjustment process, when a cross-segment write operation involving multiple data segments is triggered, identifying the target data segment involved in the cross-segment write operation, acquiring a global lock based on the sequential coordination and failure retry mechanism of the write intention lock, temporarily freezing the structure adjustment of the target data segment, and uniformly releasing resources and updating the state after the write is completed.

Description

Technical Field

[0001] This invention relates to the field of data processing, and in particular to an adaptive read-write lock partition optimization method, apparatus, device, and storage medium. Background Technology

[0002] In the field of concurrent programming, the rapid development of big data, cloud computing, and distributed systems has made scenarios of multi-threaded, high-frequency access to shared data increasingly common. Typical applications include in-memory databases, key-value stores, caching systems, and index structures. In these scenarios, read operations usually far outnumber write operations. Read-write locks, with their read-heavy, write-single characteristic, have become a core technology for balancing data consistency and concurrency performance, and their performance directly affects system throughput and response latency.

[0003] Traditional read-write locks are mainly divided into monolithic structures and static segmented structures. Monolithic read-write locks protect the entire data structure with a single lock, which is simple to implement but prone to global lock contention under high concurrency, causing a large number of requests to be blocked, and cannot meet the needs of large-scale concurrency. Static segmentation technology splits data by hash buckets, key-value ranges, etc., and each partition is equipped with an independent read-write lock. Although it alleviates some contention, it has many inherent defects.

[0004] Existing segmented read-write locks generally suffer from problems such as concentrated hot spots in segments, unbalanced load between segments, and increased lock conflicts due to dynamic changes in the read-write ratio in high-concurrency environments, making it difficult to meet the low latency and high throughput requirements of large-scale concurrent systems. The core problem with static segmentation structures is the lack of adaptability: the number of segments is fixed after initialization and cannot match dynamic loads. Too few segments lead to concentrated hot spots and intense competition, while too many segments result in resource waste. At the same time, hot segments cannot be locally expanded, and cold segments are difficult to merge, creating a situation where performance bottlenecks and resource redundancy coexist. Cross-segment write operations lack an effective coordination mechanism, and scenarios such as batch updates are prone to deadlocks and lock order disorder. In addition, existing technologies cannot achieve secure scaling without interrupting business operations, and lack access behavior monitoring and dynamic optimization mechanisms, making it impossible to adjust strategies according to the operating status and form an adaptive closed loop.

[0005] Therefore, existing technologies still need improvement and development. Summary of the Invention

[0006] This invention provides an adaptive read / write lock partition optimization method, apparatus, device, and storage medium for dynamically adapting to hot and cold data segmentation adjustments to ensure cross-segment write security.

[0007] The first aspect of this invention provides an adaptive read-write lock partition optimization method, comprising: continuously collecting real-time access metrics of each data segment protected by a read-write lock, the real-time access metrics including the number of read operations, the number of write operations, the number of lock conflicts, and the most recent access timestamp; dynamically analyzing and identifying hot data segments and cold data segments based on the changing trends of the real-time access metrics over multiple consecutive collection periods; generating local structure adjustment strategies corresponding to the hot data segments and the cold data segments respectively; adjusting the hot data segments and the cold data segments based on the local structure adjustment strategies, and during the adjustment process, when a cross-segment write operation involving multiple data segments is triggered, identifying the target data segment involved in the cross-segment write operation, obtaining a global lock based on the sequential coordination and failure retry mechanism of the write intention lock, temporarily freezing the structural adjustment of the target data segment, and uniformly releasing resources and updating the state after the write is completed.

[0008] Preferably, the step of dynamically analyzing and identifying hot and cold data segments based on the changing trend of the real-time access indicators over multiple consecutive collection periods includes: calculating the comprehensive access frequency growth rate, lock conflict change rate, and activity decay coefficient based on the most recent access timestamp for each data segment based on the real-time access indicators; comparing the comprehensive access frequency growth rate, lock conflict change rate, and activity decay coefficient for each data segment with the average value of the corresponding indicators for all data segments to obtain the relative load intensity, relative contention intensity, and relative activity of each data segment; if the relative load intensity of a data segment is greater than a first preset load intensity threshold and the relative contention intensity is greater than a first preset contention intensity threshold, and the relative activity is not greater than a preset activity threshold, then the data segment is determined to be a hot data segment; if the relative load intensity of a data segment is less than a second preset load intensity threshold and the relative contention intensity is less than a second preset contention intensity threshold, and the relative activity is greater than a preset activity threshold, then the data segment is determined to be a cold data segment.

[0009] Preferably, the method for calculating the overall access frequency growth rate includes: obtaining the overall access frequency sequence of the data segment in multiple consecutive collection periods, wherein the overall access frequency of each collection period is the sum of the number of read operations and the number of write operations; performing linear regression analysis on the overall access frequency sequence in multiple consecutive collection periods, and using the slope of the obtained regression line as the overall access frequency growth rate of the data segment.

[0010] Preferably, the method for calculating the lock conflict change rate includes: obtaining a sequence of lock conflict counts for a data segment over multiple consecutive acquisition periods; performing linear regression analysis on the sequence of lock conflict counts over multiple consecutive acquisition periods, and using the slope of the obtained regression line as the lock conflict change rate for the data segment.

[0011] Preferably, the method for calculating the activity decay coefficient includes: obtaining the most recent access timestamp sequence of the data segment within multiple consecutive acquisition periods; calculating the time interval sequence of adjacent periods based on the most recent access timestamp sequence within multiple consecutive acquisition periods; performing linear regression analysis on the calculated time interval sequence, and using the slope of the obtained regression line as the activity decay coefficient of the data segment.

[0012] Preferably, the step of generating local structure adjustment strategies corresponding to the hot data segments and the cold data segments respectively includes:

[0013] For hot data segments, a local splitting strategy is generated, which includes rules and parameters for guiding the splitting of hot data segments into multiple sub-segments; for cold data segments, a local merging strategy is generated, which includes rules and parameters for guiding the merging of cold data segments into a new segment.

[0014] Preferably, the adjustment of the hot data segments and cold data segments based on the local structure adjustment strategy, and during the adjustment process, when a cross-segment write operation involving multiple data segments is triggered, the target data segments involved in the cross-segment write operation are identified, a global lock is acquired based on the order coordination and failure retry mechanism of the write intention lock, and the structural adjustment of the target data segments is temporarily frozen. After the write is completed, resources are released and the status is updated uniformly. This includes: adjusting the hot data segments and cold data segments based on the local structure adjustment strategy, and during the adjustment process, when a cross-segment write operation involving multiple data segments is triggered, identifying all target data segments involved in the cross-segment write operation based on the segment mapping relationship corresponding to the current global structure version number; sorting the target data segments according to preset logic, and attempting to acquire the write intention lock of each target data segment in turn; if the acquisition of the write intention lock of any target data segment fails, all acquired intention locks are revoked and a new application is initiated; after successfully acquiring the write intention locks of all target data segments, the global cross-segment write status is marked, and the local splitting and merging operations of the relevant target data segments are frozen.

[0015] The write locks for each target data segment are acquired in the sorted order, and cross-segment write operations are performed uniformly. After the operation is completed, the write locks and write intention locks are released in reverse order, the global cross-segment write status flag is cleared, and the real-time access metrics for each target data segment are updated synchronously.

[0016] A second aspect of the present invention provides an adaptive read-write lock partition optimization device, comprising: a collection module for continuously collecting real-time access metrics of each data segment protected by a read-write lock, the real-time access metrics including the number of read operations, the number of write operations, the number of lock conflicts, and the most recent access timestamp; an analysis module for dynamically analyzing and identifying hot data segments and cold data segments based on the changing trends of the real-time access metrics over multiple consecutive collection periods; a generation module for generating local structure adjustment strategies corresponding to the hot data segments and the cold data segments respectively; and an adjustment module for adjusting the hot data segments and the cold data segments based on the local structure adjustment strategies. During the adjustment process, when a cross-segment write operation involving multiple data segments is triggered, the target data segment involved in the cross-segment write operation is identified, a global lock is obtained based on the sequential coordination and failure retry mechanism of the write intention lock, and the structural adjustment of the target data segment is temporarily frozen. After the write operation is completed, resources are released and the state is updated uniformly.

[0017] A third aspect of the present invention provides an adaptive read-write lock partition optimization device, comprising: a memory and at least one processor, wherein the memory stores computer-readable instructions, and the memory and the at least one processor are interconnected via a circuit; the at least one processor invokes the computer-readable instructions in the memory to cause the adaptive read-write lock partition optimization device to perform the various steps of the adaptive read-write lock partition optimization method described above.

[0018] A fourth aspect of the present invention provides a computer-readable storage medium storing computer-readable instructions that, when executed on a computer, cause the computer to perform the steps of the adaptive read-write lock partition optimization method as described above.

[0019] The technical solution provided by this invention continuously collects real-time indicators such as the number of read operations, write operations, lock conflicts, and recent access timestamps for each data segment. Based on the trend of indicator changes over multiple consecutive collection periods, it dynamically identifies hot and cold data segments and generates targeted local structure adjustment strategies. This achieves precise optimization by refining lock granularity through hot segment splitting and reducing redundant locks through cold data segment merging. This effectively alleviates lock contention caused by concentrated hotspots, reduces resource waste caused by cold segments, and improves system resource utilization efficiency and concurrency processing capabilities. Furthermore, for cross-segment write operations that may be triggered during the adjustment process, a write intention lock order coordination and failure retry mechanism is designed. Through processes such as accurately identifying target data segments, orderly acquiring lock resources, and temporarily freezing the target data segment structure adjustment, it avoids conflicts and deadlock risks between cross-segment writes and local adjustments, ensuring data consistency. While maintaining online service without interruption, it achieves dynamic optimization of segment structure and secure collaboration between cross-segment write operations, taking into account the core requirements of low latency and high throughput. It is suitable for various high-concurrency scenarios such as in-memory databases and caching systems. Attached Figure Description

[0020] Figure 1 A flowchart of the adaptive read-write lock partition optimization method provided in an embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram of the adaptive read-write lock partition optimization device provided in an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of the adaptive read-write lock partition optimization device provided in an embodiment of the present invention. Detailed Implementation

[0023] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 An adaptive read-write lock partition optimization method in an embodiment of the present invention includes:

[0025] S101. Continuously collect real-time access metrics for each data segment protected by read-write locks. Real-time access metrics include the number of read operations, the number of write operations, the number of lock conflicts, and the most recent access timestamp.

[0026] S102. Based on the changing trend of real-time access indicators over multiple consecutive collection cycles, dynamically analyze and identify hot data segments and cold data segments.

[0027] S103. Generate local structure adjustment strategies corresponding to hot data segments and cold data segments respectively;

[0028] S104. Based on the local structure adjustment strategy, hot data segments and cold data segments are adjusted. During the adjustment process, when a cross-segment write operation involving multiple data segments is triggered, the target data segment involved in the cross-segment write operation is identified. A global lock is obtained based on the order coordination and failure retry mechanism of the write intention lock, and the structure adjustment of the target data segment is temporarily frozen. After the write is completed, resources are released and the status is updated in a unified manner.

[0029] It is understood that the executing entity of this invention can be an adaptive read-write lock partition optimization device, or it can be a terminal or a server; no specific limitation is made here. This embodiment of the invention will be described using a server as an example.

[0030] This embodiment provides an adaptive read-write lock partitioning optimization method. It continuously collects real-time metrics such as the number of read operations, write operations, lock conflicts, and recent access timestamps for each data segment. Based on the trend of metric changes over multiple consecutive collection periods, it dynamically identifies hot and cold data segments and generates targeted local structure adjustment strategies. This achieves precise optimization by splitting hot data segments to refine lock granularity and merging cold data segments to reduce redundant locks. This effectively alleviates lock contention caused by concentrated hot data, reduces resource waste from cold segments, and improves system resource utilization efficiency and concurrency processing capabilities. Furthermore, for cross-segment write operations that may be triggered during the adjustment process, a write intention lock order coordination and failure retry mechanism is designed. Through precise identification of target data segments, orderly acquisition of lock resources, and temporary freezing of target data segment structure adjustments, it avoids conflicts and deadlock risks between cross-segment writes and local adjustments, ensuring data consistency. While maintaining online service without interruption, it achieves safe collaboration between dynamic segment structure optimization and cross-segment write operations, taking into account the core requirements of low latency and high throughput. It is suitable for various high-concurrency scenarios such as in-memory databases and caching systems.

[0031] In this embodiment, in step S101, a counter field (such as the number of read operations, the number of write operations, and the number of lock conflicts) supporting atomic operations is set in the metadata structure of each data segment. Whenever a corresponding event occurs (such as lock acquisition / release or a conflict occurs), the corresponding counter is incremented by an atomic instruction.

[0032] In this embodiment, a recently accessed timestamp field is maintained in each data segment. After any thread successfully acquires a read lock or write lock on that segment, it atomically updates this timestamp using the current system time (or a monotonically increasing clock).

[0033] In this embodiment, a separate monitoring thread or a fixed time window triggers the process to traverse all active data segments. For each data segment, the aforementioned counter value and the most recent access timestamp are atomically read to form a real-time access metric for that period, and the real-time access metric is stored in a circular buffer or monitoring queue.

[0034] In this embodiment, step S102 involves dynamically analyzing and identifying hot and cold data segments based on the changing trends of real-time access metrics over multiple consecutive acquisition cycles. This includes: calculating the comprehensive access frequency growth rate, lock conflict change rate, and activity decay coefficient based on the most recent access timestamp for each data segment based on real-time access metrics; comparing the comprehensive access frequency growth rate, lock conflict change rate, and activity decay coefficient for each data segment with the average value of the corresponding metrics for all data segments to obtain the relative load intensity, relative contention intensity, and relative activity of each data segment; if the relative load intensity of a data segment is greater than a first preset load intensity threshold and the relative contention intensity is greater than a first preset contention intensity threshold, and the relative activity is not greater than a preset activity threshold, then the data segment is determined to be a hot data segment; if the relative load intensity of a data segment is less than a second preset load intensity threshold and the relative contention intensity is less than a second preset contention intensity threshold, and the relative activity is greater than a preset activity threshold, then the data segment is determined to be a cold data segment.

[0035] In this embodiment, by obtaining the comprehensive access frequency sequence of data segments over multiple consecutive acquisition periods (the comprehensive access frequency for each acquisition period is the sum of the number of read operations and the number of write operations), linear regression analysis is performed on the sequence, and the slope of the resulting regression line is used as the growth rate of the comprehensive access frequency for that segment. This method, by fitting multi-period data, can more stably reflect the medium- to long-term growth trend of access load and avoid misjudgments caused by instantaneous fluctuations.

[0036] In other embodiments, the overall access frequency growth rate of each data segment is calculated by weighted summation of its read operation frequency growth rate and write operation frequency growth rate.

[0037] In this embodiment, the lock conflict change rate is used to quantify the frequency change trend of lock conflicts occurring per unit time.

[0038] To more stably reflect the medium-term trend and avoid interference from instantaneous fluctuations, a linear regression analysis method based on multi-period data is used to calculate the rate of change. Specifically, the sequence of lock collision counts in data segments over multiple consecutive acquisition periods is obtained; linear regression analysis is performed on this sequence, and the slope of the resulting regression line is used as the rate of change of lock collisions.

[0039] In other embodiments, the lock conflict change rate can also be calculated in the following way: lock conflict change rate = (current cycle conflict count - previous cycle conflict count) / acquisition cycle duration.

[0040] In this embodiment, based on the most recent access timestamp sequence of data segments across multiple consecutive acquisition periods, the time interval sequence between adjacent periods is first calculated. Then, linear regression analysis is performed on this time interval sequence, and the slope of the resulting regression line is used as the activity decay coefficient for that segment. A negative coefficient indicates a shortening access interval and increased activity; a positive coefficient indicates a lengthening access interval and decreased activity. Its absolute value reflects the rate of decay or increase. This method effectively distinguishes between segments with continuous activity and those with occasional access by quantifying the temporal distribution trend of access frequency.

[0041] In this embodiment, relative load intensity refers to the ratio of the access load (comprehensive read / write frequency) of a single data segment to the overall average load of the system, which is used to quantify whether the data segment is busier or less busy relative to all data segments.

[0042] Relative load intensity = Total access frequency of the current data segment ÷ Average total access frequency of all data segments. A relative load intensity value > 1 indicates that the load is above average; the higher the value, the heavier the load.

[0043] In this embodiment, relative contention intensity refers to the ratio of the intensity of lock contention in a single data segment to the average contention intensity of the entire system, reflecting the relative severity of concurrent conflicts within that data segment.

[0044] Relative competition intensity = rate of change of lock conflict in the current data segment ÷ average rate of change of lock conflict across all data segments.

[0045] In this embodiment, relative activity is a ratio that directly reflects the severity of the decline (or increase) in the activity of a data segment relative to the overall average level.

[0046] Relative activity level = activity decay coefficient of the current data segment ÷ average activity decay coefficient of all data segments.

[0047] The activity decay coefficient of the current data segment is calculated as follows: Based on the most recent access timestamp sequence of the current data segment in multiple consecutive acquisition periods, the interval between adjacent period timestamps is first calculated to obtain the access interval sequence; then, the access interval sequence is analyzed to quantify its trend over time, where a shortening or brief period of access interval indicates increased activity, while a longer interval indicates decreased activity; finally, this trend is quantified into a numerical coefficient, which is positively correlated with the persistence of access activity of the segment within the observation window.

[0048] A negative activity decay coefficient indicates that the access interval is shortening and the activity level of each segment is increasing.

[0049] A zero activity decay coefficient indicates a stable access interval, with no significant decrease or increase in segmented activity, indicating a continuously active state. A positive activity decay coefficient indicates an increasing access interval and decreasing segmented activity.

[0050] In this embodiment, the first preset load intensity threshold, the first preset competition intensity threshold, the preset activity threshold, the second preset load intensity threshold, and the second preset competition intensity threshold are adaptive thresholds that are dynamically adjusted according to the historical load conditions of the system. For example, the first preset load intensity threshold is 1.5, the first preset competition intensity threshold is 1.5, the second preset load intensity threshold is 0.3, the second preset competition intensity threshold is 0.1, and the preset activity threshold is 1.

[0051] If the relative activity level is less than 1, it indicates that the activity decline trend of this data segment is weaker than the average level. Specifically, when the relative activity level is negative or close to 0, it means that the activity level of this data segment has not declined or has even increased, indicating that it is continuously active or in a state of rising popularity. When the relative activity level is between 0 and 1, it means that there is some decline in the activity level of this data segment, but the degree of decline is lower than the average level, and it can still be considered relatively active.

[0052] If the relative activity level is greater than 1, it indicates that the activity level of this data segment is declining more rapidly than the system average, meaning that its activity level is decreasing faster and it is in a state of continuous decline.

[0053] If the relative activity level is 1, it indicates that the activity level decline trend of this segment is consistent with the system average level.

[0054] The following example illustrates how to dynamically analyze and identify hot and cold data segments based on real-time access metrics across multiple consecutive collection cycles. Assume there are three data segments (Seg1, Seg2, Seg3), a monitoring window of 5 collection cycles (T1 to T5), each cycle lasting 1 second, and real-time access metrics for all data segments as shown in Table 1.

[0055] Table 1 Real-time access metrics for all data segments

[0056]

[0057] Taking Seg1 as an example, the overall access frequency for each period is 12, 18, 30, 48, and 72, respectively. By performing linear regression (period number x, total number of accesses y), the slope is calculated, and the overall access frequency growth rate is 15, which means an increase of about 15 accesses per period.

[0058] The number of lock conflicts in each period were 1, 2, 5, 10, and 20, respectively. The slope of the linear regression was calculated, and the rate of change of lock conflicts was approximately 4.6, meaning that the number of conflicts increased by approximately 4.6 times per period.

[0059] The activity decay coefficient is calculated based on the most recent access timestamp.

[0060] Define delay = current period collection time - most recent access timestamp (unit: seconds).

[0061] Assuming the T1 acquisition time is 1000 seconds, then the delay of each Seg1 cycle is 0,0,0,0,0, because the timestamp is updated synchronously with the acquisition time.

[0062] The slope of the delayed linear regression is 0 (indicating no decay and continued activity).

[0063] The average trend indicator values ​​for all data segments (Seg1, Seg2, Seg3) within the same window are shown in Table 2.

[0064] Table 2 shows the average trend indicators for all data segments (Seg1, Seg2, Seg3) within the same window.

[0065]

[0066] Taking Seg1 as an example, the relative metrics for each data segment are as follows:

[0067] Relative load strength = 15.0 / 4.97 ≈ 3.02

[0068] Relative competition intensity = 4.6 / 1.67 ≈ 2.75

[0069] Relative activity = 0.0 / 0.33 = 0.0 (indicating no decay, more active than average)

[0070] Assume the judgment condition is:

[0071] Thermal data segmentation: relative load intensity > 1.5 and relative competition intensity > 1.5 and relative activity ≤ 1.

[0072] Cold data segmentation: relative load intensity < 0.5, relative competition intensity < 0.1, and relative activity > 1.

[0073] Seg1: Relative load intensity = 3.02, relative competition intensity = 2.75, relative activity = 0.0 → meets the thermal data segmentation conditions.

[0074] Seg2: Relative load intensity ≈ -0.22, relative contention intensity = 0.0, relative activity ≈ 3.03 → meets the cold data segmentation condition.

[0075] Seg3: Relative load intensity ≈ 0.20, relative contention intensity ≈ 0.24, relative activity = 0.0 → Low load but active, does not meet the cold data segmentation condition (because the activity has not decreased).

[0076] Therefore, based on trend analysis over five consecutive periods, Seg1 was identified as a hot data segment (load and contention continued to grow rapidly and remained active). Seg2 was identified as a cold data segment (load continued to decline to a low level, there were no conflicts, and activity continued to decrease). Seg3 was neither identified as a hot nor a cold data segment and belonged to a normal load segment.

[0077] In this embodiment, step S103 generates local structure adjustment strategies corresponding to hot data segments and cold data segments, including: generating a local splitting strategy for hot data segments, the local splitting strategy including rules and parameters for guiding the hot data segments to be split into multiple sub-segments; and generating a local merging strategy for cold data segments, the local merging strategy including rules and parameters for guiding the cold data segments to be merged into a new segment.

[0078] In this embodiment, a local splitting strategy is used for hot data segmentation.

[0079] The strategy aims to segment a single, highly competitive hotspot into multiple sub-segments with smaller concurrency granularity, thereby distributing concentrated access requests across multiple locks to reduce average lock wait time and improve system throughput.

[0080] The rules for split dimensions are as follows:

[0081] Hash partitioning: If the original segment is a hash bucket, the rule specifies a new hash function or seed to divide the original hash space into multiple sub-intervals.

[0082] Key-value range: If the original segment manages an ordered key-value range, the rule specifies a new split key to divide the original range into several continuous sub-ranges.

[0083] The split granularity parameter is used to determine how many sub-segments to split into. It is usually a dynamic parameter based on the relative load intensity of the hot data segment (the stronger the load, the more splits may be possible), the lock conflict frequency within the hot data segment (the more intense the conflict, the finer the granularity needs to be), and the preset maximum number of segments limit or load balancing target.

[0084] The data migration and switching rules are as follows:

[0085] Migration batch size: The amount of data migrated each time asynchronously, to balance migration speed with the impact on normal services.

[0086] Version switching conditions: Clearly define the progress of data migration (such as complete or reaching a certain consistency snapshot point) before the global mapping relationship can be atomically updated.

[0087] Old segment resource reclamation strategy: Specifies how long after the switch or under what conditions (such as confirming no residual access) the resources of the original hotspot segment will be reclaimed.

[0088] In this embodiment, a local merging strategy is used for cold data segments.

[0089] The strategy aims to merge multiple low-load, low-activity cold segments into one segment, reducing the number of locks, metadata overhead, and memory fragmentation that the system needs to maintain, thereby improving resource utilization efficiency and cache locality.

[0090] The local merge strategy generation rules and parameters constitute a merge planning scheme, including:

[0091] Segment selection and grouping rules: Determine which cold segments should be merged together.

[0092] The core rule is logical adjacency, such as buckets with adjacent hash values ​​or segments with consecutive key-value ranges.

[0093] The parameters may include a maximum merge distance to prevent physically distant groups from being merged, thus avoiding the violation of data locality.

[0094] Merged range parameter: Defines the management boundary of the new segment.

[0095] For range segmentation, there are new start and end keys.

[0096] For hash segments, that is, the range of hash values ​​covered by the new bucket after merging.

[0097] The rules for data integration and resource recycling are as follows:

[0098] Migration priority: When multiple cold segments are to be merged, the order in which they are merged may be specified.

[0099] Atomic switching point: Similar to splitting, it is stipulated that a one-time mapping switch will only be performed after all data integration is completed.

[0100] Synchronous reclamation parameters: indicate whether to release the locks and memory structures of merged cold segments immediately or asynchronously in batches after the switch.

[0101] In this embodiment, the identification results are transformed into strategies, demonstrating intelligent decision-making capabilities. It is not simply about separating when things are hot and merging when things are cold, but rather includes refined calculations on how to separate / merge more optimally.

[0102] In this embodiment, in step S104, hot data segments and cold data segments are adjusted based on a local structure adjustment strategy. During the adjustment process, when a cross-segment write operation involving multiple data segments is triggered, the target data segment involved in the cross-segment write operation is identified. A global lock is obtained based on the order coordination and failure retry mechanism of the write intention lock, and the structure adjustment of the target data segment is temporarily frozen. After the write is completed, resources are released and the state is updated uniformly, including:

[0103] The system adjusts hot and cold data segments based on a local structure adjustment strategy. During the adjustment process, when a cross-segment write operation involving multiple data segments is triggered, it identifies all target data segments involved in the cross-segment write operation based on the segment mapping relationship corresponding to the current global structure version number. The target data segments are sorted according to a preset logic, and the system attempts to acquire the write intention lock for each target data segment in turn. If the acquisition of the write intention lock for any target data segment fails, all acquired intention locks are revoked and a new application is initiated. After successfully acquiring the write intention locks for all target data segments, the global cross-segment write status is marked, and the local splitting and merging operations of the relevant target data segments are frozen. The write locks for each target data segment are acquired in sorted order, and the cross-segment write operation is executed uniformly. After the operation is completed, the write locks and write intention locks are released in reverse order, the global cross-segment write status mark is cleared, and the real-time access metrics of each target data segment are updated synchronously.

[0104] In this embodiment, the current global structure version number is a logical clock that increments with each successful structure adjustment (split / merge completion). When a thread initiates coordination, it captures and locks this version number, signifying its commitment to planning its operation path based on the currently observed, defined system structure map. This ensures that even if background structure adjustments are underway during identification, the set of target data segments identified by the thread remains logically consistent and momentarily frozen, avoiding location errors caused by map changes during reading, such as a segment being split in the identification list, leading to the omission of some data.

[0105] Before a thread performs a critical operation (acquiring a read lock or a write lock), a version consistency check is performed, which compares the thread-level structure version snapshot it holds with the latest global metadata and the overall version number of the current segment structure.

[0106] If the verification passes (both are consistent), it indicates that the global structure has not changed since the thread located the segment, and the thread can safely continue operating based on the located segment.

[0107] If the verification fails (the two are inconsistent), it indicates that after the thread locates the segment but before it is ready to execute the operation, the background has already completed at least one structural adjustment (the overall version number of the current segment structure has been incremented). At this time, the segment structure that the thread previously located based on the old version snapshot (thread-level structure version snapshot) may have become invalid (for example, the original segment has been split into multiple sub-segments, or merged with other cold segments).

[0108] In this scenario, the thread-level structure version snapshot acts as a critical safety barrier. Verification failure forces the thread to completely abandon the entire access path based on the old snapshot (including the located segment structure), and then automatically re-initiates a complete segment location process. In the new location process, the thread reads the latest overall version number of the current segment structure as the new thread-level structure version snapshot, and recalculates the segment structure index based on the latest segment mapping relationship to locate the correct target segment. Subsequently, the thread must retry acquiring the lock and executing the operation based on the new location result.

[0109] In this embodiment, all concurrent cross-segment write operations are forced to follow the same global lock acquisition order. This mathematically eliminates the necessary condition for circular waits and is one of the most reliable methods for preventing deadlocks.

[0110] A write intention lock is a lightweight declaration or reservation signal. It does not block read operations; its main function is to announce to the system (especially the background adjuster): I am ready to write this segment. Acquiring an intention lock is far less expensive than a formal write lock, making probing inexpensive.

[0111] Full rollback and retry on failure is an embodiment of the All-or-Nothing atomicity guarantee during the lock acquisition phase. If any link in the path is blocked (e.g., the data segment is being held by another write operation, or is in a state of structural unavailability), the thread immediately relinquishes all acquired resources and rolls back completely. This prevents threads from getting deadlocked due to partial lock holding. The mechanism of re-initiating the acquisition adds liveness and fairness to the system, ensuring that operations will eventually succeed when resources become available.

[0112] In this embodiment, marking the global state is equivalent to issuing a distributed transaction announcement to the entire system (especially the background structure adjuster): a cross-segment write transaction is ready, involving segments A, B, C...

[0113] For the segments mentioned in the announcement, any ongoing or upcoming local split or merge operations are immediately paused (blocked). This creates a structurally stable execution sandbox for cross-segment write operations. Within the sandbox, the container (segment) containing the data does not split or merge, thus guaranteeing the logical atomicity of write operations—all modifications are applied to a defined and immutable set of data objects.

[0114] In this embodiment, under the safe premise that the environment is stable (structural adjustment frozen) and the path has been reserved (intention lock acquisition), threads acquire formal write locks in the same order. At this time, the write lock will block other read and write operations, but due to the prior coordination guarantee, this process is fast and deterministic. Subsequently, within the critical section holding all necessary write locks, the thread atomically executes all data modifications. This typically relies on the underlying transaction mechanism to ensure that all changes are either fully committed or fully rolled back.

[0115] In this embodiment, after releasing all locks, the global cross-segment write status flag is cleared, which is equivalent to issuing a second announcement that the transaction is complete: My operation has ended, and segments A, B, C... are now unfrozen. After detecting this status change, the system can resume structural adjustment operations on the frozen segments.

[0116] In this embodiment, to ensure the correctness and data consistency of concurrent access during the dynamic adjustment of hot or cold data segmentation, the access execution flow is as follows:

[0117] Segment positioning process:

[0118] Before performing any read or write operation, the accessing thread first determines the data segment containing its target data (i.e., segment location). The core of this process is capturing and fixing the system architecture view upon which the thread depends, preventing inconsistencies due to structural changes in subsequent operations. The specific steps are as follows:

[0119] ① The thread reads the current global structure version number and records it as the current global version number.

[0120] ② The thread stores the current global version number in the thread-level structure version snapshot field of its thread-local context, forming the structure version snapshot on which it is based for this access.

[0121] ③ The thread uses the metadata such as the number of segments and hash seed corresponding to this version snapshot to calculate the index of the target segment based on the key of the data to be accessed.

[0122] ④ The thread also stores the calculated index of the target segment in its thread-local context (e.g., in the target segment structure field).

[0123] ⑤ The thread retrieves the corresponding segment structure object from the global segment table (segment structure table) based on the segment structure index and returns it.

[0124] Through this process, the accessing thread obtains a target segment bound to a specific global version, providing a deterministic structural context for its subsequent operations.

[0125] Read access process:

[0126] The read access process, based on segment location, adds version consistency verification to ensure that read operations do not access erroneous or migrated data during intermediate structural adjustments. The process is as follows:

[0127] ① The thread execution segment location process obtains the target segment structure object and its thread-level structure version snapshot.

[0128] ② Before the thread prepares to acquire the segment read lock, read the current global metadata and the overall version number of the current segment structure again.

[0129] ③ Compare the overall version number of the current segment structure read this time with the thread-level structure version snapshot saved locally by the thread. If they match, it means that the structure has not changed since the segment location, and execution continues; if they do not match, it means that the segment structure has changed (such as splitting or merging), and the thread must abandon the current location result and jump back to step ① to execute the complete segment location process to relocate based on the latest structure.

[0130] ④ After the version verification passes, the thread acquires the read lock for that segment structure.

[0131] ⑤ The thread performs data read operations while holding a read lock.

[0132] ⑥ After the operation is completed, release the read lock.

[0133] ⑦ The thread updates the read operation count (such as readCount) and other statistical information of this segment structure. This information will be collected by the background access behavior monitor for subsequent trend analysis and decision-making.

[0134] The adaptive read-write lock partition optimization method in the embodiments of the present invention has been described above. The apparatus in the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 2 The implementation methods of the adaptive read-write lock partition optimization device in this invention include:

[0135] The acquisition module 201 is used to continuously collect real-time access metrics for each data segment protected by the read-write lock. The real-time access metrics include the number of read operations, the number of write operations, the number of lock conflicts, and the most recent access timestamp.

[0136] Analysis module 202 is used to dynamically analyze and identify hot data segments and cold data segments based on the changing trend of the real-time access index over multiple consecutive collection cycles.

[0137] The generation module 203 is used to generate local structure adjustment strategies corresponding to the hot data segments and the cold data segments, respectively.

[0138] The adjustment module 204 is used to adjust the hot data segment and the cold data segment based on the local structure adjustment strategy. During the adjustment process, when a cross-segment write operation involving multiple data segments is triggered, the target data segment involved in the cross-segment write operation is identified, a global lock is obtained based on the order coordination and failure retry mechanism of the write intention lock, and the structure adjustment of the target data segment is temporarily frozen. After the write is completed, the resources are released and the status is updated in a unified manner.

[0139] In this embodiment, real-time metrics such as the number of read operations, write operations, lock conflicts, and recent access timestamps for each data segment are continuously collected. Based on the trend of metric changes over multiple consecutive collection periods, hot and cold data segments are dynamically identified, and targeted local structure adjustment strategies are generated. This achieves precise optimization by splitting hot data segments to refine lock granularity and merging cold data segments to reduce redundant locks. This effectively alleviates lock contention caused by concentrated hot data, reduces resource waste caused by cold segments, and improves system resource utilization efficiency and concurrency processing capabilities. At the same time, for cross-segment write operations that may be triggered during the adjustment process, a write intention lock order coordination and failure retry mechanism is designed. Through processes such as accurately identifying target data segments, orderly acquiring lock resources, and temporarily freezing the target data segment structure adjustment, conflicts and deadlock risks between cross-segment writes and local adjustments are avoided, ensuring data consistency. Under the premise of maintaining online business without interruption, dynamic optimization of segment structure and safe collaboration of cross-segment write operations are achieved, taking into account the core requirements of low latency and high throughput of the system. It is suitable for various high-concurrency scenarios such as in-memory databases and caching systems.

[0140] Figure 2 The structure of the adaptive read-write lock partition optimization device shown does not constitute a limitation on the adaptive read-write lock partition optimization device, and can implement the steps of the adaptive read-write lock partition optimization method provided in the above method embodiments.

[0141] above Figure 2 The adaptive read-write lock partition optimization device in this embodiment of the invention is described in detail from the perspective of modular functional entities. The adaptive read-write lock partition optimization device in this embodiment of the invention is described in detail from the perspective of hardware processing.

[0142] Figure 3 This is a schematic diagram of an adaptive read-write lock partition optimization device provided in an embodiment of the present invention. The device 300 can vary significantly due to different configurations or performance characteristics. It may include one or more central processing units (CPUs) 310 (e.g., one or more processors) and a memory 320, and one or more storage media 330 (e.g., one or more mass storage devices) for storing application programs 333 or data 332. The memory 320 and storage media 330 can be temporary or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown), each module including a series of instruction operations on the device 300. Furthermore, the processor 310 may be configured to communicate with the storage media 330 and execute a series of instruction operations on the storage media on the device 300.

[0143] Device 300 may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input / output interfaces 360, and / or one or more operating systems 331, such as Windows Server, MacOSX, Unix, Linux, FreeBSD, etc.

[0144] This invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of an adaptive read-write lock partition optimization method.

[0145] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0146] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0147] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An adaptive read-write lock partitioning optimization method, characterized in that, The adaptive read-write lock partition optimization method comprises: continuously collecting real-time access indicators of each data segment protected by the read-write lock, the real-time access indicators comprising the number of read operations, the number of write operations, the number of lock conflicts, and the latest access timestamp; based on the change trend of the real-time access indicators in consecutive multiple collection periods, dynamically analyzing and identifying hot data segments and cold data segments; generating local structure adjustment strategies corresponding to the hot data segments and the cold data segments, respectively; adjusting the hot data segments and the cold data segments based on the local structure adjustment strategies, and in the adjustment process, when a cross-segment write operation involving multiple data segments is triggered, identifying target data segments involved in the cross-segment write operation, obtaining global locking based on the sequential coordination and failure retry mechanism of the write intention lock, temporarily freezing the structure adjustment of the target data segments, and uniformly releasing resources and updating the state after the write is completed; the adjustment of the hot data segments and the cold data segments based on the local structure adjustment strategies, and in the adjustment process, when a cross-segment write operation involving multiple data segments is triggered, identifying target data segments involved in the cross-segment write operation, obtaining global locking based on the sequential coordination and failure retry mechanism of the write intention lock, temporarily freezing the structure adjustment of the target data segments, and uniformly releasing resources and updating the state after the write is completed, comprises: adjusting the hot data segments and the cold data segments based on the local structure adjustment strategies, and in the adjustment process, when a cross-segment write operation involving multiple data segments is triggered, identifying all target data segments involved in the cross-segment write operation based on the segment mapping relationship corresponding to the current global structure version number; sorting the target data segments according to a preset logic, and sequentially attempting to obtain the write intention lock of each target data segment, if the write intention lock of any target data segment fails to be obtained, then revoking all the intention locks that have been obtained and reinitiating the application; after successfully obtaining the write intention lock of all the target data segments, marking the global cross-segment write state, freezing the local splitting and merging operations of the related target data segments; obtaining the write lock of each target data segment in the order, uniformly executing the cross-segment write operation, releasing the write lock and the write intention lock in the reverse order after the operation is completed, clearing the global cross-segment write state marker, and synchronously updating the real-time access indicators of each target data segment.

2. The adaptive read-write lock partitioning optimization method of claim 1, wherein, the dynamic analysis and identification of the hot data segments and the cold data segments based on the change trend of the real-time access indicators in consecutive multiple collection periods, comprises: based on the real-time access indicators, calculating the comprehensive access frequency growth rate of the number of read operations and the number of write operations, the lock conflict change rate, and the activity decay coefficient based on the latest access timestamp of each data segment in consecutive multiple collection periods; comparing the comprehensive access frequency growth rate, the lock conflict change rate, and the activity decay coefficient of each data segment with the average values of the corresponding indicators of all data segments to obtain the relative load intensity, the relative competition intensity, and the relative activity of each data segment; If the relative load intensity of a data segment is greater than the first preset load intensity threshold and the relative competition intensity is greater than the first preset competition intensity threshold, and the relative activity is not greater than the preset activity threshold, the data segment is determined as a hot data segment; If the relative load intensity of a data segment is less than the second preset load intensity threshold and the relative competition intensity is less than the second preset competition intensity threshold, and the relative activity is greater than the preset activity threshold, the data segment is determined as a cold data segment.

3. The adaptive read-write lock partitioning optimization method of claim 2, wherein, The calculation method of the comprehensive access frequency growth rate comprises: obtaining a comprehensive access frequency sequence of the data segment in continuous multiple collection periods, the comprehensive access frequency of each collection period being the sum of the read operation times and the write operation times; performing linear regression analysis on the comprehensive access frequency sequence in the continuous multiple collection periods, and taking the slope of the obtained regression straight line as the comprehensive access frequency growth rate of the data segment.

4. The adaptive read-write lock partitioning optimization method of claim 2, wherein, The calculation method of the lock conflict change rate comprises: obtaining a lock conflict number sequence of the data segment in continuous multiple collection periods; performing linear regression analysis on the lock conflict number sequence in the continuous multiple collection periods, and taking the slope of the obtained regression straight line as the lock conflict change rate of the data segment.

5. The adaptive read-write lock partitioning optimization method of claim 2, wherein, The calculation method of the activity decay coefficient comprises: obtaining a latest access timestamp sequence of the data segment in continuous multiple collection periods; calculating a time interval sequence of adjacent periods based on the latest access timestamp sequence in the continuous multiple collection periods; performing linear regression analysis on the calculated time interval sequence, and taking the slope of the obtained regression straight line as the activity decay coefficient of the data segment.

6. The adaptive read-write lock partitioning optimization method of claim 1, wherein, The respective generation of the local structure adjustment strategies corresponding to the hot data segment and the cold data segment comprises: for the hot data segment, generating a local splitting strategy, the local splitting strategy comprising rules and parameters for guiding the splitting of the hot data segment into multiple sub-segments: for the cold data segment, generating a local merging strategy, the local merging strategy comprising rules and parameters for guiding the merging of the cold data segment into a new segment.

7. An adaptive read-write lock partitioning optimization apparatus, comprising: comprise: a collection module for continuously collecting real-time access indicators of each data segment protected by read-write lock, the real-time access indicators comprising read operation times, write operation times, lock conflict numbers, and latest access timestamps; an analysis module for dynamically analyzing and identifying hot data segments and cold data segments based on the change trend of the real-time access indicators in continuous multiple collection periods; a generation module for generating local structure adjustment strategies corresponding to the hot data segment and the cold data segment, respectively; The adjustment module is configured to adjust the hot data segments and the cold data segments based on a local structure adjustment strategy, and in the adjustment process, when a cross-segment write operation involving multiple data segments is triggered, target data segments involved in the cross-segment write operation are identified, a global lock is obtained based on a sequential coordination and failure retry mechanism of a write intention lock, and the structure adjustment of the target data segments is temporarily frozen, and after the write is completed, resources are uniformly released and the state is updated. The target data segments are sorted according to a preset logic, write intention locks of the target data segments are successively tried to be obtained, if the write intention lock of any target data segment fails to be obtained, all the intention locks that have been obtained are revoked and the application is reinitiated, after the write intention locks of all the target data segments are successfully obtained, a global cross-segment write state is marked, and local splitting and merging operations of the related target data segments are frozen, write locks of the target data segments are obtained according to the sorting order, the cross-segment write operation is uniformly executed, after the operation is completed, the write locks and the write intention locks are released according to the reverse order, the global cross-segment write state mark is cleared, and real-time access indexes of the target data segments are synchronously updated.

8. An adaptive read-write lock partitioning optimization device, comprising: The memory and at least one processor are included, and the memory stores computer readable instructions; The at least one processor invokes the computer readable instructions in the memory to perform the steps of the adaptive read-write lock partition optimization method in any one of claims 1-6.

9. A computer-readable storage medium having stored thereon computer-readable instructions, wherein, The computer readable instructions are executed by the processor to implement the steps of the adaptive read-write lock partition optimization method in any one of claims 1-6.

Images