Partition lock acquisition method and device, storage medium, electronic equipment and program

Abstract

The invention discloses a partition lock obtaining method and device, a storage medium, electronic equipment and a program, and relates to the field of distributed storage systems.The partition lock obtaining method comprises the steps that under the condition that multiple operation requests sent by one or more clients are received, the multiple operation requests are pre-checked through a root path filter, a directory filter and a lock type filter, the first operation request does not need to be locked and / or does not cause lock competition, and a second operation request causing lock competition is obtained; and calculating a lock competition frequency corresponding to the second operation request, and determining a target lock area corresponding to the second operation request according to the lock competition frequency so as to reduce the lock competition frequency of the hotspot area. Through the method and the device, the problem that lock competition is serious in lock management mechanisms such as a global lock mechanism and a fixed partition lock in related technologies is solved, and then lock competition in a hot spot area can be reduced.

Description

Technical Field

[0001] The present application relates to the field of distributed storage systems, and in particular to a method and device for obtaining a partition lock, a storage medium, an electronic device, and a program. Background Art

[0002] In related technologies, the lock management mechanisms of distributed storage metadata servers (MDSs) include: 1) Global locks: This mechanism sets a global read-write lock for metadata operations. Read operations share the lock resource, while write operations hold an exclusive lock. When a client acquires an exclusive lock for a write operation, read and write requests from other clients must wait until the lock is released. This approach is straightforward and easy to implement and manage. However, in high-concurrency scenarios, all requests under this global lock mechanism compete for the same lock, resulting in a large number of requests waiting in queues, extremely low system throughput, and severely limiting the system's concurrent processing capabilities. 2) Fixed partition locks: This mechanism divides the lock space into multiple fixed zones based on rules such as directory hierarchy and file hashes, with each zone independently locked. For example, directories at different levels of the directory tree are assigned to different lock zones, allowing operations in different zones to proceed concurrently and reducing lock contention. However, the lock granularity of fixed partition locks cannot be adjusted dynamically. Once a partition is determined, it is difficult to change, making it difficult to adapt to dynamic load fluctuations in different time periods and business scenarios. In some cases, some partition locks may be highly contested while other partition locks may be idle.

[0003] Therefore, lock management mechanisms such as the global lock mechanism and the fixed partition lock in related technologies have the problem of serious lock contention.

[0004] Therefore, the serious lock contention problem existing in lock management mechanisms such as the global lock mechanism and the fixed partition lock in related technologies has not yet been effectively solved. Summary of the Invention

[0005] The present application provides a partition lock acquisition method and device, a storage medium, an electronic device, and a program to at least solve the serious lock competition problem existing in lock management mechanisms such as global lock mechanisms and fixed partition locks in related technologies.

[0006] The present application provides a method for acquiring a partition lock, comprising: upon receiving multiple operation requests sent by one or more clients, pre-checking the multiple operation requests through a filter index to filter a first operation request from the multiple operation requests to obtain a second operation request, wherein the filter index includes at least one of the following: a root path filter, a directory filter, and a lock type filter; the first operation request is an operation request that does not require locking and / or does not cause lock contention; the second operation request is an operation request that causes lock contention; calculating the number of lock contentions corresponding to the second operation request, and determining a target lock area corresponding to the second operation request based on the number of lock contentions; and acquiring the partition lock corresponding to the second operation request based on the target lock area.

[0007] The present application also provides a device for acquiring a partition lock, comprising: a pre-check module, for pre-checking the multiple operation requests through a filter index when receiving multiple operation requests sent by one or more clients, so as to filter the first operation request among the multiple operation requests and obtain a second operation request, wherein the filter index includes at least one of the following: a root path filter, a directory filter, and a lock type filter, the first operation request is: an operation request that does not require locking and / or an operation request that does not cause lock contention, and the second operation request is an operation request that causes lock contention; a calculation module, for calculating the number of lock contentions corresponding to the second operation request, and determining the target lock area corresponding to the second operation request based on the number of lock contentions; an acquisition module, for acquiring the partition lock corresponding to the second operation request based on the target lock area.

[0008] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned partition lock acquisition methods when executing the computer program.

[0009] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods for acquiring a partition lock are implemented.

[0010] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned partition lock acquisition methods when executed by a processor.

[0011] Through this application, when multiple operation requests are received from one or more clients, the multiple operation requests are pre-checked using a root path filter, a directory filter, and a lock type filter to filter out first operation requests that do not require locking and / or do not cause lock contention, thereby obtaining second operation requests that cause lock contention. The number of lock contentions corresponding to the second operation request is calculated, and the target lock area corresponding to the second operation request is determined based on the number of lock contentions to reduce the number of lock contentions in hotspot areas. Through this application, the problem of severe lock contention in lock management mechanisms such as global lock mechanisms and fixed partition locks in related technologies is solved, thereby reducing lock contention in hotspot areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0013] Figure 1 This is a hardware structure block diagram of a computer terminal for a partition lock acquisition method according to an embodiment of the present application;

[0014] Figure 2 is a flowchart of a method for acquiring a partition lock according to an embodiment of the present application;

[0015] Figure 3 is a diagram of a distributed storage system architecture according to an optional embodiment of the present application;

[0016] Figure 4 This is a flowchart of a distributed storage metadata lock optimization method based on a multi-stage Bloom filter according to an optional embodiment of the present application;

[0017] Figure 5 4 is a framework diagram of a partition lock acquisition device according to an embodiment of the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0019] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0020] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0021] In conjunction with the specific application environment architecture or specific hardware architecture on which the execution of the partition lock acquisition method depends, the specific application environment architecture or specific hardware architecture is described herein.

[0022] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure diagram of a computer terminal for a partition lock acquisition method according to an embodiment of the present application. Figure 1 As shown, the computer terminal may include one or more ( Figure 1 Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor unit (MPU) or a programmable logic device (PLD) and a processing device) and a memory 104 for storing data. The computer terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. For example, the computer terminal may also include Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0023] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for determining the interactive state in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0024] Transmission device 106 is used to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by a computer terminal's communications provider. In one embodiment, transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0025] Figure 2 This is a flowchart of a partition lock acquisition method according to an embodiment of the present application, which can be applied to Figure 1 In a computer terminal, such as Figure 2 As shown, the process includes the following steps:

[0026] Step S202: Upon receiving multiple operation requests sent by one or more clients, pre-check the multiple operation requests using a filter index to filter a first operation request from the multiple operation requests to obtain a second operation request, wherein the filter index includes at least one of the following: a root path filter, a directory filter, and a lock type filter; the first operation request is an operation request that does not require locking and / or does not cause lock contention; and the second operation request is an operation request that causes lock contention.

[0027] The above operation request refers to a request sent by a client to the distributed storage system to perform a specific file system operation (for example, reading, writing, creating, or deleting a file).

[0028] The root path filter is used to check whether the top-level path prefix of the operation request exists in the active lock record;

[0029] The directory filter is used to check whether the target file in the specific directory pointed to by the operation request has been locked;

[0030] The lock type filter is used to check whether the type of operation requested (read or write) may conflict with other existing locks of the same type.

[0031] Step S204: Calculate the number of lock contention events corresponding to the second operation request, and determine the target lock region corresponding to the second operation request according to the number of lock contention events.

[0032] The number of lock contention times refers to the number of times a lock resource in a target lock region is requested by multiple clients within a certain period of time.

[0033] Step S206: Acquire the partition lock corresponding to the second operation request according to the target lock area.

[0034] Partition locks are a locking mechanism used in distributed storage systems to control access to resources within a target lock zone. Each partition lock can operate independently of other partition locks, allowing concurrent access to different partitions at the same time. This reduces global lock latency and lock contention, thereby improving overall system concurrency. Partition locks typically support two types: read locks and write locks. Read locks allow multiple clients to access the lock simultaneously, while write locks require exclusive access.

[0035] Through the partition lock acquisition method of the present application, when multiple operation requests are received from one or more clients, the multiple operation requests are pre-checked through the root path filter, directory filter, and lock type filter to filter the first operation request that does not require locking and / or does not cause lock contention, and obtain the second operation request that causes lock contention; the number of lock contentions corresponding to the second operation request is calculated, and the target lock area corresponding to the second operation request is determined based on the number of lock contentions to reduce the number of lock contentions in hot spots. Through this application, the problem of serious lock contention in lock management mechanisms such as global lock mechanisms and fixed partition locks in related technologies is solved, thereby reducing lock contention in hot spots.

[0036] Optionally, the pre-checking of the multiple operation requests by using the filter index in step S202 includes the following steps:

[0037] Step 1: Parse the multiple operation requests to obtain a first request path corresponding to each operation request;

[0038] Step 2: Extract the top-level directory prefix contained in each first request path, and perform a first-level pre-check on each operation request based on the root path filter to determine whether there is a first active lock operation under each top-level directory prefix, wherein the top-level directory prefix is ​​used to indicate the highest-level directory in the first request directory corresponding to each first request path; wherein the specific steps of the first-level pre-check may include:

[0039] Obtaining a first number of top-level directories expected to be stored in the root path filter and a first false positive rate corresponding to the root path filter; and calculating a first digit group size of a first digit group included in the root path filter and a second number of first hash functions for mapping the top-level directory prefix to the first digit group according to a first formula and a second formula, wherein the first formula is: , the second formula is: , p1 is the first false positive rate, k1 is the second number, m1 is the first digit group size, and n1 is the first number; calculate the first hash value corresponding to each top-level directory prefix based on the first hash function of the second number; determine the first index value corresponding to the first hash value according to the first digit group of the first digit group size; determine the first bit identifier of the first index position corresponding to the first index value in the root path filter; determine whether the active lock operation exists under each top-level directory prefix according to the first bit identifier.

[0040] It can be understood that by calculating the bit array size and the number of hash functions of the Bloom filter, an effective pre-check of active lock operations on the top-level directory prefix can be achieved, thereby filtering out the first operation request that does not require locking or does not cause lock contention. Specifically:

[0041] 1) Obtain the first number (n1) of top-level directories expected to be stored by the root path Bloom filter (i.e., the root path filter) and the first false positive rate (p1) corresponding to the root path filter. Assume that the system expects to manage 10,000 top-level directory prefixes, and set the first false positive rate (p1) of the root path Bloom filter to no more than 0.001 (i.e., 0.1%) to ensure efficient filtering while maintaining high accuracy.

[0042] 2) Calculate the root path Bloom filter bit array size m1 and the root path Bloom filter hash function number k1 based on the first formula and the second formula.

[0043] 3) Calculate the first hash value: For each top-level directory prefix, calculate its corresponding first hash value using the first hash function of the second number (k1).

[0044] 4) Determine the first index value: Perform a modulo operation on the first hash value and the first digit group size (m1) to obtain the corresponding first index value.

[0045] 5) Set bit flag: In the first digit group of the root path Bloom filter, set the bit flag of the first index position obtained by the hash value mapping to 1, indicating that there may be an active lock operation on the top-level directory prefix.

[0046] 6) Active lock pre-check: When a client sends an operation request, the MDS uses the root path Bloom filter to check the bit flags of the top-level directory prefix in the request. If the bit flags of all mapping locations are 1, it is considered that there may be an active lock operation and further inspection is required. If the bit flag of any mapping location is 0, it is determined that there is no active lock operation under the top-level directory prefix. The request is the first operation request and can be directly released.

[0047] Based on this technical solution, by carefully configuring Bloom filter parameters, we can quickly pre-check operation requests. This reduces the lock resource management burden on the MDS and improves system responsiveness, especially in high-concurrency scenarios. Furthermore, the root path filter quickly filters out requests with no active lock operations under the top-level directory prefix, reducing the subsequent lock contention detection and waiting overhead.

[0048] Step 3. When it is determined that the first active lock operation exists under the top-level directory prefix corresponding to the third operation request among the multiple operation requests, filter the third operation request, and pre-check the fourth operation request among the multiple operation requests except the third operation request through the directory filter and / or the lock type filter, wherein the first operation request includes: the third operation request.

[0049] Pre-checking the fourth operation request except the third operation request among the multiple operation requests by using the directory filter and / or the lock type filter includes the following steps 1 to 3:

[0050] Step 1: parse each fourth operation request to determine the second request path corresponding to the fourth operation request;

[0051] Step 2: Determine the second request directory corresponding to each second request path, and perform a secondary pre-check on the first files contained in each second request directory based on each directory filter corresponding to each second request directory to determine whether there is a second active lock record under each first file; wherein the secondary pre-check steps are:

[0052] Obtaining a third number of files expected to be stored in each second requested directory corresponding to each directory filter and a second false positive rate corresponding to each directory filter; and calculating a second digit group size of the second digit group included in each directory filter and a fourth number of second hash functions used to map the first file information corresponding to each first file to the second digit group according to a third formula and a fourth formula, wherein the third formula is: , the fourth formula is: , p2 is the second false positive rate, k2 is the fourth number, m2 is the second digit group size, and n2 is the third number; performing hash calculation on each first file information based on a second hash function of the fourth number to obtain a second hash value corresponding to each first file information, wherein the first file information includes at least one of the following: the first file name of the first file, the first file identifier of the first file, and the first metadata corresponding to the first file; determining a second index value corresponding to the second hash value according to a second digit group of the second digit group size; determining a second bit identifier of a second index position corresponding to each second index value in each directory filter; and determining whether the active lock operation exists under each first file according to the second bit identifier.

[0053] It is understandable that the directory Bloom filter (directory filter) is the second layer in the multi-level Bloom filter system, which is used to detect in a more fine-grained manner whether a file operation request may involve lock contention. Specifically:

[0054] 1) Directory Bloom filter parameter calculation: For each directory, MDS needs to obtain the third number of files expected to be stored by the directory filter (n2) and the corresponding second false positive rate (p2) for the directory filter. Assume that a directory is expected to store 5000 files, and the second false positive rate p2 of the directory Bloom filter is set to no more than 0.002 (i.e., 0.2%) to ensure that the filter maintains high efficiency and high accuracy.

[0055] 2) Calculate the directory Bloom filter bit array size m2 and the number of directory Bloom filter hash functions k2 according to the third and fourth formulas above.

[0056] The directory Bloom filter construction process is similar to the root path Bloom filter, but focuses more on active lock detection at the file level:

[0057] 3) Calculating a second hash value: For each piece of first file information (eg, file name, file identifier, metadata) in the directory, calculate its corresponding second hash value using a second hash function of a fourth quantity (k2).

[0058] 4) Determine the second index value: Perform a bitwise operation on the calculated second hash value and the second digit group size (m2) to obtain the corresponding second index value.

[0059] 5) Setting the bit flag: In the second digit group of each directory Bloom filter, the bit flag of the second index position obtained by mapping the second hash value is set to 1, indicating that there may be an active lock operation on the file.

[0060] 6) Active lock pre-check: When a request that passes the root path filter enters the directory filter check phase, the MDS uses the directory Bloom filter to check the bit flags of the target file (i.e., the first file) in the request. If the bit flags of all mapping locations are 1, it is considered that an active lock operation may exist and further inspection is required. If the bit flag of any mapping location is 0, it is determined that there is no active lock operation on the file. The request can be directly released or the lock acquisition can be skipped and the operation can be carried out directly.

[0061] It is understandable that through the directory Bloom filter, MDS can quickly pre-check lock requests at the file level, further filtering out requests without active lock operations under the target file. The pre-check process reduces the overhead of subsequent lock acquisition and improves processing efficiency.

[0062] Step 3. When it is determined that the second active lock record exists under the first file corresponding to the fifth operation request among multiple fourth operation requests, filter the fifth operation request and pre-check the sixth operation request among the multiple fourth operation requests except the fifth operation request through the lock type filter, wherein the first operation request includes: the fifth operation request.

[0063] The process of pre-checking the sixth operation request excluding the fifth operation request among the plurality of fourth operation requests through the lock type filter includes the following steps (1) to (3):

[0064] Step (1) parsing the sixth operation request to determine the request type corresponding to the sixth operation request;

[0065] Step (2) determines the target type corresponding to the lock type filter corresponding to each sixth operation request according to the request type, and performs a three-level pre-check on each sixth operation request based on the lock type filter of the target type to determine whether the sixth operation request of each request type has a same-type lock conflict, wherein the lock type filter of the target type includes at least one of the following: a write lock type filter and a read lock type filter, and the same-type lock conflict is used to indicate that the lock operations corresponding to multiple sixth operation requests of any request type are operations requesting to obtain the same type of lock at the same time; wherein the three-level pre-check process specifically includes:

[0066] Obtain the number of target type lock operations expected to occur per second by the lock type filter of the target type and the third false positive rate corresponding to the lock type filter of the target type; calculate the third digit group size of the third digit group included in the lock type filter of the target type and the fifth number of the third hash function used to map the request type to the third digit group according to the fifth formula and the sixth formula, wherein the fifth formula is , the sixth formula is: , p3 is the third false positive rate, k3 is the fifth number, m2 is the third digit group size, and n3 is the number of times; a third hash function based on the fifth number is used to perform hash calculation on the request type of each sixth operation request to obtain a third hash value corresponding to each request type; a third index value corresponding to the third hash value is determined according to a third digit group of the third digit group size; a third bit identifier of a third index position corresponding to the third index value is determined in a lock type filter of the target type; and whether there is a lock conflict of the same type in the sixth operation request of each request type according to the third bit identifier.

[0067] It is understandable that the lock type Bloom filter (i.e., lock type filter) is the last layer in the Bloom filter system. It is used to accelerate the detection of conflicts between locks of the same type and further optimize the lock request processing process. Specifically:

[0068] 1) Obtain parameters: Obtain the expected number of lock operations of the target type per second (n3) and the corresponding third false positive rate (p3) for the lock type filter. Assume that in a specific directory, the average number of read (or write) lock operations per second is 2000. Set the third false positive rate (p3) to no more than 0.008 (0.8%) to maintain a good balance between filtering accuracy and efficiency.

[0069] 2) Calculation parameters: Calculate the Bloom filter bit array size (m3) and the number of hash functions (k3) using the fifth and sixth formulas above.

[0070] 3) Calculate the third hash value: For each lock request type (such as read lock or write lock), use the third hash function of the fifth number (k3) to calculate its corresponding third hash value.

[0071] 4) Determine the third index value: Perform a bitwise operation on the calculated third hash value and the third digit group size (m3) to obtain the corresponding third index value.

[0072] 5) Set the third bit flag: In the third digit group of the lock type filter of the target type, set the third bit flag of the third index position obtained by mapping the third hash value to 1, indicating that the lock type may have a lock conflict of the same type.

[0073] 6) Active lock conflict pre-check: When an operation request sent by a client enters the lock type filter check phase, the MDS uses the lock type filter to check the bit flags of the request type (read or write). If the third bit flags of all mapping positions are 1, a lock conflict of the same type is considered, requiring further inspection. If the third bit flag of any mapping position is 0, it can be preliminarily determined that there is no lock conflict for that lock type, simplifying subsequent processing.

[0074] It is understandable that the lock-type Bloom filter can further limit the scope of conflict detection, optimizing the conflict detection step that originally required traversing all lock records to only detecting conflicts in the Bloom filter hit records, reducing the time complexity of conflict detection from O(n) to O(1), and improving the system response speed.

[0075] Time complexity is O(n): The execution time or resource requirements of an algorithm are linear functions of the input data size (n). Specifically, as the input data size increases, the algorithm's runtime or resource requirements also increase linearly. For example, traversing an array or list typically has a time complexity of O(n) because each element must be accessed once. If the list has 100 elements, 100 operations are required; if it has 1000 elements, 1000 operations are required, and so on.

[0076] Time complexity O(1): The execution time or resource consumption of the algorithm does not increase with the increase of the input data size (n), but maintains constant time or space consumption. That is, no matter how large the input data is, the running time or required resources of the algorithm are fixed and do not increase. O(1) complexity operations include: accessing any position in an array, accessing key-value pairs in a hash table, etc. These operations can be completed immediately through direct addressing or hash function calculation without traversing the entire data structure.

[0077] Step (3) When it is determined that multiple seventh operation requests in the multiple sixth operation requests have the same type of lock conflicts, filter the multiple seventh operation requests to obtain the second operation request, wherein the first operation request includes: the multiple seventh operation requests.

[0078] Optionally, before or after pre-checking the multiple operation requests through the filter index in the above-mentioned step S202, the method further includes: in the case of receiving a first lock request corresponding to the first top-level directory stored in the root path filter, counting based on the first counter in the root path filter to increase the value of the first counter to a first value, wherein the first counter is the counter corresponding to the first top-level directory; in the case of determining that the second lock request corresponding to the second top-level directory stored in the root path filter has been deleted, counting based on the second counter in the root path filter to reduce the value of the second counter to a second value, wherein the second counter is the counter corresponding to the second top-level directory; in the case of determining that the first file lock request under the request directory corresponding to the first directory filter is received, counting based on the third counter in the first directory filter to increase the value of the third counter to a third value, wherein the third counter a counter corresponding to the file indicated by the first file lock request; when it is determined that the second file lock request under the request directory corresponding to the second directory filter has been deleted, counting is performed based on the fourth counter in the second directory filter to increase the value of the fourth counter to a fourth value, wherein the fourth counter is the counter corresponding to the file indicated by the second file lock request, and the directory filter includes: the first directory filter and the second directory filter; when it is determined that a first lock request of the first request type is received, counting is performed based on the fifth counter in the lock type filter corresponding to the first request type to increase the value of the fifth counter to a fifth value; when it is determined that the second lock request of the second request type has been deleted, counting is performed based on the sixth counter in the lock type filter corresponding to the second request type to reduce the value of the sixth counter to a sixth value, wherein the first request type and the second request type are both: read lock request or write lock request.

[0079] It is understandable that this application also defines the update operation of the counter, specifically:

[0080] (1) Root path Bloom filter counter operation, including:

[0081] 1) Incrementing the First Value: Upon receiving the first lock request for the first top-level directory (e.g., / data), the MDS updates the corresponding first counter in its root path Bloom filter. For example, if a lock request is requested for a file in the / data directory, the MDS increments the first counter mapped to the directory prefix by the first value. For example, the MDS increments the first counter from 5 to 6 (the first value).

[0082] 2) Decrease to a Second Value: Upon determining that the second lock request corresponding to the second top-level directory (e.g., / non_active) has been deleted, the MDS updates the second counter in its root path Bloom filter. For example, if the lock on the / non_active directory is released, the MDS decrements the second counter mapped to that directory prefix to the second value. For example, the MDS decrements the second counter from 14 to 13 (the second value).

[0083] (2) Directory Bloom filter counter operations, including:

[0084] 1) Incrementing the Third Value: Upon receiving a first file lock request in the request directory corresponding to the first directory filter (e.g., / data / active_files (active files in the data directory)), the MDS updates the third counter corresponding to the first directory filter. For example, if the lock request is for / data / active_files / file1.txt (file 1 in the active file list in the data directory), the MDS increments the third counter corresponding to that file to the third value.

[0085] 2) Reduced to a fourth value: When it is determined that the second file lock request under the request directory corresponding to the second directory filter (for example, / data / inactive_files (inactive files under the data directory)) has been deleted, the MDS updates its fourth counter in the second directory filter and reduces the fourth counter value of the file mapping to the fourth value.

[0086] (3) Lock type Bloom filter counter operations, including:

[0087] 1) Increment the fifth value: Upon receiving a first lock request of the first request type (e.g., a read lock request), the MDS updates the corresponding fifth counter in the lock type filter. For example, if the request is a read lock, the MDS increments the fifth counter in the read lock type filter to the fifth value.

[0088] 2) Reduce to a sixth value: When determining that the second lock request of the second request type (such as a write lock request) has been deleted, the MDS updates its sixth counter in the lock type filter and reduces the sixth counter value corresponding to the lock type to the sixth value.

[0089] In summary, by using counters, the Bloom filter can more accurately reflect the real-time status of the lock operation, avoid the accumulation of false positives in the Bloom filter caused by dynamic changes in the lock operation, and improve the accuracy and effectiveness of the Bloom filter.

[0090] Optionally, the calculation of the number of lock contentions corresponding to the second operation request in the above-mentioned step S204 includes: parsing the second operation request to determine that the second operation request corresponds to a third request path; performing a hash operation on the third request path to determine a fourth hash value corresponding to the third request path; performing a modulo operation on the fourth hash value to determine the first area index corresponding to the fourth hash value, and determining the first initial lock area corresponding to the third request path based on the first area index; counting the number of lock contentions contained in the first initial lock area based on a preset time interval, and determining the number of lock contentions contained in the first initial lock area as the number of lock contentions corresponding to the second operation request.

[0091] It is understandable that this application also defines a dynamic partition lock strategy, which can optimize lock contention management by adjusting the lock region size in real time. Specifically:

[0092] 1) Parsing and Hashing: When the MDS receives a second operation request from the client, it parses the request to determine the actual third request path. For example, if the client sends a write operation request with the path: / user / profiles / user1000 / settings.json (where / user represents the user directory, / profiles represents the configuration folder, / user1000 represents the subdirectory for a specific user, and settings.json is a settings file stored in JSON (Lightweight Data Interchange Format)), the MDS can parse the specific file path to: / user / profiles / user1000 / settings.json.

[0093] The MDS then performs a hash operation on the parsed third request path. Any suitable hash function, such as Message-Digest Algorithm 5 (MD5) or Secure Hash Algorithm 256 (SHA256), can be used to convert the path into a fourth hash value. For example, assume that the hash value of / user / profiles / user1000 / settings.json is 123456.

[0094] 2) Determine the lock region: The MDS performs a modulo operation on the fourth hash value to determine the first region index corresponding to the fourth hash value. Assume there are N lock regions, and the initial size of each region is S. Initially, each region contains S files or directories. By applying 123456 modulo N, a number between 0 and N-1 is obtained as the first region index.

[0095] 3) Lock contention statistics: The dynamic partition lock strategy also includes real-time lock contention statistics. MDS counts the number of lock contentions for each first initial lock region at a preset interval (for example, every 100ms) and uses this number as the lock contention count for the second operation request. The lock contention count reflects the lock resource usage within a region, specifically the number of clients attempting to acquire locks for that region at the same time.

[0096] This technical solution dynamically adjusts the size of lock regions based on the number of lock contentions. When lock contention counts are too high in a region, the strategy automatically reduces the region size and separates hotspots, thereby reducing lock contention and improving concurrent processing capabilities. Conversely, when lock contention counts are low, the strategy merges cold regions and increases the region size, reducing lock management overhead and improving resource utilization.

[0097] Optionally, determining the target lock region corresponding to the second operation request according to the lock contention count in step S204 includes the following methods:

[0098] Method 1:

[0099] Determine the historical maximum number of lock contentions in the first initial lock region corresponding to the second operation request within a target time period before the current time; adjust the size of a first region corresponding to the first initial lock region according to the number of lock contentions and the historical maximum number of lock contentions; specifically, adjust the size of the first region using the seventh formula, where the seventh formula is: , S is the second region size obtained by adjusting the first region size, S0 is the first region size, C max is the maximum number of lock contention times in history, C is the number of lock contention times, and α is the adjustment coefficient. The adjusted first initial lock region is determined as the target lock region.

[0100] It is understandable that method 1 is achieved by monitoring and analyzing the historical contention of lock requests. Specifically:

[0101] 1) Determine the historical maximum number of lock contentions: MDS needs to determine the historical maximum number of lock contentions in the first initial lock region (i.e., the lock region located by hashing) corresponding to a specific operation request (e.g., the second operation request, i.e., a write operation request to / user / profiles / user1000 / settings.json) within a target time period before the current time (e.g., within ten minutes before the current time).

[0102] 2) Adjust the lock area size based on the number of lock contentions: MDS adjusts the lock area size based on the current number of lock contentions (C) and the historical maximum number of lock contentions (C max ), combined with the adjustment coefficient (α), the size of the first initial lock area (S0) is adjusted using the seventh formula above.

[0103] Among them, if the current lock contention number C is close to the historical maximum lock contention number C max , which means that the lock area faces high competition pressure. In the seventh formula (C max -C) / C max The value of will be smaller. Adding the adjustment coefficient α and multiplying it by S0 will reduce S, thereby splitting the region, reducing the number of files in each sub-region, and ultimately reducing the scope of lock contention; if the current lock contention number C is much lower than the historical maximum lock contention number C max , in the seventh formula (C max -C) / C max If the value of is large, adding the adjustment coefficient α and multiplying it by S0 will increase S, thereby merging adjacent low-contention areas, increasing the number of files in the lock area, improving resource utilization and reducing management overhead.

[0104] 3) Determine the target lock area: The adjusted lock area is determined as the target lock area.

[0105] Through the above technical solution, adjusting the lock region size enables the system to reduce lock management overhead under low load, and effectively disperse lock competition and improve resource utilization under high load.

[0106] Method 2:

[0107] Determine the historical maximum number of lock contentions in the first initial lock region corresponding to the second operation request within a target time period before the current time; determine a first size relationship between the lock contention number and the historical maximum number of lock contentions; if the first size relationship indicates that a first lock contention number among multiple lock contentions corresponding to multiple second operation requests is less than or equal to the historical maximum number of lock contentions, determine the second initial lock region corresponding to the first lock contention number as a first low contention region; merge the first low contention regions based on a preset rule to generate a target lock region corresponding to the first lock contention number. If the first size relationship indicates that a third lock contention number among multiple lock contentions corresponding to multiple second operation requests is greater than the historical maximum number of lock contentions, determine the fourth initial lock region corresponding to the third lock contention number as a first high contention region, and suspend receiving lock requests for the first high contention region; and distribute lock records contained in the first high contention region to multiple first sub-lock regions, wherein the target lock region corresponding to the third lock contention number includes: any first sub-lock region.

[0108] It is understandable that dynamic lock region adjustment also includes the identification and merging of low-contention regions. Specifically:

[0109] 1) Determine the historical maximum lock contention count: When the MDS receives a second operation request, in addition to evaluating the lock contention count for the current lock region, it also needs to determine the historical maximum lock contention count for the first initial lock region corresponding to the second operation request (i.e., the lock region assigned to this request) within a target time period before the current time (for example, 10 minutes).

[0110] 2) Determine the lock contention relationship: Compare the number of lock contentions C in the current lock zone with the historical maximum number of lock contentions C max The first size relationship between them.

[0111] 3) Identification and Merging of First Low-Contentment Regions: Once it is determined that the first relationship between the number of lock contentions and the historical maximum number of lock contentions indicates that contention counts are lower than the historical maximum, the second initial lock region with contention counts lower than or equal to the historical maximum number of lock contentions is marked as the first low-contention region. These first low-contention regions are then merged based on pre-set rules to generate the target lock region.

[0112] The merged target lock area will have a wider file range and larger lock management granularity. For example, the size of each area originally containing 128 files will be adjusted to 256 or more, thereby reducing the number of locks in the system and reducing the additional overhead caused by lock management.

[0113] 4) Determining the first high contention region: Once it is determined that the first magnitude relationship between the number of lock contentions and the historical maximum number of lock contentions indicates that the number of contentions is higher than the historical maximum number, the second initial lock region having the number of contentions higher than the historical maximum number of lock contentions is marked as the first high contention region.

[0114] 5) Pause accepting lock requests: Once the most contention-ridden region is identified, suspend accepting lock requests for that region. This prevents new lock requests from interfering with the split process, ensuring the correctness of the split operation and data consistency.

[0115] 6) Lock record reallocation: All lock records in the first high-contention zone are distributed across multiple first sub-zones. Each first sub-zone independently manages its own lock resources, thereby distributing lock contention. Specifically, the original zone can be remapped based on the hash value of the file path, splitting it into two or more sub-zones, each with independent read-write locks.

[0116] The above technical solution effectively reduces the number of locks and lock management overhead by merging the first low-contention region in low-contention situations, freeing up more resources for data transmission and processing, and improving overall system efficiency. Furthermore, by automatically splitting the first high-contention region, hotspots can be dispersed, reducing the concentration of lock contention and thereby improving the throughput of concurrent operations.

[0117] Method 3:

[0118] Determine a second size relationship between the number of lock contention times and a first preset threshold; when the second size relationship indicates that a second lock contention number among multiple lock contention times corresponding to multiple second operation requests is less than or equal to the first preset threshold, determine a third initial lock area corresponding to the second lock contention number as a second low contention area; merge the second low contention areas based on preset rules to generate a target lock area corresponding to the second lock contention number.

[0119] Determine a third size relationship between the number of lock contention times and a second preset threshold; when the third size relationship indicates that there is a fourth lock contention number greater than the second preset threshold among the multiple lock contention times corresponding to the multiple second operation requests, determine the fifth initial lock area corresponding to the fourth lock contention number as the second high contention area, and suspend receiving lock requests for the second high contention area; allocate the lock records contained in the second high contention area to multiple second sub-lock areas, wherein the target lock area corresponding to the fourth lock contention number includes: any second sub-lock area.

[0120] It is understandable that by monitoring the relationship between the number of lock contention times and a preset threshold, dynamic adjustment of lock regions can be achieved, that is, merging lock regions with low contention and splitting lock regions with high contention. Specifically:

[0121] 1) Determining a second relationship between the number of lock contention attempts and a preset threshold: Determine a second relationship between the number of lock contention attempts corresponding to multiple second operation requests (i.e., multiple read and write operation requests to the file system) and the first preset threshold within a target time period before the current time. For example, assuming the first preset threshold T1 is set to 50 attempts per second, if the number of lock contention attempts C2 for a region is less than or equal to T1 (i.e., C2 ≤ T1), the region is determined to be a second-low contention region.

[0122] 2) Merge the second lowest contention area: Once an area is determined to be the second lowest contention area, it is merged based on preset rules to generate a target lock area.

[0123] The purpose of the merge operation is to reduce the number of lock regions, thereby reducing the lock management overhead and improving resource utilization.

[0124] For example, if the lock contention counts for lock regions 56 and 57 are both lower than or equal to the first preset threshold T1 within the same time period, these two regions are considered the second lowest contention regions. They can be merged into a new region, such as region 56A, and the target lock region is this merged region 56A.

[0125] 3) Determining a third relationship between the number of lock contentions and the second preset threshold: Determine a third relationship between the number of lock contentions corresponding to the plurality of second operation requests and the second preset threshold within a target time period before the current time. For example, assuming the second preset threshold T2 is set to 150 times / second, if the number of lock contentions C4 in a region is greater than T2 (i.e., C4 > T2), then the region is determined to be the second most contention region.

[0126] 4) Splitting the Second Most Contention Region: Once a region is identified as the second most contention region, lock requests for that region are temporarily stopped to avoid unnecessary lock contention during the split operation. The lock records for the original region are then distributed across multiple second-tier sub-lock regions, ensuring that each sub-region has sufficient resources to handle requests.

[0127] For example, if the number of lock contentions in lock region 58 suddenly jumps to 200 times / second, exceeding the second preset threshold T2, the MDS will stop accepting new lock requests for region 58 and redistribute the lock records contained in that region to the two second sub-lock regions, namely 58A and 58B. In this way, the lock contention originally concentrated in region 58 is dispersed across the two sub-regions.

[0128] In summary, merging the second-lowest contention region reduces the number of locks, lowers lock management overhead, and improves system resource utilization. Splitting the second-highest contention region disperses hotspot contention, reduces the likelihood of lock conflicts, and thus improves concurrency performance.

[0129] Optionally, after pre-checking the multiple operation requests through the filter index in the above step S202 to filter the first operation request among the multiple operation requests and obtaining the second operation request, the method further includes: when it is determined based on the root path filter, directory filter and lock type filter that the first operation request does not exist in the multiple operation requests, prohibiting the calculation of the number of lock contentions corresponding to the second operation request, and determining to allow the one or more clients to execute the multiple operation requests.

[0130] It is understandable that the present application also defines a fast request pre-check mechanism that utilizes a multi-level Bloom filter (root path filter, directory filter, and lock type filter). Specifically: a three-level Bloom filter is used to pre-check the operation request to determine whether there is a need to acquire a lock. If, through the detection of the root path filter, directory filter, and lock type filter, it is determined that there is no lock contention related to the first operation request in multiple operation requests (that is, the path prefix, target file, or lock type of the first operation request is not marked in the Bloom filter), it can be quickly determined that these operation requests do not need to calculate the number of lock contentions, thereby directly allowing the client to execute. Through the above pre-check, if all three-level filters do not detect the existence of the lock related to the first operation request, then the client can be directly allowed to execute the second operation request without further calculating the number of lock contentions corresponding to the second operation request. This is because the pre-check mechanism of the Bloom filter can effectively filter out those operation requests that will not cause lock contention.

[0131] In order to better understand the process of the above-mentioned partition lock acquisition method, the implementation process of the above-mentioned partition lock acquisition method is described below in combination with an optional embodiment, but it is not used to limit the technical solution of the embodiment of this application.

[0132] Distributed storage systems aim to address the scalability, reliability, and performance bottlenecks of traditional storage architectures. Their core architecture integrates object storage, block storage, and file system functions, and uses the Controlled Replication Under Scalable Hashing (CRUSH) algorithm to achieve automatic data distribution and management without a central node. Through decentralized design, dynamic expansion mechanisms, multiple replicas, and erasure code fault-tolerance strategies, they balance high performance and high availability, and are widely used in cloud computing, big data, artificial intelligence (AI), and other fields.

[0133] MDS stems from the need for efficient metadata management in distributed file systems, aiming to address the performance bottlenecks and scalability issues of traditional centralized metadata servers. Its core technical architecture is based on the Reliable Autonomic Distributed Object Store (RADOS) underlying storage. It utilizes the CRUSH algorithm to achieve distributed metadata storage and automated management. It supports master-slave or active-active deployments with multiple MDS nodes to enhance the concurrent processing of metadata operations. MDS utilizes a hierarchical caching mechanism (e.g., memory cache and solid-state drive (SSD) cache) to accelerate metadata access. It also optimizes metadata operation efficiency for large numbers of small files through technologies such as directory sharding and transaction logging. Furthermore, it leverages consensus protocols such as the Paxos / Raft consensus algorithm (Paxos / Raft) to ensure strong consistency across metadata replicas. This enables dynamic scalability, high availability, and low-latency metadata management for massive file storage scenarios (e.g., AI training data and scientific research datasets), making it a key component for high-performance file storage in distributed file systems.

[0134] In a distributed storage file system, in a multi-threaded concurrent operation environment, existing technologies implement concurrency control through POSIX read-write locks or hardware atomic operations (for example, the x86 Architecture's LOCK ADD Instruction (x86's LOCK ADD for short)). However, this lacks a dynamic adaptation mechanism for distributed storage metadata lock management.

[0135] The following lock management mechanisms also exist in related technologies:

[0136] Fixed partition lock: However, the fixed partition lock solution still has the problem that when the operation involves multiple partitions, it is necessary to obtain locks for multiple partitions, which increases the probability of deadlock. In addition, the operation process is complicated and reduces the efficiency of the system.

[0137] State machine-based lock management: By defining multiple lock states (for example, read lock, write lock, exclusive lock, etc.) and state transition rules, a state machine is constructed to manage locks. When a client requests a lock, the MDS determines whether operations such as locking and lock state transition are allowed based on the state machine rules, thereby handling complex concurrent request scenarios. However, state machine-based lock management has a high state transition overhead. As concurrent requests increase, the state machine's state judgment and transition operations consume a large amount of system resources, resulting in increased operation delays and affecting system performance. Waiter queue management is also difficult: After a lock waiting request enters the waiter queue, the wake-up mechanism may trigger a "herd effect," where multiple waiting threads are awakened but only one can obtain the lock, resulting in resource waste and performance degradation.

[0138] Locking strategies bound to metadata structures: Different types of locks are set for different metadata structures (e.g., index nodes (Inodes) and directory entries (Dentries)). For example, Dentry uses device node name (DN) and directory version number (DentryVersion) locks, while Inode uses multiple other types of locks. Depending on the specific operation type (create, delete, modify, etc.), corresponding locks are applied to the relevant metadata structure. However, the above technical solutions suffer from high coupling of lock strategies: the lock strategies are closely dependent on the metadata structure, requiring complex operations (such as batch file deletion and directory movement) to simultaneously acquire multiple different types of locks, resulting in a cumbersome operation process and prone to lock conflicts. They also lack overall optimization: locks for different metadata structures are managed independently, making optimization difficult at the system level. This results in low efficiency when handling complex operations involving multiple metadata structures.

[0139] In addition, under the traditional architecture, file I / O requests initiated by the client must undergo complex metadata processing, specifically:

[0140] 1) The file system client sends a path resolution request to the MDS, and the MDS locates the corresponding inode metadata of the file by traversing the directory tree (Dentry);

[0141] 2) The client applies for lock resources from the MDS based on the operation type (read / write). The MDS determines whether to grant lock permissions based on the global lock or fixed partition lock policy. If the lock is successfully acquired, the client obtains the data location information and directly interacts with the OSD for data.

[0142] However, this process exposes significant bottlenecks in high-concurrency scenarios: global locks cause request serialization, fixed partition locks cannot dynamically adapt to the load, and each lock request requires traversing lock records to detect conflicts (time complexity O(n)), making MDS a key bottleneck in system performance.

[0143] Based on the above problems, an optional embodiment of the present application proposes a method for dynamically partitioning lock requests of a distributed storage metadata server (Metadata Server, MDS for short) using a multi-level Bloom filter, which relates to the field of distributed storage technology and is used to improve the concurrency performance and stability of distributed storage systems. Specifically: by constructing a three-level Bloom filter index (root path, directory, lock type) in the MDS metadata lock management module, a multi-layer fast pre-check of lock requests is achieved, and combined with a dynamic partition lock mechanism and cross-architecture atomic update technology, the problems of low concurrency performance and severe lock contention of traditional lock strategies are solved. The main method flow is as follows: First, a three-level Bloom filter is constructed, the root path filter (m1=65536, k1=4) filters invalid path prefixes, the directory filter (m2=32768, k2=3) detects active locks of the target file, and the lock type filter (m3=8192, k3=2) filters conflicts of the same type of locks. The three-step pre-check can filter out more than 30% of invalid requests. Secondly, a dynamic partition lock mechanism is adopted, and the path hash is initially divided into N areas (S=128 files / area), and the algorithm is used. ( ) Real-time adjustment of region size, high contention The lock request processing process was reduced to 32, and expanded to 1024 in low-contention situations. After the hotspot area was split, the number of contentions was reduced by 85%. Finally, the lock request processing process was optimized. Partition locks were located using path hashing. This combined with the x86 platform's LOCKADD instruction (a lock addition instruction provided by x86 architecture processors) enabled atomic counter updates, reducing latency from 200ns to 80ns. Other architectures are compatible through pthread_rwlock (threaded read-write lock). This solution increased lock request processing throughput by 200% and reduced average latency by 74%, effectively resolving the performance bottleneck of distributed storage metadata lock management.

[0144] Bloom filters, among other technologies, address the space and efficiency bottlenecks encountered when processing large amounts of data using set structures. By combining hash functions with bit arrays, they achieve efficient data filtering using probabilistic judgments, with a space complexity as low as zero. Their core applications include database cache penetration protection, spam interception, search engine Uniform Resource Locator (URL) deduplication, and blockchain transaction verification. They can store 100 million elements in 20MB of memory and enable rapid query. With their advantages in space and time efficiency, they have become a key technology for filtering massive amounts of data in the big data era.

[0145] The core value of applying Bloom filters to MDS locks lies in rapidly filtering invalid lock requests, reducing lock table query overhead, and lowering distributed synchronization costs. Despite limitations such as false positives and deletions, parameter tuning and a periodic rebuild mechanism can achieve a balance between performance and accuracy in real-world scenarios, making it a key optimization tool for metadata management in large-scale distributed file systems.

[0146] Figure 3 This is a diagram of a distributed storage system architecture according to an optional embodiment of the present application. Figure 3 As shown:

[0147] The distributed storage system uses a decentralized, distributed architecture. Its core components include: client, monitor, metadata server (MDS), and object storage device (OSD). Each module works together through a high-speed network (such as Ethernet and InfiniBand) to achieve data storage and management. The architecture layer is described as follows:

[0148] 1. Client layer (Client, i.e. Figure 3 Distributed storage client):

[0149] position:

[0150] The entry point for users to interact with distributed storage, providing three interfaces: block, object, and file.

[0151] Core components:

[0152] librados library (RADOS client library): encapsulates the underlying Application Programming Interface (API) and is responsible for interacting with the cluster.

[0153] CRUSH Client Module: This module caches the CRUSH map locally and calculates data storage locations. Inputs include the object name and storage pool rules (e.g., number of replicas, fault domain hierarchy). Processing steps: Generate a random number based on the object name hash. It traverses the CRUSH bucket hierarchy (e.g., host -> cabinet -> data center) according to the storage pool rules and selects OSDs based on weights. This ensures balanced data distribution and meets redundancy policies (e.g., deploying replicas across cabinets).

[0154] 2. Distributed storage system:

[0155] Includes: object gateway service that interacts with objects, block device service that interacts with blocks, and file system service that interacts with files.

[0156] 3. Control plane (Monitor / MDS):

[0157] Monitor (such as Figure 3 Monitoring service cluster 1…n): Maintain cluster metadata (OSD Map, CRUSH Map, authentication information), and ensure multi-node consistency through the Paxos algorithm.

[0158] Interaction with the client: The client pulls the latest cluster map when it first connects, and then updates it through heartbeat synchronization.

[0159] MDS (file scenarios, such as Figure 3 MDS metadata clusters 1…n): manage file system metadata (directory tree, permissions), cache hot metadata to accelerate access.

[0160] The distributed storage system in the optional embodiment of the present application is optimized for MDS file lock requests. Specifically, the MDS file lock requests are optimized through the following structure:

[0161] The MDS metadata lock management module, the three-level Bloom filter index module (including: root path filter, directory filter and lock type filter), the dynamic partition lock management module and the atomic operation adaptation module jointly execute the lock request processing process through the above modules.

[0162] 4. Data plane (OSD cluster): MDS metadata cluster 1…n.

[0163] OSD nodes: Hardware components include a central processing unit (CPU), memory, disks (hard disk drives (HDDs) / solid state drives (SSDs)), and network interfaces (gigabit / 10G network cards). They are responsible for data storage, replication, erasure coding, and consistency maintenance, and perceive cluster topology through the OSD map.

[0164] 5. Together with MDS metadata clusters 1…n and monitoring service clusters 1…n, it forms a unified, self-controlled, and scalable distributed storage consistency management system storage pool, including:

[0165] 1) Storage pool 1, storage pool 2, ..., storage pool n;

[0166] 2) Controllable and Scalable Distributed Data Balanced Placement Algorithm (CRUSH): Ensures that data is evenly distributed across the cluster while taking into account data redundancy and failure domains.

[0167] 3) Group 1, Group 2, Group 3, ..., Group n: Indicates that the physical storage devices are divided into multiple groups. Each group can contain multiple storage nodes to facilitate management and optimize data distribution.

[0168] 4) Storage backend engine;

[0169] 5) Node 1, Node 2, ..., Node n in OSD 1...n.

[0170] Figure 4 This is a flowchart of a distributed storage metadata lock optimization method based on a multi-level Bloom filter according to an optional embodiment of the present application, which can be applied to Figure 3 In the system, specifically, Figure 4 As shown:

[0171] Step S401: receiving a client request (e.g., a lock request for / data / file.txt (i.e., requesting to lock the / data / file.txt resource));

[0172] The client of the distributed storage system sends a file operation request to the MDS;

[0173] Step S402, MDS path resolution → top-level directory prefix / data;

[0174] Extract the top-level directory prefix from the request path (for example, / data in / data / file.txt).

[0175] In the distributed storage IO process, the file path requested by the client (such as / data / file.txt) must first be resolved to the top-level directory prefix (such as / data).

[0176] After parsing the top-level directory prefix, an optional embodiment of the present application constructs a three-level Bloom filter index of root path, directory, and lock type to implement multi-layer fast pre-check of lock requests. Among them, the root path filter quickly locates whether there is an active lock for the request path prefix; the directory filter screens whether there is a lock record for the first file in the current directory; and the lock type filter further filters conflicts with the same type of locks. That is, in the read and write IO link of the distributed storage file system, the three-level filter forms a layer-by-layer filtering mechanism of "path prefix → directory file → operation type". The three-step pre-check can filter out more than 30% of invalid requests and narrow the conflict detection range from the full amount of lock records to the extremely small set of Bloom filter hits. Specifically:

[0177] Step S403, root path filter check (m1=65536, k1=4);

[0178] The root path filter stores the top-level directory prefix corresponding to all active locks.

[0179] Root path Bloom filter: In the distributed storage IO process, the file path requested by the client (such as / cluster / data / file.txt) must first be resolved to the top-level directory prefix (such as / cluster). The root path Bloom filter stores the top-level directory prefixes corresponding to all active locks. Its parameter setting logic is as follows: Store the top-level directory path prefix, bit array size m1=65536, k1=4, through the formula Calculate and control the false positive rate p≤0.001 to quickly locate whether there is an active lock on the path prefix. The bit array size is set to m1=65536 and k1=4 according to the following method: Assuming that the MDS metadata service is expected to manage n=10,000 top-level directories, according to the Bloom filter false positive rate formula: (i.e. the first formula) requires controlling the false positive rate p≤0.001, combined with the formula for the optimal number of hash functions: (i.e. the second formula) The solution of the simultaneous equations is m1≈65536(2^16), k1≈4, and p1≈0.00098≤0.001, which meets the high reliability pre-inspection requirements.

[0180] The invalid paths filtered out by the root path filter refer to paths whose top-level directory prefixes do not have any active lock records in the current MDS metadata lock management context.

[0181] Step S404, determining whether there is an active lock prefix;

[0182] Use the root path Bloom filter (m1=65536, k1=4) to detect whether there is an active lock record associated with the top-level directory prefix.

[0183] If the root path filter indicates that there is no active lock associated with this top-level directory prefix (i.e., the first active lock), it means that the top-level path prefix currently has no lock contention, and the path resolution request can be directly released without entering the subsequent lock contention check (i.e., executing step S405).

[0184] If the root path filter may have an active lock (ie, the root path Bloom filter returns "true"), the next step is to check the directory Bloom filter (ie, execute step S406).

[0185] Active lock records refer to locks (such as read and write locks) currently held by clients in the MDS. The root path filter uses probabilistic mapping within the Bloom filter to quickly determine whether a path prefix belongs to the set of "possibly active locks." For example, a client requests access to / non_exist_dir / file.txt (where / non_exist_dir is a file in a non-existent directory and / file.txt is the file name being accessed within that directory path). However, the non_exist_dir directory has never been created in the MDS, so the root path filter has no mapping record for the top-level directory prefix / non_exist_dir. Alternatively, the directory / data has been created, but no clients currently hold any locks (read or write) on that directory or its subnodes. In this case, when a client requests a lock on / data / file.txt, the root path filter has no active lock record for the / data prefix.

[0186] In other words, the root path filter pre-check is triggered when: after the MDS completes the path resolution and enters the lock processing process for the first time;

[0187] The root path filter's technical logic: parses the path's top-level directory prefix (e.g., / data) and queries the root path's Bloom filter (m1=65536, k1=4). If no corresponding mapping exists, the path is deemed invalid and returned directly to the client (without subsequent lock contention), reducing invalid I / O requests by over 30%. The algorithm also reduces the false positive rate, ensuring that over 99.9% of invalid paths are correctly filtered.

[0188] Step S405: If the root path filter indicates that there is no active lock associated with the top-level directory prefix, the request is released and processed normally.

[0189] Step S406 , if the root path filter indicates that there is an active lock associated with this top-level directory prefix, the directory (Bloom) filter checks (m2=32768, k2=3);

[0190] When the root path filter check passes, continue to resolve the path to the specific directory (for example: / data).

[0191] Directory Bloom filter: Dynamic management of file name indexes. When the I / O process resolves to a specific directory (such as / cluster / data), the directory Bloom filter indexes the file names in that directory. Each directory has a separate filter with m2=32768 and k2=3. This filter stores the directory's file name index. A counting Bloom filter is used, with a 16-bit integer counter to support deletion. The settings of m2=32768 and k2=3 are based on the expected storage of n=5000 files per directory. Using the third and fourth formulas, we get m2=32768(2^15) and k2=3, where p≈0.0015≤0.002. A 16-bit counter array (counter range 0-65535) is used to decrement the counter when a file is deleted, preventing the accumulation of false positives.

[0192] Step S407 , checking whether there is an active lock (i.e., a second active lock) for the file file.txt (request file name);

[0193] Use the Bloom filter for the directory (m2=32768, k2=3) to check whether there are any active lock records related to the requested file name in the current directory;

[0194] If the directory Bloom filter indicates that there is no relevant active lock, it means that the first file currently has no lock contention, and the process of obtaining the directory-level lock can be skipped, and the client is directly allowed to perform the file operation (ie, execute step S408).

[0195] If the directory filter returns that there may be an active lock (ie, the directory Bloom filter hits), the lock type Bloom filter is checked to further narrow the conflict detection range (ie, execute step S409 ).

[0196] In other words, the directory filter is triggered when the root path filter passes and the target directory (such as / data) is located;

[0197] The technical logic of the directory filter: A directory Bloom filter (m2=32768, k2=3) is used to check whether there is an active lock record for the first file name (e.g., file.txt). If not, MDS directly allows the operation without acquiring a directory-level lock, saving 20-50μs of lock contention time.

[0198] Step S408, skip lock acquisition;

[0199] Step S409, lock type filter check (m3=8192, k3=2);

[0200] After the directory filter check passes, confirm the type of operation (read or write) requested by the client.

[0201] Fast screening of read-write lock conflicts. During the I / O operation type confirmation phase (e.g., read / write), the lock type filter subdivides the index by operation type, specifically by read / write lock, with m3 = 8192 and k3 = 2, accelerating detection of conflicts for the same type of lock. In an optional embodiment of this application, for both read / write lock operations, n3 = 2000 times / second, m3 = 8192 (2^13), k3 = 2, and p ≈ 0.008 are used to meet the requirements for fast conflict screening.

[0202] Step S410, determining whether there is a conflict between locks of the same type (read locks or write locks);

[0203] After the directory filter passes, the corresponding lock type filter is queried based on the operation type (read / write). Conflict detection is performed only on the matching lock records, reducing the O(n) complexity to O(1). This means using a dedicated lock type Bloom filter (m3=8192, k3=2) to check for conflicts with the same type of lock (read lock or write lock).

[0204] If the lock type filter indicates that there is no lock conflict of the same type, the conflict detection range can be further narrowed down to only perform conflict detection on the records that hit the Bloom filter, rather than traversing the entire lock record set (ie, executing step S411 ).

[0205] If the lock type filter is hit, it indicates that there may be a lock conflict of the same type, and it is necessary to enter the lock conflict detection phase (ie, execute step S412).

[0206] In other words, the lock type filter is triggered after the operation type (read / write) is determined. The technical logic is to query the corresponding lock type filter (m3=8192, k3=2) according to the operation type (such as write lock), and only perform conflict detection on the matched lock records, reducing the number of lock records to be traversed from n to n×p (p≤0.008), and reducing the conflict detection time from O(n) to O(1).

[0207] Through the above three-level lock filtering mechanism, the lock request processing time is reduced from 300ms in the traditional solution to approximately 120ms, and the number of hardware bus contention times is reduced by 75%, fundamentally optimizing the metadata lock bottleneck in the distributed storage IO process.

[0208] Step S411, allowing the operation;

[0209] Step S412, dynamic partition lock acquisition: path hash positioning area;

[0210] The core value of dynamic partition locks lies in upgrading the "static partitioning" of traditional fixed partition locks to "load-adaptive partitioning" to solve the competition problem in hot spots during peak IO periods.

[0211] The specific steps for dynamic partition lock area division include:

[0212] 1) Initial Partitioning: Based on a balanced partitioning strategy based on path hashing, a hash operation (e.g., MD4) is performed on the file paths within a directory. The hash values ​​are then mapped modulo-wise to N regions, each initially containing S = 128 files. For example, if a directory contains 10,000 files, the initial partitioning is N = 10,000 / 128 ≈ 78 regions. Coupled with the I / O process, when a client requests / data / file.txt, the MDS calculates the hash value of file.txt, modulo-wise calculates the region index, and directly locates the corresponding partition lock, avoiding global lock contention. When a client sends a file operation request (e.g., write / data / file.txt) to the MDS, the MDS resolves the path and pre-checks it through a three-level Bloom filter (root path → directory → lock type). If a match is found, the initial partitioning phase begins. At this point, the MDS performs a hash operation (e.g., MD4) on the file paths within the target directory and maps the hash values ​​modulo-wise to N regions, each initially containing S = 128 files. Each region has an independent read-write lock.

[0213] 2) Dynamic Adjustment Mechanism: Based on adaptive optimization of the number of contention times, the number of lock contentions C in each region is regularly counted, and the region size is adjusted according to the formula: .

[0214] Where S is the current region size (the number of files contained in each region); The initial region size (128) is set to 128 because it needs to match the CPU L2 cache line size (128KiB). Each region occupies about 128×2B (counter) = 256B of memory. The 64 regions total 16KiB, which can be fully resident in the L1 cache to reduce memory access latency. α=0.3 is the adjustment coefficient, which controls the adjustment sensitivity and avoids frequent oscillations. It is set to 0.3 because this coefficient controls the region size adjustment range to ±30% / time, which avoids too slow adjustment ( Too small) causes hotspot accumulation, and prevents too fast adjustment (α is too large) causing lock structure reconstruction overhead; C is the number of lock contentions in the current area (times / second); C max The historical maximum number of contention times (peak value in the past 10 minutes). When C>0.8C maxWhen C<0.2C, it is a high competition, S is reduced to 1 / 2 of the original (minimum 32) to split the hot spot area to reduce competition; when C<0.2C max When the contention is low, S is expanded to 2 times (maximum 1024) and cold regions are merged to reduce lock management overhead.

[0215] Trigger timing in the I / O process: The number of lock contentions in each region is counted every 100ms (obtained through the wait_count parameter of pthread_rwlock), triggering dynamic adjustment to ensure a fast response when the I / O load suddenly changes (such as batch file creation).

[0216] 3) Hotspot area splitting: fine-grained unlocking contention, when the number of regional contentions exceeds 100 times / second (experimentally measured bus conflict rate exceeds 70% under this threshold), the splitting operation is triggered, and the splitting process is: suspend new lock requests for the area; remap the lock records of the original area to 2 sub-areas according to the hash value; allocate independent read-write locks to each sub-area; resume lock requests, and new requests are allocated according to the sub-area hash. The optional embodiment of the present application takes into account the compatibility of the locks, and proposes that during the splitting process, the IO requests being processed are not affected, and only new requests are processed according to the new partitioning rules to avoid IO interruptions. Among them, based on the optional embodiment of the present application, the threshold is selected as when the number of contentions exceeds 100 times / second because the bus arbitration delay increases from 50ns to 200ns, and triggering the split can reduce the delay to less than 80ns.

[0217] In summary, the optional embodiment of the present application introduces a dynamic partition lock mechanism based on the number of competitions, and the optional embodiment of the present application proposes an algorithm formula (Formula 7) Real-time adjustment of the lock area size, hotspot area (C>0.8C max ) Automatic splitting, cold area (C<0.2C max ) Dynamic merging ensures that the lock granularity is precisely matched to the load.

[0218] This involves traversing potentially conflicting lock records identified by the Bloom filter and checking their actual lock status. The lock zone to which the file belongs is determined based on the path hash, and a lock for the corresponding partition is acquired. Read operations use shared locks, while write operations require exclusive locks.

[0219] In other words, partition lock acquisition is to locate the lock area according to the path hash and obtain the corresponding read-write lock (shared lock for read operation and exclusive lock for write operation).

[0220] Partition lock acquisition is triggered when all three levels of filtering are successful, and then the actual lock contention phase begins.

[0221] Technical logic: Calculate the hash value of the target file path and locate the lock area according to the dynamic partitioning rule (current area size S); obtain a shared lock for read operations (allowing multi-thread concurrency), and obtain an exclusive lock for write operations (single-thread exclusive); if the lock acquisition fails (conflict), enter the waiter (waiting thread) queue to wait, avoiding CPU spin waste; related to the performance of the IO process: dynamic partitioning reduces the number of lock contention times from 10^6 times / second to 1.5×10^5 times / second in the 8-thread scenario, and increases IO throughput by 316% (for example, from 1200ops / s to 3800ops / s).

[0222] Through the process steps of step S412 above, high concurrency scenarios are optimized. When the IO load suddenly increases (such as 100 clients operating the same directory at the same time), the dynamic partition lock quickly splits the region size from 128 to 32, reducing the number of lock contentions from 800 times / second to 200 times / second, and increasing IO throughput by 210%. Improved resource utilization: Under low load, the region size is expanded to 1024, the number of locks is reduced from 78 to 10, and the lock management overhead is reduced from 35% of the IO time to 8%, freeing up CPU resources to handle actual data transmission. Hardware affinity optimization: The dynamically adjusted region size is always a power of 2 (32 / 64 / 128 / 1024), matching the memory paging mechanism (4KiB pages), reducing memory fragmentation and cross-page access.

[0223] In summary, by deeply integrating the dynamic partition lock into the file IO process in the above-mentioned step S412, the optional embodiment of the present application realizes the "load perception-dynamic adaptation-performance optimization" closed loop of metadata lock management, fundamentally solving the hot spot competition and resource waste problems of fixed partition locks in related technologies.

[0224] Step S413, conflict detection: traverse the Bloom filter hit records;

[0225] Conflict detection is to filter out potentially conflicting lock records through a Bloom filter and only traverse the records that hit the Bloom filter, reducing the conflict detection time from O(n) to O(1).

[0226] The conflict detection is triggered after acquiring the partition lock and before executing specific operations.

[0227] Technical Logic: The lock records that hit the Bloom filter only account for p (≤0.008) of the total number of records, significantly reducing the traversal range. The lock range (such as byte-level lock) of the hit records is checked for conflicts. Only after confirming that there are no conflicts, the I / O request is released.

[0228] Time consumption comparison: The traditional solution takes 300 μs to traverse 1000 lock records, while this solution only traverses 8 records (1000 × 0.008), reducing the time consumption to 20 μs and reducing the I / O latency from 359 ms to less than 120 ms.

[0229] Step S414 , atomic operation updates the counter (x86: LOCK ADD).

[0230] For the Bloom filter counter update process, we combine hardware atomic operations (for example, the LOCK ADD instruction on the x86 platform) with a lock protection mechanism (mutex lock on the ARM platform) to achieve efficient cross-architecture compatible updates, reducing the counter update latency from 200ns to 80ns.

[0231] The specific steps of atomic operation to update the counter include:

[0232] 1) Architecture Adaptation and Update Logic Triggering Timing: After a lock is successfully acquired, before an I / O operation is performed (adding a lock) or after it is completed (releasing a lock). Technical Implementation: x86 architecture: Use the LOCK ADD instruction to atomically update the Bloom filter counter, for example: LOCK ADD DWORD PTR [counter], 1; atomically increment the counter.

[0233] Single update latency is reduced from 200ns to 80ns, reducing bus contention by 60%. Other architectures: Counter updates are protected by pthread_rwlock to ensure cross-platform compatibility. Update logic: When adding a lock, the corresponding hash position counter is incremented by 1 to record the active lock status; when releasing a lock, the counter is decremented by 1 (if ≥ 1) to avoid false positive accumulation.

[0234] 2) Technical basis for the parameters of the embodiment, Bloom filter parameters (when n=10000): m=65536, k=4, the false positive rate at this time , meeting data consistency requirements; the 16-bit counter supports a maximum count of 65535, which is suitable for high-frequency IO scenarios (such as 100,000 operations per second can last for 6553 seconds). Dynamic adjustment parameters: C max The maximum number of contentions in the past 10 minutes is used to adapt to periodic fluctuations in I / O load (such as weekday peaks and nighttime troughs). α=0.3 ensures moderate region size adjustments and avoids jitter in I / O processes caused by frequent partition reconstruction.

[0235] That is, the IO processing flow of the distributed storage system of the optional embodiment of the present application can be divided into the following core stages: client request stage, the client sends a file operation request (such as write / data / file.txt) to the MDS; path resolution stage, the MDS resolves the path and locates the target directory (such as / data); metadata lock processing stage: three-level Bloom filter pre-check; dynamic partition lock acquisition; conflict detection and counter update; data access stage: MDS returns the data location, and the client interacts with the OSD; lock release stage: the lock is released after the operation is completed, and the Bloom filter counter is updated.

[0236] In summary, the optional embodiments of the present application transform the metadata lock bottleneck in the traditional IO process into a performance acceleration point, which is particularly suitable for high-frequency IO scenarios such as big data analysis and cloud storage, and provides core technical support for the large-scale deployment of distributed storage systems. Invalid request filtering chain: Three-level Bloom filtering intercepts 30% of invalid IO requests, reducing MDS computing power consumption; Dynamic lock granularity adaptation: Adjust the partition size in real time according to the IO load, split hot areas during high concurrency, and merge cold areas during low load, increasing hardware resource utilization by 85%; Atomic update and conflict detection optimization: Counter update delay is reduced by 60%, conflict detection time is shortened by 93%, and end-to-end IO delay is reduced from 359ms to 80-120ms; Cross-architecture compatibility: Automatically adapt hardware characteristics through compile-time macro definitions to ensure performance consistency in different IO scenarios.

[0237] The optional embodiment of the present application can also extend the three-level Bloom filter architecture to edge distributed storage scenarios, combine the lightweight characteristics of edge nodes to optimize parameters (such as reducing the bit array to 8KB), and solve the edge device metadata lock conflict problem; combine with the AI ​​load prediction algorithm, and adjust the lock area division threshold in advance through the historical competition data training model to achieve "prediction-adaptation" dual-driven optimization; integrate the characteristics of new storage media (such as 3DX Point (three-dimensional cross-point memory)), reconstruct the counter atomic operation logic based on its low latency advantage, and further reduce the update delay to 50ns; expand to cross-cloud storage environments, and dynamically adjust the cross-region lock synchronization strategy through the Bloom filter hash collision probability to reduce the cross-data center lock conflict overhead; develop a Bloom filter status visualization monitoring module to display the false positive rate and lock area competition heat map of each level in real time to assist in operation and maintenance optimization.

[0238] In addition, an optional embodiment of the present application further provides a computer-readable storage medium, and the above-mentioned method according to the optional embodiment of the present application can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0239] Through the optional embodiments of this application, performance can be improved through throughput optimization: In tests on an Intel Xeon 8-core platform, lock operation throughput under 8 threads increased from 1200 ops / s to 3800 ops / s, a 316% increase, approaching the 380% speedup of the atomic operation solution described in the paper. Latency is reduced: The average latency of lock requests is reduced from 359ms to 80-120ms, with a 72% reduction in latency from the journaled_completion_queued to sub_op_committed phase (from logging completion queue to sub-op submission confirmation), consistent with the partitioned lock optimization results described in the paper. Lock contention is reduced: Dynamic partitioning reduces contention in hotspots by 85%, with performance degradation of <8% at 128 elements / lock, validating the relationship between partitioned lock granularity and performance described in the paper. Memory usage is reduced: Multi-level Bloom filter compression reduces memory usage from 1.8GB to 0.65GB for 10 million files, saving 64%. Hardware Resource Utilization: Atomic operations reduce counter update latency from 200ns to 80ns (x86 architecture), reducing bus contention and aligning with the paper's conclusions on the impact of hardware shared resources. Anti-blocking Capability: Long transaction locks are automatically migrated to independent threads to avoid global MDS blocking. Dynamic adjustment mechanisms achieve optimization within 100ms even when load fluctuations are ±50%.

[0240] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0241] This embodiment also provides a partition lock acquisition device, which is used to implement the above embodiments and preferred implementations. Details that have already been described will not be repeated. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

[0242] Figure 5 is a framework diagram of a partition lock acquisition device according to an embodiment of the present application, such as Figure 5 As shown, the device includes:

[0243] a pre-check module 52 configured to, upon receiving multiple operation requests from one or more clients, pre-check the multiple operation requests using a filter index to filter a first operation request from the multiple operation requests to obtain a second operation request, wherein the filter index includes at least one of the following: a root path filter, a directory filter, and a lock type filter; the first operation request is an operation request that does not require locking and / or does not cause lock contention; and the second operation request is an operation request that causes lock contention;

[0244] a calculation module 54 configured to calculate the number of lock contentions corresponding to the second operation request, and determine a target lock region corresponding to the second operation request according to the number of lock contentions;

[0245] The acquisition module 56 is configured to acquire the partition lock corresponding to the second operation request according to the target lock area.

[0246] Through the partition lock acquisition device of the present application, when multiple operation requests are received from one or more clients, the multiple operation requests are pre-checked through the root path filter, directory filter, and lock type filter to filter the first operation request that does not require locking and / or does not cause lock contention, and obtain the second operation request that causes lock contention; the number of lock contentions corresponding to the second operation request is calculated, and the target lock area corresponding to the second operation request is determined based on the number of lock contentions to reduce the number of lock contentions in hot spots. Through this application, the problem of serious lock contention in lock management mechanisms such as global lock mechanisms and fixed partition locks in related technologies is solved, thereby reducing lock contention in hot spots.

[0247] In an exemplary embodiment, the pre-check module 52 is also used to parse the multiple operation requests to obtain a first request path corresponding to each operation request; extract the top-level directory prefix contained in each first request path, and perform a first-level pre-check on each operation request based on the root path filter to determine whether there is a first active lock operation under each top-level directory prefix, wherein the top-level directory prefix is ​​used to indicate the highest-level directory in the first request directory corresponding to each first request path; when it is determined that the first active lock operation exists under the top-level directory prefix corresponding to the third operation request in the multiple operation requests, filter the third operation request, and pre-check the fourth operation request in the multiple operation requests except the third operation request through the directory filter and / or the lock type filter, wherein the first operation request includes: the third operation request.

[0248] In an exemplary embodiment, the pre-check module 52 is further configured to obtain a first number of top-level directories expected to be stored in the root path filter and a first false positive rate corresponding to the root path filter; and calculate a first digit group size of a first digit group included in the root path filter and a second number of a first hash function used to map the top-level directory prefix to the first digit group according to a first formula and a second formula, wherein the first formula is: , the second formula is: , p1 is the first false positive rate, k1 is the second number, m1 is the first digit group size, and n1 is the first number; calculate the first hash value corresponding to each top-level directory prefix based on the first hash function of the second number; determine the first index value corresponding to the first hash value according to the first digit group of the first digit group size; determine the first bit identifier of the first index position corresponding to the first index value in the root path filter; determine whether the active lock operation exists under each top-level directory prefix according to the first bit identifier.

[0249] In an exemplary embodiment, the pre-check module 52 is also used to parse each fourth operation request to determine the second request path corresponding to the fourth operation request; determine the second request directory corresponding to each second request path, and perform a secondary pre-check on the first files contained in each second request directory based on each directory filter corresponding to each second request directory to determine whether there is a second active lock record under each first file; when it is determined that the second active lock record exists under the first file corresponding to the fifth operation request in multiple fourth operation requests, filter the fifth operation request, and pre-check the sixth operation request in the multiple fourth operation requests except the fifth operation request through the lock type filter, wherein the first operation request includes: the fifth operation request.

[0250] In an exemplary embodiment, the pre-check module 52 is further configured to obtain a third number of files expected to be stored in each second requested directory corresponding to each directory filter and a second false positive rate corresponding to each directory filter; and calculate a second digit group size of the second digit group included in each directory filter and a fourth number of the second hash function used to map the first file information corresponding to each first file to the second digit group according to a third formula and a fourth formula, wherein the third formula is: , the fourth formula is: , p2 is the second false positive rate, k2 is the fourth number, m2 is the second digit group size, and n2 is the third number; performing hash calculation on each first file information based on a second hash function of the fourth number to obtain a second hash value corresponding to each first file information, wherein the first file information includes at least one of the following: the first file name of the first file, the first file identifier of the first file, and the first metadata corresponding to the first file; determining a second index value corresponding to the second hash value according to a second digit group of the second digit group size; determining a second bit identifier of a second index position corresponding to each second index value in each directory filter; and determining whether the active lock operation exists under each first file according to the second bit identifier.

[0251] In an exemplary embodiment, the pre-check module 52 is also used to parse the sixth operation request to determine the request type corresponding to the sixth operation request; determine the target type corresponding to the lock type filter corresponding to each sixth operation request according to the request type, and perform a three-level pre-check on each sixth operation request based on the lock type filter of the target type to determine whether there is a same-type lock conflict for the sixth operation request of each request type, wherein the lock type filter of the target type includes at least one of the following: a write lock type filter, a read lock type filter, and the same-type lock conflict is used to indicate that the lock operations corresponding to multiple sixth operation requests of any request type are operations requesting to obtain the same type of lock at the same time; when it is determined that multiple seventh operation requests among multiple sixth operation requests have the same-type lock conflict, filter the multiple seventh operation requests to obtain the second operation request, wherein the first operation request includes: the multiple seventh operation requests.

[0252] In an exemplary embodiment, the pre-check module 52 is further configured to obtain the number of lock operations of the target type expected to occur per second by the lock type filter of the target type and a third false positive rate corresponding to the lock type filter of the target type; and calculate the third digit group size of the third digit group included in the lock type filter of the target type and the fifth number of the third hash function used to map the request type to the third digit group according to the fifth formula and the sixth formula, wherein the fifth formula is , the sixth formula is: , p3 is the third false positive rate, k3 is the fifth number, m2 is the third digit group size, and n3 is the number of times; a third hash function based on the fifth number is used to perform hash calculation on the request type of each sixth operation request to obtain a third hash value corresponding to each request type; a third index value corresponding to the third hash value is determined according to a third digit group of the third digit group size; a third bit identifier of a third index position corresponding to the third index value is determined in a lock type filter of the target type; and whether there is a lock conflict of the same type in the sixth operation request of each request type according to the third bit identifier.

[0253] In an exemplary embodiment, the pre-check module 52 is further used to, upon receiving a first lock request corresponding to a first top-level directory stored in the root path filter, count based on a first counter in the root path filter to increase the value of the first counter to a first value, wherein the first counter is a counter corresponding to the first top-level directory; upon determining that a second lock request corresponding to a second top-level directory stored in the root path filter has been deleted, count based on a second counter in the root path filter to reduce the value of the second counter to a second value, wherein the second counter is a counter corresponding to the second top-level directory; upon determining that a first file lock request under a request directory corresponding to the first directory filter has been received, count based on a third counter in the first directory filter to increase the value of the third counter to a third value, wherein the third counter is an indication of the first file lock request. a counter corresponding to the file; in a case where it is determined that the second file lock request under the request directory corresponding to the second directory filter has been deleted, counting is performed based on the fourth counter in the second directory filter to increase the value of the fourth counter to a fourth value, wherein the fourth counter is the counter corresponding to the file indicated by the second file lock request, and the directory filter includes: the first directory filter and the second directory filter; in a case where it is determined that a first lock request of a first request type is received, counting is performed based on the fifth counter in the lock type filter corresponding to the first request type to increase the value of the fifth counter to a fifth value; in a case where it is determined that the second lock request of the second request type has been deleted, counting is performed based on the sixth counter in the lock type filter corresponding to the second request type to reduce the value of the sixth counter to a sixth value, wherein the first request type and the second request type are both: read lock request or write lock request.

[0254] In an exemplary embodiment, the calculation module 54 is also used to parse the second operation request to determine that the second operation request corresponds to a third request path; perform a hash operation on the third request path to determine a fourth hash value corresponding to the third request path; perform a modulo operation on the fourth hash value to determine the first area index corresponding to the fourth hash value, and determine the first initial lock area corresponding to the third request path based on the first area index; count the number of lock contentions contained in the first initial lock area based on a preset time interval, and determine the number of lock contentions contained in the first initial lock area as the number of lock contentions corresponding to the second operation request.

[0255] In an exemplary embodiment, the calculation module 54 is also used to determine the historical maximum number of lock contentions in the first initial lock area corresponding to the second operation request within the target time period before the current time; adjust the first area size corresponding to the first initial lock area according to the number of lock contentions and the historical maximum number of lock contentions, and determine the adjusted first initial lock area as the target lock area.

[0256] In an exemplary embodiment, the calculation module 54 is further configured to adjust the size of the first region using a seventh formula, wherein the seventh formula is: , S is the second region size obtained by adjusting the first region size, S0 is the first region size, C max is the historical maximum lock contention number, C is the lock contention number, and α is the adjustment coefficient.

[0257] In an exemplary embodiment, the calculation module 54 is also used to determine the historical maximum lock contention number in the first initial lock area corresponding to the second operation request within the target time period before the current time; determine a first size relationship between the lock contention number and the historical maximum lock contention number; when the first size relationship indicates that there is a first lock contention number less than or equal to the historical maximum lock contention number among the multiple lock contention numbers corresponding to multiple second operation requests, determine the second initial lock area corresponding to the first lock contention number as the first low contention area; merge the first low contention area based on preset rules to generate a target lock area corresponding to the first lock contention number.

[0258] In an exemplary embodiment, the calculation module 54 is also used to determine a second size relationship between the number of lock contention times and a first preset threshold; when the second size relationship indicates that there is a second lock contention number among the multiple lock contention times corresponding to multiple second operation requests that is less than or equal to the first preset threshold, the third initial lock area corresponding to the second lock contention number is determined as a second low contention area; the second low contention area is merged based on a preset rule to generate a target lock area corresponding to the second lock contention number.

[0259] In an exemplary embodiment, the calculation module 54 is also used to determine the historical maximum lock contention number in the first initial lock area corresponding to the second operation request within the target time period before the current time; determine a first size relationship between the lock contention number and the historical maximum lock contention number; when the first size relationship indicates that there is a third lock contention number greater than the historical maximum lock contention number among the multiple lock contention numbers corresponding to multiple second operation requests, the fourth initial lock area corresponding to the third lock contention number is determined as the first high contention area, and the reception of lock requests for the first high contention area is suspended; the lock records contained in the first high contention area are allocated to multiple first sub-lock areas, wherein the target lock area corresponding to the third lock contention number includes: any first sub-lock area.

[0260] In an exemplary embodiment, the calculation module 54 is also used to determine a third size relationship between the number of lock contention times and a second preset threshold; when the third size relationship indicates that there is a fourth lock contention number greater than the second preset threshold among the multiple lock contention times corresponding to multiple second operation requests, the fifth initial lock area corresponding to the fourth lock contention number is determined as the second high contention area, and the reception of lock requests for the second high contention area is suspended; the lock records contained in the second high contention area are allocated to multiple second sub-lock areas, wherein the target lock area corresponding to the fourth lock contention number includes: any second sub-lock area.

[0261] In an exemplary embodiment, the pre-check module 52 is also used to prohibit calculating the number of lock contentions corresponding to the second operation request and determine whether the one or more clients are allowed to execute the multiple operation requests when it is determined based on the root path filter, directory filter and lock type filter that the first operation request does not exist in the multiple operation requests.

[0262] For the description of the features in the embodiment corresponding to the partition lock acquisition device, please refer to the relevant description of the embodiment corresponding to the partition lock acquisition method, which will not be repeated here.

[0263] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned partition lock acquisition method embodiments.

[0264] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above-mentioned partition lock acquisition method embodiments when running.

[0265] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0266] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the computer program implements the steps of any of the above-mentioned partition lock acquisition method embodiments.

[0267] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned partition lock acquisition method embodiments are implemented.

[0268] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0269] The above is a detailed introduction to the acquisition of a partition lock provided by this application. This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

Claims

1. A method for acquiring a partition lock, characterized in that: include: When receiving multiple operation requests sent by one or more clients, pre-checking the multiple operation requests by using a filter index to filter a first operation request from the multiple operation requests to obtain a second operation request, wherein the filter index includes at least one of the following: a root path filter, a directory filter, and a lock type filter, the first operation request is an operation request that does not require locking and / or does not cause lock contention, and the second operation request is an operation request that causes lock contention; Calculating the number of lock contentions corresponding to the second operation request, and determining a target lock region corresponding to the second operation request according to the number of lock contentions; A partition lock corresponding to the second operation request is acquired according to the target lock area.

2. The method for obtaining a partition lock according to claim 1, wherein: Pre-checking the multiple operation requests using a filter index includes: Parsing the multiple operation requests to obtain a first request path corresponding to each operation request; Extracting a top-level directory prefix included in each first request path, and performing a first-level pre-check on each operation request based on the root path filter to determine whether there is a first active lock operation under each top-level directory prefix, wherein the top-level directory prefix is ​​used to indicate the highest-level directory in the first request directory corresponding to each first request path; When it is determined that the first active lock operation exists under the top-level directory prefix corresponding to the third operation request among the multiple operation requests, the third operation request is filtered, and the fourth operation request other than the third operation request among the multiple operation requests is pre-checked through the directory filter and / or the lock type filter, wherein the first operation request includes: the third operation request.

3. The method for obtaining a partition lock according to claim 2, wherein: Performing a first-level pre-check on each operation request based on the root path filter to determine whether there is a first active lock operation under each top-level directory prefix includes: Obtaining a first number of top-level directories expected to be stored by the root path filter and a first false positive rate corresponding to the root path filter; A first byte size of a first byte group included in the root path filter and a second number of a first hash function for mapping the top-level directory prefix to the first byte group are calculated according to a first formula and a second formula, wherein the first formula is: , the second formula is: , p1 is the first false positive rate, k1 is the second number, m1 is the first digit group size, and n1 is the first number; Calculate a first hash value corresponding to each top-level directory prefix based on a second number of first hash functions; Determine a first index value corresponding to the first hash value according to a first digit group of the first digit group size; Determining, in the root path filter, a first bit identifier of a first index position corresponding to the first index value; It is determined whether the active lock operation exists under each top-level directory prefix according to the first bit flag.

4. The method for obtaining a partition lock according to claim 2, wherein: Pre-checking a fourth operation request other than the third operation request among the multiple operation requests by using the directory filter and / or the lock type filter includes: parsing each fourth operation request to determine a second request path corresponding to the fourth operation request; Determine a second request directory corresponding to each second request path, and perform a secondary pre-check on the first files contained in each second request directory based on each directory filter corresponding to each second request directory to determine whether a second active lock record exists under each first file; When it is determined that the second active lock record exists under the first file corresponding to the fifth operation request among multiple fourth operation requests, the fifth operation request is filtered, and the sixth operation request among the multiple fourth operation requests except the fifth operation request is pre-checked through the lock type filter, wherein the first operation request includes: the fifth operation request.

5. The method for obtaining a partition lock according to claim 4, wherein: Performing a secondary pre-check on the first files contained in each second request directory based on each directory filter corresponding to each second request directory to determine whether a second active lock record exists under each first file, including: Obtaining a third number of files expected to be stored in each second requested directory corresponding to each directory filter and a second false positive rate corresponding to each directory filter; The second byte size of the second byte group included in each directory filter and the fourth quantity of the second hash function used to map the first file information corresponding to each first file to the second byte group are calculated according to the third formula and the fourth formula, wherein the third formula is: , the fourth formula is: , p2 is the second false positive rate, k2 is the fourth number, m2 is the second digit group size, and n2 is the third number; performing a hash calculation on each piece of first file information based on a fourth number of second hash functions to obtain a second hash value corresponding to each piece of first file information, where the first file information includes at least one of the following: a first file name of the first file, a first file identifier of the first file, and first metadata corresponding to the first file; Determine a second index value corresponding to the second hash value according to a second digit group of a second digit group size; Determining, in each directory filter, a second bit identifier of a second index position corresponding to each second index value; Determine whether the active lock operation exists under each first file according to the second bit flag.

6. The method for obtaining a partition lock according to claim 4, wherein: Pre-checking a sixth operation request other than the fifth operation request among the plurality of fourth operation requests by using the lock type filter includes: Parsing the sixth operation request to determine a request type corresponding to the sixth operation request; Determine, according to the request type, a target type corresponding to a lock type filter corresponding to each sixth operation request, and perform a three-level pre-check on each sixth operation request based on the lock type filter of the target type to determine whether there is a same-type lock conflict for the sixth operation request of each request type, wherein the lock type filter of the target type includes at least one of the following: a write lock type filter and a read lock type filter, and the same-type lock conflict is used to indicate that the lock operations corresponding to multiple sixth operation requests of any request type are operations requesting to obtain the same type of lock at the same time; When it is determined that multiple seventh operation requests among the multiple sixth operation requests have the same type of lock conflicts, the multiple seventh operation requests are filtered to obtain the second operation request, wherein the first operation request includes: the multiple seventh operation requests.

7. The method for obtaining a partition lock according to claim 6, wherein: A three-level pre-check is performed on each sixth operation request based on the lock type filter of the target type to determine whether the sixth operation request of each request type has a lock conflict of the same type, including: Obtaining the number of lock operations of the target type expected to occur per second by the lock type filter of the target type and a third false positive rate corresponding to the lock type filter of the target type; The third byte size of the third byte group included in the lock type filter of the target type and the fifth number of the third hash function used to map the request type to the third byte group are calculated according to the fifth formula and the sixth formula, wherein the fifth formula is , the sixth formula is: , p3 is the third false positive rate, k3 is the fifth number, m2 is the third digit group size, and n3 is the number of times; performing a hash calculation on the request type of each sixth operation request based on the fifth number of third hash functions to obtain a third hash value corresponding to each request type; Determine a third index value corresponding to the third hash value according to a third digit group of a third digit group size; Determining, in the lock type filter of the target type, a third bit identifier of a third index position corresponding to the third index value; It is determined according to the third bit flag whether there is a lock conflict of the same type in the sixth operation request of each request type.

8. The method for obtaining a partition lock according to claim 1, wherein: Before or after pre-checking the plurality of operation requests using the filter index, the method further includes: Upon receiving a first lock request corresponding to a first top-level directory stored in the root path filter, counting based on a first counter in the root path filter to increase a value of the first counter by a first value, wherein the first counter is a counter corresponding to the first top-level directory; When it is determined that the second lock request corresponding to the second top-level directory stored in the root path filter has been deleted, counting based on a second counter in the root path filter to reduce the value of the second counter to a second value, wherein the second counter is a counter corresponding to the second top-level directory; Upon determining that a first file lock request is received under a request directory corresponding to a first directory filter, counting based on a third counter in the first directory filter to increase a value of the third counter to a third value, wherein the third counter is a counter corresponding to the file indicated by the first file lock request; When it is determined that the second file lock request under the request directory corresponding to the second directory filter has been deleted, counting based on a fourth counter in the second directory filter to increase the value of the fourth counter to a fourth value, wherein the fourth counter is a counter corresponding to the file indicated by the second file lock request, and the directory filter includes: the first directory filter and the second directory filter; When it is determined that a first lock request of a first request type is received, counting based on a fifth counter in a lock type filter corresponding to the first request type to increase a value of the fifth counter to a fifth value; When it is determined that the second lock request of the second request type has been deleted, counting is performed based on the sixth counter in the lock type filter corresponding to the second request type to reduce the value of the sixth counter to a sixth value, wherein the first request type and the second request type are both: read lock request or write lock request.

9. The method for obtaining a partition lock according to claim 1, wherein: Calculating the number of lock contentions corresponding to the second operation request includes: Parsing the second operation request to determine that the second operation request corresponds to a third request path; Performing a hash operation on the third request path to determine a fourth hash value corresponding to the third request path; performing a modulo operation on the fourth hash value to determine a first region index corresponding to the fourth hash value, and determining a first initial lock region corresponding to the third request path based on the first region index; The number of lock contentions contained in the first initial lock region is counted based on a preset time interval, and the number of lock contentions contained in the first initial lock region is determined as the number of lock contentions corresponding to the second operation request.

10. The method for obtaining a partition lock according to claim 1, wherein: Determining a target lock region corresponding to the second operation request according to the number of lock contention times includes: Determine a historical maximum number of lock contentions in the first initial lock region corresponding to the second operation request within a target time period before a current time; The first region size corresponding to the first initial lock region is adjusted according to the number of lock contentions and the historical maximum number of lock contentions, and the adjusted first initial lock region is determined as the target lock region.

11. The method for obtaining a partition lock according to claim 10, wherein: Adjusting the size of the first region corresponding to the first initial lock region according to the number of lock contention times and the historical maximum number of lock contention times includes: The size of the first region is adjusted using the seventh formula, wherein the seventh formula is: , S is the second region size obtained by adjusting the first region size, S0 is the first region size, C max is the historical maximum lock contention number, C is the lock contention number, and α is the adjustment coefficient.

12. The method for obtaining a partition lock according to claim 1, wherein: Determining a target lock region corresponding to the second operation request according to the number of lock contention times further includes: Determine a historical maximum number of lock contentions in the first initial lock region corresponding to the second operation request within a target time period before a current time; Determine a first magnitude relationship between the number of lock contention times and the historical maximum number of lock contention times; If the first size relationship indicates that among multiple lock contention times corresponding to multiple second operation requests, a first lock contention time is less than or equal to the historical maximum lock contention time, determining the second initial lock region corresponding to the first lock contention time as a first low contention region; The first low contention areas are merged based on a preset rule to generate a target lock area corresponding to the first lock contention number.

13. The method for obtaining a partition lock according to claim 1, wherein: Determining a target lock region corresponding to the second operation request according to the number of lock contention times further includes: Determining a second magnitude relationship between the number of lock contention times and a first preset threshold; If the second size relationship indicates that a second lock contention number among the multiple lock contention times corresponding to the multiple second operation requests is less than or equal to the first preset threshold, determining the third initial lock region corresponding to the second lock contention number as a second low contention region; The second low contention areas are merged based on a preset rule to generate a target lock area corresponding to the second lock contention number.

14. The method for acquiring a partition lock according to claim 1, wherein: Determining a target lock region corresponding to the second operation request according to the number of lock contention times further includes: Determine a historical maximum number of lock contentions in the first initial lock region corresponding to the second operation request within a target time period before a current time; Determine a first magnitude relationship between the number of lock contention times and the historical maximum number of lock contention times; If the first size relationship indicates that a third number of lock contention times among the multiple lock contention times corresponding to the multiple second operation requests is greater than the historical maximum number of lock contention times, determining the fourth initial lock region corresponding to the third number of lock contention times as a first high contention region, and suspending receiving lock requests for the first high contention region; The lock records contained in the first high contention area are distributed to multiple first sub-lock areas, wherein the target lock area corresponding to the third lock contention number includes: any first sub-lock area.

15. The method for acquiring a partition lock according to claim 1, wherein: Determining a target lock region corresponding to the second operation request according to the number of lock contention times further includes: Determining a third magnitude relationship between the number of lock contention times and a second preset threshold; If the third size relationship indicates that a fourth lock contention count among the multiple lock contention counts corresponding to the multiple second operation requests is greater than the second preset threshold, determining the fifth initial lock region corresponding to the fourth lock contention count as a second high contention region, and suspending receiving lock requests for the second high contention region; The lock records included in the second high contention area are distributed to multiple second sub-lock areas, wherein the target lock area corresponding to the fourth lock contention number includes: any second sub-lock area.

16. The method for acquiring a partition lock according to claim 1, wherein: After pre-checking the multiple operation requests using a filter index to filter the first operation request from the multiple operation requests and obtaining the second operation request, the method further includes: When it is determined based on the root path filter, the directory filter and the lock type filter that the first operation request does not exist in the multiple operation requests, it is prohibited to calculate the number of lock contentions corresponding to the second operation request, and it is determined that the one or more clients are allowed to execute the multiple operation requests.

17. A partition lock acquisition device, characterized in that: include: a pre-check module configured to, upon receiving multiple operation requests sent by one or more clients, pre-check the multiple operation requests using a filter index to filter a first operation request from the multiple operation requests to obtain a second operation request, wherein the filter index includes at least one of the following: a root path filter, a directory filter, and a lock type filter; the first operation request is an operation request that does not require locking and / or does not cause lock contention; and the second operation request is an operation request that causes lock contention; a calculation module, configured to calculate the number of lock contentions corresponding to the second operation request, and determine a target lock region corresponding to the second operation request according to the number of lock contentions; An acquisition module is configured to acquire the partition lock corresponding to the second operation request according to the target lock area.

18. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the method for acquiring a partition lock as claimed in any one of claims 1 to 16 when executing the computer program.

19. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the method for acquiring a partition lock according to any one of claims 1 to 16 are implemented.

20. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for acquiring a partition lock according to any one of claims 1 to 16 are implemented.

Images