File system conflict detection method, device, medium and program product

Abstract

The present application provides a file system conflict detection method, device, medium, and program product, which can be applied to the field of file systems. The conflict detection method includes: in response to a conflict detection instruction for the file system, for any control node among multiple control nodes, performing a first-stage block conflict detection on a first data block belonging to any control node according to a shared bitmap of the file system, obtaining a first detection result, wherein the first detection result indicates a data block with a conflict in the first data block; in response to completion of the first-stage block conflict detection, sending a second data block belonging to the target control node to a target control node among the multiple control nodes, so that the target control node performs a second-stage block conflict detection on the second data block according to the shared bitmap, obtaining a second detection result, wherein the second detection result indicates a data block with a conflict in the second data block, and the target control node is different from the any control node.

Description

Technical Field

[0001] The present application relates to the field of file systems, and more specifically to a conflict detection method, device, medium, and program product for a file system. Background Art

[0002] The repair process of a distributed file system often includes conflict detection for data blocks. However, in a distributed file system, a file's metadata belongs to a single controller, while the file's data may belong to multiple controllers. When multiple controllers concurrently perform conflict detection, and encounter a file whose data spans multiple controllers, cross-controller data blocks must be detected through remote calls. This introduces communication overhead and processing pressure on the controller, resulting in low conflict detection efficiency. Summary of the Invention

[0003] In view of the above problems, the present application provides a file system conflict detection method, device, medium and program product.

[0004] According to a first aspect of the present application, a conflict detection method for a file system is provided, where the file system includes multiple control nodes, and the conflict detection method includes: in response to a conflict detection instruction for the file system, for any control node among the multiple control nodes, performing a first-stage block conflict detection on a first data block belonging to the any control node according to a shared bitmap of the file system, to obtain a first detection result, wherein the first detection result indicates a data block in conflict with the first data block, and the shared bitmap represents a bitmap that can be shared and accessed; and in response to completion of the first-stage block conflict detection, sending a second data block belonging to a target control node among the multiple control nodes, so that the target control node performs a second-stage block conflict detection on the second data block according to the shared bitmap, to obtain a second detection result, wherein the second detection result indicates a data block in conflict with the second data block, and the target control node is different from the any control node.

[0005] A second aspect of the present application provides a conflict detection device for a file system, wherein the file system includes a plurality of control nodes, and the conflict detection device includes:

[0006] A first block conflict detection module is configured to respond to a conflict detection instruction for the file system and, for any of the multiple control nodes, perform a first stage block conflict detection on a first data block belonging to any of the multiple control nodes according to a shared bitmap of the file system to obtain a first detection result, wherein the first detection result indicates a data block in conflict with the first data block, and the shared bitmap represents a bitmap that can be shared and accessed.

[0007] a second block conflict detection module configured to, in response to completion of the first-stage block conflict detection, send a second data block belonging to a target control node among the multiple control nodes, so that the target control node performs a second-stage block conflict detection on the second data block according to the shared bitmap to obtain a second detection result, wherein the second detection result indicates that a conflicting data block exists in the second data block, and the target control node is different from any of the control nodes.

[0008] The third aspect of the present application provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above-mentioned conflict detection method.

[0009] The fourth aspect of the present application further provides a computer-readable storage medium having a computer program or instruction stored thereon, which implements the steps of the above-mentioned conflict detection method when the above-mentioned computer program or instruction is executed by a processor.

[0010] The fifth aspect of the present application further provides a computer program product, comprising a computer program or instructions, which implement the steps of the above-mentioned conflict detection method when executed by a processor.

[0011] By utilizing a shared bitmap, the control node can quickly identify potentially conflicting first data blocks locally and only send second data blocks that require cross-control node processing to the target control node, filtering out invalid communications and reducing network transmission overhead. Furthermore, by having the control node perform a first-stage block conflict check on its own first data blocks, and then having the target control node perform a second-stage block conflict check on cross-control node second data blocks, this separates the local and cross-control detection processes, helping to reduce latency caused by cross-node communication and lowering memory consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above contents and other objects, features and advantages of the present application will become more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0013] Figure 1 A diagram showing an application scenario of a file system conflict detection method according to an embodiment of the present application is shown;

[0014] Figure 2 A flowchart of a file system conflict detection method according to an embodiment of the present application is shown;

[0015] Figure 3 A schematic diagram of a conflict detection method according to an embodiment of the present application is shown;

[0016] Figure 4 A schematic diagram of determining conflicting data blocks according to an embodiment of the present application is shown;

[0017] Figure 5 A flowchart of the first stage block conflict detection according to an embodiment of the present application is shown;

[0018] Figure 6 A schematic diagram of storage space division according to an embodiment of the present application is shown;

[0019] Figure 7 A structural block diagram of a file system conflict detection device according to an embodiment of the present application is shown;

[0020] Figure 8 A block diagram of an electronic device suitable for implementing a conflict detection method for a file system according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0021] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present application. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present application. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present application.

[0022] The terms used herein are only for describing specific embodiments and are not intended to limit this application. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0023] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0024] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0025] Explanation of terms:

[0026] Metadata is additional information that describes and manages user data stored in a file system (FS). It does not directly contain the actual file content used by users, but rather records key information such as the FS's organizational structure, permissions, timestamps, and storage locations.

[0027] When the underlying storage medium of a FS fails, some FS data and metadata are lost. Metadata loss can cause metadata inconsistency, so FS repair is required to restore the metadata to a consistent state.

[0028] The underlying storage of a FS is typically object-based or block-based, essentially representing two different levels of abstraction for data organization and management. During FS repair, block occupancy conflict detection is generally required. This process involves two types of block occupancy conflicts: free block occupancy, where a file occupies a block that is free in the global block occupancy bitmap. This type of block occupancy conflict detection only involves a single global block occupancy bitmap query and is relatively simple. The other type is duplicate block occupancy, where the same block is occupied multiple times by one or more files. This type of block occupancy conflict detection requires a full file scan and is more complex.

[0029] Some related technologies typically use interval trees to detect overlaps in order to detect duplicate block occupancy. These trees can perform interval queries within a time complexity of O(log n + k), where O(log n) is the search time required to locate a potentially overlapping interval in the interval tree, and k is the number of intervals that overlap with the target interval. However, this method uses an average of 32 bytes per key, resulting in high memory consumption in scenarios with a large number of intervals.

[0030] In other related technologies, the detection of duplicate block occupancy can include: first using a Bloom filter to quickly check for duplicates, and then using an interval tree to accurately detect suspected duplicate blocks. However, when the Bloom filter achieves a false alarm rate of 1%, each key occupies an average of 2 bytes, and 2TB of memory is required in a scenario with 1T keys, and the memory consumption is still large. In addition, sharding can also be performed. If the granularity of the sharding is finer, the less data needs to be loaded each time, and the memory usage is lower, but the possibility of cross-shard queries increases, resulting in more disk accesses. On the contrary, the coarser the granularity of the sharding, the higher the memory usage, but the possibility of cross-shard queries decreases.

[0031] In a distributed file system, a file's metadata belongs to a single controller, while its data may belong to multiple controllers. Therefore, in addition to the aforementioned issues, when multiple controllers concurrently scan a file, if data spans multiple controllers, the original controller must be notified to perform a duplicate check. This inefficient cross-controller duplication check affects conflict detection efficiency.

[0032] To solve the technical problem, an embodiment of the present application provides a conflict detection method for a file system, wherein the file system includes multiple control nodes, and the conflict detection method includes: in response to a conflict detection instruction for the file system, for any control node among the multiple control nodes, performing a first stage block conflict detection on a first data block belonging to any control node according to a shared bitmap of the file system, obtaining a first detection result, wherein the first detection result indicates a data block in conflict with the first data block, and the shared bitmap represents a bitmap that can be shared and accessed; and in response to completion of the first stage block conflict detection, sending a second data block belonging to the target control node to a target control node among the multiple control nodes, so that the target control node performs a second stage block conflict detection on the second data block according to the shared bitmap, obtaining a second detection result, wherein the second detection result indicates a data block in conflict with the second data block, and the target control node is different from any control node. By utilizing the shared bitmap, the control node can quickly identify data blocks that may be in conflict locally, and only send the second data block that needs to be processed across control nodes to the target control node, thereby filtering invalid communications and reducing network transmission overhead. In addition, the control node first performs the first stage block conflict detection on the first data block belonging to itself, and then the target control node performs the second stage block conflict detection on the second data block belonging to another node, thereby realizing the process separation of local detection and cross-control detection, which helps to reduce the delay caused by cross-node communication and reduce memory consumption.

[0033] Figure 1 The following diagram shows an application scenario of a conflict detection method for a file system according to an embodiment of the present application.

[0034] like Figure 1 As shown, the file system 100 according to this embodiment may include an A control node 110 and a B control node 120. Upon receiving a conflict detection instruction for the file system 100, the A control node 110 and the B control node 120 perform block conflict detection in parallel.

[0035] Specifically, the A control node 110 scans the first file data 111 belonging to it in the file system, and then determines the first data blocks, such as data block A1 and data block A2, belonging to the A control node 110 in the first file data 111, performs a first-stage block conflict detection on the first data blocks, obtains a first detection sub-result 112, and adds the second data blocks, such as data block A3, belonging to the B control node 120 to the first to-be-processed list 113; at the same time, the B control node 120 scans the second file data 121 belonging to it in the file system, and then determines the first data blocks, such as data block B1 and data block B2, belonging to the B control node 120 in the second file data 121, performs a first-stage block conflict detection on the first data blocks, obtains a second detection sub-result 122, and adds the second data blocks, such as data block B3, belonging to the A control node 110 to the second to-be-processed list 123.

[0036] After both the A control node 110 and the B control node 120 complete the first stage block conflict detection, the first detection sub-result 112 and the second detection sub-result 122 form the first detection result of the first stage block conflict detection. Then, the pending list is sent to the peer controller, so that the peer controller performs the second stage block conflict detection on the second data block in the pending list. Specifically, the A control node 110 sends the first pending list 113 to the B control node 120, and the B control node 120 performs the second stage block conflict detection on the second data block in the first pending list 113, such as data block A3; at the same time, the B control node 120 sends the second pending list 123 to the A control node 110, and the A control node 110 performs the second stage block conflict detection on the second data block in the second pending list 123, such as data block B3. After both the A control node 110 and the B control node 120 complete the second stage block conflict detection, the block conflict detection for the file system is completed.

[0037] It should be understood that Figure 1 The number of control nodes in FIG is only for illustration. Any number of control nodes may be provided according to implementation requirements.

[0038] The following will be based on Figure 1 The scene described by Figures 2 to 6 The conflict detection method of the file system of the application embodiment is described in detail.

[0039] Figure 2 A flowchart of a file system conflict detection method according to an embodiment of the present application is shown.

[0040] like Figure 2As shown, the conflict detection method of the file system of this embodiment includes operations S210 to S220. The file system of this embodiment includes multiple control nodes. Each control node can be controlled by a controller.

[0041] In operation S210, in response to a conflict detection instruction for the file system, a first stage block conflict detection is performed on a first data block belonging to any control node among the multiple control nodes according to a shared bitmap of the file system to obtain a first detection result.

[0042] The conflict detection instruction of the file system can be a command triggered in the distributed system to determine whether there are conflicting data blocks. The conflicting data blocks may be the same data blocks occupied multiple times by one or more files. The first detection result indicates that there are conflicting data blocks in the first data block.

[0043] A bitmap is a data structure composed of binary bits (0 or 1), where each bit represents a binary state: 0 indicates "no", "free", "unoccupied", etc.; 1 indicates "yes", "occupied", "allocated", etc.

[0044] In the file system, each data block corresponds to 1 bit in the bitmap. For example, if the file system includes 8 data blocks, the corresponding bitmap can be bitmap: 01011001, where data blocks 2, 4, 5, and 8 are allocated.

[0045] The shared bitmap in the embodiment of the present application represents a bitmap that can be shared and accessed, that is, a bitmap that can be accessed by multiple control nodes in the file system. For example, the shared bitmap can be stored on a logical unit (LUN) that can be accessed by multiple control nodes.

[0046] It should be noted that the shared bitmap in the embodiment of the present application is a bitmap used for block conflict detection, that is, the initial bitmap state of each bit in the shared bitmap is idle, that is, each bit is 0. During the block conflict detection process, the initial bitmap state of each bit in the shared bitmap is dynamically updated.

[0047] The first data block represents a data block belonging to any control node among the data blocks corresponding to the file belonging to any control node. For example, the data blocks corresponding to the file belonging to the first control node include data block a1 and data block a2. Where data block a1 belongs to the first control node, and data block a2 does not belong to the first control node, then data block a1 is the first data block and data block a2 is the second data block.

[0048] Performing a first-stage block conflict detection on a first data block belonging to any control node according to a shared bitmap of the file system may include: determining, based on a bitmap status in the shared bitmap, whether a bitmap status corresponding to the first data block is an idle state; if the bitmap status is an idle state, indicating that the first data block is not a data block with a conflict; and updating the bitmap status of the first data block to a used state; and if the bitmap status is an used state, indicating that the first data block is a data block with a conflict.

[0049] In operation S220, in response to completion of the first-stage block conflict detection, a second data block belonging to the target control node is sent to a target control node among the multiple control nodes, so that the target control node performs a second-stage block conflict detection on the second data block according to the shared bitmap to obtain a second detection result.

[0050] The second detection result indicates that there is a conflicting data block in the second data block.

[0051] The target control node is different from any control node, and the target control node may be the control node to which the second data block belongs. It should be noted that since multiple control nodes perform block conflict detection in parallel, the target control node is not a fixed control node, but is a target control node relative to a certain data block.

[0052] For example, the data blocks corresponding to the file belonging to the first control node include data block a1 and data block a2, where data block a1 belongs to the first control node and data block a2 belongs to the second control node. Then data block a1 is the first data block, data block a2 is the second data block, and the second control node is the target control node relative to data block a2.

[0053] For another example, the data blocks corresponding to the file belonging to the second control node include data block b1 and data block b2, where data block b1 belongs to the first control node and data block b2 belongs to the second control node. Then data block b2 is the first data block, data block b1 is the second data block, and the first control node is the target control node relative to data block b1.

[0054] It should be noted that there may be multiple target control nodes. For example, the data blocks corresponding to a file belonging to the first control node include data block a1, data block a2, and data block a3. Data block a1 belongs to the first control node, data block a2 belongs to the second control node, and data block a3 belongs to the third control node. In this case, data block a1 is the first data block, and data blocks a2 and a3 are both second data blocks. The second control node is the target control node for data block a2, and the third control node is the target control node for data block a3.

[0055] According to an embodiment of the present application, a first stage block conflict detection is performed on the first data block belonging to any control node according to the shared bitmap of the file system by multiple control nodes to obtain a first detection result; then, after the first stage block conflict detection is completed, the second data block belonging to the target control node is sent to the target control node among the multiple control nodes; at this time, the multiple control nodes simultaneously enter the second stage block conflict detection, obtain the second detection result, and finally complete the block conflict detection. By utilizing the shared bitmap, the control node can quickly identify the data blocks that may have conflicts locally, and only send the second data blocks that need to be processed across control nodes to the target control node, filtering invalid communications and reducing network transmission overhead. In addition, the control node first performs the first stage block conflict detection on the first data block belonging to itself, and then the target control node performs the second stage block conflict detection on the second data block across the control node, thereby realizing the separation of the processes of local detection and cross-control detection, which helps to reduce the delay caused by cross-node communication and reduce memory consumption.

[0056] According to an embodiment of the present application, during conflict detection, each controller first scans the file belonging to the current controller and directly performs the first-stage block conflict detection on the first data block belonging to the current controller. The second data block belonging to other controllers in the file is added to a pending list, pending the second-stage block conflict detection. When all controllers have completed the first-stage block conflict detection, the second-stage block conflict detection begins.

[0057] Figure 3 A schematic diagram of a conflict detection method according to an embodiment of the present application is shown.

[0058] Figure 3 The embodiment shown takes two control nodes as an example to illustrate the conflict detection method of the embodiment of the present application.

[0059] like Figure 3As shown, the conflict detection method of this embodiment may include: in response to the conflict detection instruction 310 for the file system, the A control node 110 and the B control node 120 simultaneously scan the files belonging to the current control node, for example, the A control node 110 scans the first file data 111 belonging to the A control node 110, and the B control node 120 scans the second file data 121 belonging to the B control node 120. Then, the A control node 110 and the B control node 120 simultaneously perform the first stage block conflict detection operation S320, specifically: the A control node 110 performs the first stage block conflict detection operation S320 on the first data blocks belonging to it in the first file data 111, such as data block A1 and data block A2, to obtain the first detection sub-result 112, and adds the second data block belonging to the B control node 120, such as data block A3, to the first to-be-processed list 113, waiting for the execution of the second stage block conflict detection operation S330. At the same time, the B control node 120 performs operation S320 first-stage block conflict detection on the first data blocks belonging to it in the second file data 121, such as data block B1 and data block B2, obtains the second detection sub-result 122, and adds the second data blocks belonging to the A control node 110, such as data block B3, to the second to-be-processed list 123, waiting for operation S330 second-stage block conflict detection.

[0060] After both control node A 110 and control node B 120 complete the first phase of block conflict detection, they send a pending list to the peer control node. For example, control node A 110 sends first pending list 113 to control node B 120, and control node B 120 sends second pending list 123 to control node A 110. Thereafter, control node A 110 and control node B 120 simultaneously perform operation S330 for the second phase of block conflict detection. Specifically, control node A 110 performs operation S330 for the second phase of block conflict detection based on second pending list 123, obtaining a third detection sub-result 114; control node B 120 performs operation S330 for the second phase of block conflict detection based on first pending list 113, obtaining a fourth detection sub-result 124; thereafter, a final detection result 350 is obtained based on the first detection sub-result 112, the second detection sub-result 122, the third detection sub-result 114, and the fourth detection sub-result 124.

[0061] According to an embodiment of the present application, the first data blocks include multiple data blocks; according to the shared bitmap of the file system, performing the first stage block conflict detection on the first data block belonging to any control node includes: determining the bitmap status of each of the multiple first data blocks according to the shared bitmap loaded into the memory, wherein the bitmap status represents the usage status of the first data block; and determining the data block with conflict among the multiple first data blocks according to the bitmap status.

[0062] Within the file system, storage space is divided into multiple fixed-size blocks, such as 8K, 16K, 64K, and 1M. Each control node is responsible for managing a specific range of blocks. When appending to a file, the file system dynamically allocates data blocks based on the requested size and stores them in the file's metadata.

[0063] In some embodiments, performing a first-stage block conflict detection on a first data block belonging to any control node according to a shared bitmap of a file system may include: loading the shared bitmap into memory, traversing all first data blocks belonging to the current control node, querying the bitmap status of each first data block in the shared bitmap, and determining whether there is a conflict based on the bitmap status.

[0064] Direct querying of the shared bitmap after loading it into memory eliminates the need for frequent access to slow storage media like disks, improving conflict detection efficiency and system throughput. Furthermore, using the shared bitmap as a global status identifier across multiple control nodes ensures that all control nodes have a consistent understanding of the data block usage status.

[0065] According to an embodiment of the present application, determining a data block with a conflict among multiple first data blocks based on a bitmap status may include: for any first data block among the multiple first data blocks, in response to the bitmap status representing that the usage status of any first data block is the first state, determining that any first data block is a data block with a conflict.

[0066] The first state may be a state indicating that the first data block is allocated, and the first state may be represented by an identifier “1”.

[0067] When the usage status is the first state, it indicates that the data block has already been allocated to one file. If the data block is now allocated to another file, it indicates that the data block is duplicated and is a data block with a conflict. For example, if data block 1 has already been allocated to file 1, the bitmap status corresponding to data block 1 is allocated. When performing block conflict detection on the file, if file 2 also contains data block 1, the bitmap status obtained by querying the shared bitmap is allocated, and data block 1 is now marked as a data block with a conflict.

[0068] Figure 4 A schematic diagram of determining conflicting data blocks according to an embodiment of the present application is shown.

[0069] like Figure 4As shown, storage space 400 includes data block 1, data block 2, ..., data block n. The first file data 111 scanned by control node A 110 includes data block A1, data block A2, and data block A3, and the second file data 121 scanned by control node B 120 includes data block B1, data block B2, and data block B3. Each data block has a starting address and an address length. Based on the starting address and address length recorded for each data block, it can be determined that data block A1 corresponds to data block 1, data block A2 corresponds to data block 2, data block A3 and data block B1 both correspond to data block 3, and data blocks B2 and B3 both correspond to data block n. Therefore, it can be determined that data block 3 and data block n are conflicting data blocks.

[0070] When the bitmap state indicates that the usage state of any first data block is the second state, the bitmap state corresponding to any first data block in the shared bitmap is updated to the first state.

[0071] The second state may represent a state in which the first data block is not allocated, and the first state may be represented by an identifier “0”.

[0072] When the usage status is the first status, it indicates that the data block has not been allocated, and the bitmap status of the data block can be directly changed.

[0073] When the usage status of the first data block is the second state, the allocation status of the data block can be clearly marked by updating the corresponding state in the shared bitmap to the first state. In the subsequent detection process, it can be clearly seen that the data block has been allocated, avoiding misjudgment in the subsequent block conflict detection process.

[0074] According to an embodiment of the present application, the above-mentioned conflict detection method may further include: before sending the second data block belonging to the target control node to the target control node among multiple control nodes, persistently storing the updated shared bitmap, wherein the target control node performs a second-stage block conflict detection on the second data block according to the updated shared bitmap.

[0075] The target control node performs the second phase of conflict detection based on the persistent shared bitmap, which can ensure that the bitmap information it obtains is the latest state and reduce conflict misjudgments caused by network delays or asynchronous updates.

[0076] Figure 5 A flowchart of the first stage block conflict detection according to an embodiment of the present application is shown.

[0077] It should be noted that the first phase block conflict detection process executed by multiple control nodes included in the file system is the same. Figure 5 The block conflict detection process in the first phase is described in detail by taking one of the control nodes, for example, the first control node, as an example.

[0078] like Figure 5 As shown, the first stage block conflict detection of this embodiment includes operations S510 to S590.

[0079] In operation S510, files belonging to a first control node are scanned, wherein data of each file corresponds to at least one data block.

[0080] In operation S520, one of the data blocks corresponding to the file is acquired to obtain a current data block.

[0081] In operation S530, it is determined whether the current data block belongs to the first control node. If so, operations S540 to S580 are executed; if not, operation S590 is executed.

[0082] In operation S540 , the shared bitmap is loaded into the memory.

[0083] In operation S550, a bitmap status of the current data block is determined according to the shared bitmap.

[0084] In operation S560, it is determined whether the bitmap state is the first state. If so, operation S570 is performed; if not, operation S580 is performed.

[0085] In operation S570 , the bitmap state of the current data block in the shared bitmap is updated to a second state.

[0086] In operation S580 , it is determined that the current data block is a data block with a conflict.

[0087] In operation S590 , the current data block is added to the first to-be-processed list.

[0088] After all control nodes in the file system complete the first phase of block conflict detection, they send the pending list to their corresponding control nodes, and then enter the second phase of block conflict detection. Second-stage block conflict detection is similar to the first phase, differing only in that the second phase of block conflict detection is performed on the data blocks in the pending list. This will not be further described here.

[0089] According to an embodiment of the present application, the storage space of the file system includes multiple storage areas, each of the multiple storage areas is configured with an area bitmap, and the storage area includes multiple data blocks.

[0090] The storage space is the underlying management space composed of the underlying storage components of the file system, such as mechanical hard drives, solid-state drives, magnetic disks and other persistent storage media.

[0091] The capacity of each storage area can be in the GB level, for example, each storage area is 1GB. The capacity of the data block is usually 8K, 16K, 64K, 1M, etc.

[0092] Figure 6 A schematic diagram of storage space division according to an embodiment of the present application is shown.

[0093] like Figure 6 As shown, storage space 400 is divided into multiple storage areas of equal size and connected end to end, such as storage area 0, storage area 1, etc. Each storage area also includes multiple data blocks. Each storage area reserves a region bitmap for block conflict detection, such as region bitmap 610 and region bitmap 620. The region bitmap can be located within the storage area or in a shared area associated with the storage area, such as a logical unit (LUN). The region bitmap is dynamically updated during block conflict detection.

[0094] It should be noted that each storage area belongs to a single control unit. In this case, the method for determining the first data block should be determined in conjunction with the storage area. For example, file data belonging to the first control node corresponds to data block a1, and data block a1 belongs to storage area m. If storage area m belongs to the first control node, data block a1 is the first data block of the first control node. If storage area m does not belong to the first control node, data block a1 is the second data block of the first control node.

[0095] In some embodiments, performing a first stage block conflict detection on a first data block belonging to any control node based on a shared bitmap for a file system may include: determining a target storage area among multiple storage areas based on a storage area to which the first data block belongs; loading a target area bitmap of the target storage area into a memory, and performing block conflict detection on the first data block based on the target area bitmap loaded into the memory.

[0096] The file system's shared bitmap is divided into regional bitmaps based on storage areas. Only the regional bitmap for the target storage area is loaded into memory, avoiding the waste of memory resources caused by loading the entire bitmap. In addition, operations are performed only on the regional bitmap for the target storage area, reducing the data traversal range, reducing conflict detection time, and improving response speed in concurrent scenarios.

[0097] According to an embodiment of the present application, any control node includes multiple threads; the above-mentioned conflict detection method also includes: dividing the files belonging to any control node into multiple file sets according to a preset division rule; performing a first-stage block conflict detection on the first data block belonging to any control node according to the shared bitmap of the file system includes: multiple threads processing multiple file sets in parallel to perform a first-stage block conflict detection on the first data block belonging to any control node in the file set.

[0098] The preset partitioning rule can use hash sharding to partition the files scanned by any control node, obtaining multiple shards, i.e., file collections, and each thread independently scans a single shard.

[0099] Files managed by any control node are divided into multiple file collections according to pre-set rules. Multithreaded parallel processing of these collections reduces the execution time of the first phase of block conflict detection to 1 / Nth of that of a single thread (N is the number of threads), helping to improve processing efficiency in concurrent scenarios. Furthermore, multithreaded parallel processing avoids the blocking issues inherent in a single thread, fully utilizing the multi-core processor resources of the control node to streamline detection tasks.

[0100] According to an embodiment of the present application, the above-mentioned conflict detection method also includes: for any thread among multiple threads, in response to any thread completing the first-stage block conflict detection, updating the state of any thread to the third state; in response to the states of multiple threads all being the third state, determining that the first-stage block conflict detection is completed.

[0101] The third state can be any state that can distinguish the completion of the first phase block conflict detection. For example, it can be represented by any identifier. For example, "phase2" can be used to represent the third state.

[0102] By updating the thread status to the third state, marking detection completion, the control node can monitor the task progress of each thread in real time, avoiding missed or duplicate detections. The determination that all threads are in the third state ensures the integrity of multi-threaded parallel detection tasks and improves the accuracy of detection results.

[0103] In some embodiments, files in a file system may be divided into multiple processing batches, and the first-stage block conflict detection and the second-stage block conflict detection are periodically performed on a batch-by-batch basis, with one batch of file data processed in each cycle. Furthermore, each thread processes a fixed number of files in one cycle, and each processing cycle may specifically include the following operations:

[0104] Phase 1 detects data blocks belonging to the current control node. If the file's corresponding data block belongs to the current control node, the shared bitmap is loaded into memory and the bitmap status of the data block is checked to see if it is 0. If it is 0, indicating no conflict, the bitmap status is updated to 1; otherwise, a conflict exists. If the data block belongs to the peer controller, the file is added to the pending list and awaits processing in phase 2. After all files in the current cycle are scanned, the shared bitmap in memory is persisted, and the current thread state is switched to phase 2.

[0105] Phase 2: Detecting Data Blocks Belonging to the Peer Control Node: When all threads are in phase 2, all control nodes enter phase 2, scan the pending list, and perform block conflict detection on the data blocks belonging to the peer controller in the file. This involves loading a shared bitmap into memory and checking whether the bitmap status of the data block is 0. If it is 0, indicating no conflict, the bitmap status is updated to 1. Otherwise, a conflict exists. After scanning the pending list, the shared bitmap in memory is persisted and the current thread state is switched to phase 2_end. If the pending list is empty, the current thread state is switched directly to phase 2_end.

[0106] When the status of all threads is phase2_end, the processing of the next cycle is started and the status of the current thread is switched to phase1.

[0107] By dividing the full scan into multiple batches and following the batch periodicity so that a certain number of files are scanned in each cycle, memory consumption can be reduced.

[0108] Load balancing among multiple threads is achieved by dividing the files into fixed-size processing batches and allocating an equal amount of files to each thread.

[0109] In addition, Phase 1 prioritizes data blocks belonging to the local control node, using a memory bitmap to quickly detect and update status. Phase 2 centrally processes data blocks belonging to the peer control node, ensuring cross-node consistency through a state synchronization mechanism. This two-phase separation reduces the frequency of cross-node communication and improves overall efficiency.

[0110] In addition, through the orderly transition of thread states (phase1→phase2→phase2_end) and global state synchronization, a thread state-driven automated detection process is realized.

[0111] In summary, the embodiments of the present application achieve high performance, high reliability and scalability of conflict detection in a distributed environment through batch processing, two-stage detection, periodic persistence and thread state-driven collaboration, which helps to improve the system's resource utilization and data security, and is suitable for block conflict detection in large-scale file systems.

[0112] Based on the above file system conflict detection method, the present application also provides a file system conflict detection device. Figure 7 The device is described in detail.

[0113] Figure 7 A structural block diagram of a file system conflict detection device according to an embodiment of the present application is shown.

[0114] like Figure 7As shown, the file system conflict detection device 700 of this embodiment includes a first block conflict detection module 710 and a second block conflict detection module 720.

[0115] The first block conflict detection module 710 is configured to, in response to a conflict detection instruction for the file system, perform a first-stage block conflict detection on a first data block belonging to any of the multiple control nodes based on the file system's shared bitmap, thereby obtaining a first detection result. The first detection result indicates a conflicting data block within the first data block, and the shared bitmap represents a bitmap that allows shared access. In one embodiment, the first block conflict detection module 710 performs operation S210 described above, which will not be further described herein.

[0116] The second block conflict detection module 720 is configured to, in response to completion of the first-stage block conflict detection, send a second data block belonging to a target control node among the multiple control nodes, so that the target control node performs a second-stage block conflict detection on the second data block according to the shared bitmap, and obtains a second detection result, wherein the second detection result indicates that a conflicting data block exists in the second data block, and the target control node is different from any of the control nodes. In one embodiment, the second block conflict detection module 720 can be configured to perform operation S220 described above, which will not be further described herein.

[0117] According to an embodiment of the present application, the above-mentioned first data block includes multiple ones.

[0118] According to an embodiment of the present application, the first block conflict detection module 710 includes: a first determination submodule and a second determination submodule.

[0119] The first determining submodule is configured to determine a bitmap status of each of the plurality of first data blocks according to the shared bitmap loaded into the memory, wherein the bitmap status represents a usage status of the first data block.

[0120] The second determining submodule is configured to determine conflicting data blocks among the plurality of first data blocks according to the bitmap status.

[0121] According to an embodiment of the present application, the second determining submodule includes: a determining unit.

[0122] The determining unit is configured to determine, for any first data block among the plurality of first data blocks, that the use state of the first data block is the first state, in response to the bitmap state indicating that the use state of the first data block is the first state, that the first data block is a conflicting data block.

[0123] According to an embodiment of the present application, the second determining submodule further includes: an updating unit.

[0124] The updating unit is configured to update the bitmap state corresponding to the any first data block in the shared bitmap to the first state in response to the bitmap state indicating that the usage state of the any first data block is the second state.

[0125] According to an embodiment of the present application, the above-mentioned conflict detection device also includes: a persistent storage module.

[0126] A persistent storage module is used to persistently store the updated shared bitmap before sending the second data block belonging to the target control node among the multiple control nodes, wherein the target control node performs a second-stage block conflict detection on the second data block according to the updated shared bitmap.

[0127] According to an embodiment of the present application, the storage space of the file system includes multiple storage areas, each of the multiple storage areas is configured with an area bitmap, and the storage area includes multiple data blocks.

[0128] According to an embodiment of the present application, the first block conflict detection module 710 includes: a third determination submodule and a block conflict detection submodule.

[0129] The third determining submodule is configured to determine a target storage area among the plurality of storage areas according to the storage area to which the first data block belongs.

[0130] The block conflict detection submodule is configured to load the target area bitmap of the target storage area into a memory, and perform block conflict detection on the first data block according to the target area bitmap loaded into the memory.

[0131] According to an embodiment of the present application, any of the above control nodes includes multiple threads.

[0132] According to an embodiment of the present application, the above-mentioned conflict detection device further includes: a division module.

[0133] The division module is used to divide the files belonging to any of the above control nodes into multiple file sets according to preset division rules.

[0134] According to an embodiment of the present application, the first block conflict detection module includes: a parallel processing submodule.

[0135] The parallel processing submodule is configured to process the plurality of file sets in parallel using the plurality of threads, so as to perform a first-stage block conflict detection on a first data block belonging to any of the control nodes in the plurality of file sets.

[0136] According to an embodiment of the present application, the above-mentioned conflict detection device further includes: an updating module and a determining module.

[0137] The updating module is configured to update the state of any thread among the plurality of threads to a third state in response to completion of the first-stage block conflict detection by any thread.

[0138] The determining module is configured to determine that the first-stage block conflict detection is completed in response to the states of the plurality of threads being in the third state.

[0139] According to embodiments of the present application, any multiple modules in the first block conflict detection module 710 and the second block conflict detection module 720 can be combined into a single module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in a single module. According to embodiments of the present application, at least one of the first block conflict detection module 710 and the second block conflict detection module 720 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented in hardware or firmware through any other reasonable means of circuit integration or packaging, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or any appropriate combination of any of these. Alternatively, at least one of the first block conflict detection module 710 and the second block conflict detection module 720 can be at least partially implemented as a computer program module that, when executed, can perform the corresponding functions.

[0140] Figure 8 A block diagram of an electronic device suitable for implementing a conflict detection method for a file system according to an embodiment of the present application is shown.

[0141] like Figure 8 As shown, an electronic device 800 according to an embodiment of the present application includes a processor 801, which can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) 802 or a program loaded from a storage unit 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present application.

[0142] Various programs and data required for the operation of the electronic device 800 are stored in the RAM 803. The processor 801, ROM 802, and RAM 803 are connected to each other via a bus 804. The processor 801 performs various operations of the method flow according to the embodiment of the present application by executing the programs in the ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than the ROM 802 and the RAM 803. The processor 801 may also perform various operations of the method flow according to the embodiment of the present application by executing the programs stored in the one or more memories.

[0143] According to an embodiment of the present application, electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to bus 804. Electronic device 800 may also include one or more of the following components connected to I / O interface 805: an input unit 806 including a keyboard, mouse, etc.; an output unit 807 including components such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage unit 808 including components such as a hard disk; and a communication unit 809 including a network interface card such as a local area network (LAN) card or a modem. Communication unit 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. Removable media 811, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 810 as needed, so that computer programs read from the removable media can be installed into storage unit 808 as needed.

[0144] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the method according to the embodiments of this application is implemented.

[0145] According to an embodiment of the present application, a computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present application, a computer-readable storage medium may include the ROM 802 and / or RAM 803 described above and / or one or more memories other than ROM 802 and RAM 803.

[0146] The embodiments of the present application also include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided in the embodiments of the present application.

[0147] The computer program executes the above functions defined in the system / device of the embodiment of the present application when the processor 801 executes the computer program. According to the embodiment of the present application, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0148] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 809, and / or installed from a removable medium 811. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0149] In such an embodiment, the computer program can be downloaded and installed from the network via the communication section 809, and / or installed from the removable medium 811. When the computer program is executed by the processor 801, the above-mentioned functions defined in the system of the embodiment of the present application are performed. According to the embodiment of the present application, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0150] According to an embodiment of the present application, the program code for executing the computer program provided by the embodiment of the present application can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0151] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0152] Those skilled in the art will appreciate that the features described in the various embodiments of this application may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in this application. In particular, the features described in the various embodiments of this application may be combined and / or coupled in various ways without departing from the spirit and teachings of this application. All such combinations and / or couplings fall within the scope of this application.

[0153] The embodiments of the present application have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present application. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present application, those skilled in the art may make various substitutions and modifications, and these substitutions and modifications should all fall within the scope of the present application.

Claims

1. A conflict detection method for a file system, wherein the file system includes a plurality of control nodes, characterized in that: The conflict detection method comprises: In response to a conflict detection instruction for the file system, performing, for any one of the multiple control nodes, a first stage block conflict detection on a first data block belonging to the any one control node according to a shared bitmap of the file system, to obtain a first detection result, wherein the first detection result indicates a data block in the first data block that has a conflict, and the shared bitmap represents a bitmap for shared access by multiple control nodes in the file system; and In response to completion of the first-stage block conflict detection, sending a second data block belonging to a target control node among the multiple control nodes, so that the target control node performs a second-stage block conflict detection on the second data block according to the shared bitmap to obtain a second detection result, wherein the second detection result indicates that a conflicting data block exists in the second data block, and the target control node is different from any of the control nodes; The first data blocks include a plurality of blocks; and performing the first-stage block conflict detection on the first data block belonging to any one of the control nodes according to the shared bitmap of the file system includes: determining a bitmap status of each of the plurality of first data blocks according to the shared bitmap loaded into the memory, wherein the bitmap status represents a usage status of the first data block; A data block having a conflict among the plurality of first data blocks is determined according to the bitmap status.

2. The conflict detection method according to claim 1, wherein: Determining a conflicting data block among the plurality of first data blocks according to the bitmap status includes: For any first data block among the plurality of first data blocks, in response to the bitmap state indicating that the usage state of the any first data block is the first state, the any first data block is determined to be a data block with a conflict.

3. The conflict detection method according to claim 2, characterized in that: The conflict detection method further includes: In response to the bitmap status indicating that the usage status of any one of the first data blocks is the second status, the bitmap status corresponding to the any one of the first data blocks in the shared bitmap is updated to the first status.

4. The conflict detection method according to claim 3, wherein: The conflict detection method further includes: Before sending the second data block belonging to the target control node to the target control node among the multiple control nodes, the updated shared bitmap is persistently stored, wherein the target control node performs a second stage block conflict detection on the second data block according to the updated shared bitmap.

5. The conflict detection method according to claim 1, wherein: The storage space of the file system includes a plurality of storage areas, each of the plurality of storage areas is configured with a region bitmap, and the storage area includes a plurality of data blocks; The performing the first-stage block conflict detection on the first data block belonging to any one of the control nodes according to the shared bitmap of the file system includes: determining a target storage area among the multiple storage areas according to the storage area to which the first data block belongs; A target area bitmap of the target storage area is loaded into a memory, and block conflict detection is performed on the first data block according to the target area bitmap loaded into the memory. The conflict detection method according to claim 1 , wherein: Any of the control nodes includes multiple threads; The conflict detection method further includes: Dividing the files belonging to any one of the control nodes into multiple file sets according to a preset division rule; Performing a first-stage block conflict detection on a first data block belonging to any one of the control nodes according to the shared bitmap of the file system includes: The multiple threads process the multiple file sets in parallel to perform first-stage block conflict detection on the first data blocks in the multiple file sets belonging to any one of the control nodes.

7. The conflict detection method according to claim 6, characterized in that: The conflict detection method further includes: For any thread among the multiple threads, in response to the completion of the first-stage block conflict detection by the any thread, updating the state of the any thread to a third state; In response to the states of the plurality of threads all being in the third state, it is determined that the first-stage block conflict detection is completed.

8. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the conflict detection method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the conflict detection method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the conflict detection method according to any one of claims 1 to 7 are implemented.

Images