Conflict detection method and device for file system, medium and program product

Abstract

The invention provides a conflict detection method and device for a file system, a medium and a program product, which can be applied to the field of file systems. The conflict detection method comprises the following steps: in response to a conflict detection instruction aiming at a file system, aiming at any control node in a plurality of control nodes, executing first-stage block conflict detection on a first data block belonging to any control node according to a shared bitmap of the file system to obtain a first detection result, the first detection result indicates data blocks with conflicts in the first data blocks; and in response to completion of the first-stage block conflict detection, sending a second data block belonging to a target control node to the target control node in the plurality of control nodes, so that the target control node executes second-stage block conflict detection on the second data block according to the shared bitmap to obtain a second detection result, the second detection result indicates that conflicting data blocks exist in the second data blocks, and the target control node is different from any control node.

Description

Technical Field

[0001] This application relates to the field of file systems, and more particularly to a method, device, medium, and program product for conflict detection in a file system. Background Art

[0002] During the repair process of a distributed file system, it often includes a conflict detection process for data blocks. However, in a distributed file system, the metadata of a file belongs to a single controller, and the data of the file may belong to multiple controllers. When multiple controllers concurrently execute the conflict detection process, when encountering a file with data across multiple controllers, the detection of data blocks across controllers needs to be completed through remote calls, thereby introducing communication overhead and the processing pressure on the home controller, resulting in low conflict detection efficiency. Summary of the Invention

[0003] In view of the above problems, this application provides a method, device, medium, and program product for conflict detection in a file system.

[0004] According to the first aspect of this application, there is provided a method for conflict detection in a file system. The file system includes multiple control nodes. The conflict detection method includes: in response to a conflict detection instruction for the file system, for any one of the multiple control nodes, according to the shared bitmap of the file system, performing a first-stage block conflict detection on the first data blocks belonging to any one of the control nodes to obtain a first detection result, where the first detection result indicates that there are conflicting data blocks in the first data blocks, and the shared bitmap represents a bitmap that can be shared for access; and in response to the completion of the first-stage block conflict detection, sending the second data blocks belonging to the target control node to the target control node among the multiple control nodes, so that the target control node performs a second-stage block conflict detection on the second data blocks according to the shared bitmap to obtain a second detection result, where the second detection result indicates that there are conflicting data blocks in the second data blocks, and the target control node is different from any one of the control nodes.

[0005] The second aspect of this application provides a conflict detection device for a file system. The file system includes multiple control nodes. The conflict detection device includes:

[0006] A first block conflict detection module, configured to, in response to a conflict detection instruction for the file system, for any one of the multiple control nodes, according to the shared bitmap of the file system, perform a first-stage block conflict detection on the first data blocks belonging to any one of the control nodes to obtain a first detection result, where the first detection result indicates that there are conflicting data blocks in the first data blocks, and the shared bitmap represents a bitmap that can be shared for access.

[0007] The second block conflict detection module is configured to, in response to the completion of the first - stage block conflict detection, send the second data block belonging to the target control node among the multiple control nodes to the target control node, so that the target control node performs a second - stage block conflict detection on the second data block according to the shared bitmap to obtain a second detection result, where the second detection result indicates the data blocks with conflicts in the second data block, and the target control node is different from any of the control nodes.

[0008] A third aspect of the present application provides an electronic device, including: one or more processors; a memory for storing one or more computer programs, where the one or more processors execute the one or more computer programs to implement the steps of the conflict detection method.

[0009] A fourth aspect of the present application further provides a computer - readable storage medium, on which computer programs or instructions are stored, and when the computer programs or instructions are executed by a processor, the steps of the conflict detection method are implemented.

[0010] A fifth aspect of the present application further provides a computer program product, including computer programs or instructions, and when the computer programs or instructions are executed by a processor, the steps of the conflict detection method are implemented.

[0011] By using the shared bitmap, the control node can quickly identify the first data blocks that may have conflicts locally, and only send the second data blocks that need to be processed across control nodes to the target control node, filtering out invalid communications and reducing network transmission overhead. In addition, by the control node performing the first - stage block conflict detection on the first data blocks belonging to itself first, and then the target control node performing the second - stage block conflict detection on the second data blocks across ownership, the separation of the local detection and cross - control detection processes is achieved, which helps to reduce the latency caused by cross - node communication and reduce memory consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Through the following description of the embodiments of the present application with reference to the drawings, the above - mentioned content and other objects, features, and advantages of the present application will become clearer. In the drawings:

[0013] Figure 1 An application scenario diagram of the conflict detection method of the file system according to an embodiment of the present application is shown;

[0014] Figure 2 A flowchart of the conflict detection method of the file system according to an embodiment of the present application is shown;

[0015] Figure 3 A schematic diagram of the conflict detection method according to an embodiment of the present application is shown;

[0016] Figure 4 A schematic diagram showing data blocks with conflicts determined according to an embodiment of the present application;

[0017] Figure 5 A flowchart showing the first-stage block conflict detection according to an embodiment of the present application;

[0018] Figure 6 A schematic diagram showing the division of a storage space according to an embodiment of the present application;

[0019] Figure 7 A structural block diagram showing a conflict detection device for a file system according to an embodiment of the present application;

[0020] Figure 8 A block diagram showing an electronic device suitable for implementing a conflict detection method for a file system according to an embodiment of the present application. Detailed implementation manners

[0021] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present application. In the following detailed description, for the sake of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present application. However, obviously, one or more embodiments can be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present application.

[0022] The terms used herein are merely for describing specific embodiments and are not intended to limit the present application. The terms "including", "comprising", etc. used herein indicate the presence of the described features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0023] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0024] In the case of using expressions such as "at least one of A, B, and C", generally, it should be interpreted according to the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include, but is not limited to, a system having only A, only B, only C, having A and B, having A and C, having B and C, and / or having A, B, and C).

[0025] Term explanation:

[0026] Metadata is additional information that describes and manages user data stored in a file system (FS). It does not directly contain the actual file content used by users, but records key information such as the organizational structure, permissions, timestamps, and storage locations of the FS.

[0027] When a failure occurs in the underlying storage medium of the FS, it may cause partial data and metadata in the FS to be lost. The loss of metadata will lead to the metadata being in an inconsistent state. Therefore, it is necessary to perform FS repair to restore the metadata to a consistent state.

[0028] The underlying storage of the FS is usually designed based on objects or blocks, which are essentially two different abstraction levels of data organization and management. During the process of repairing the FS, it is generally necessary to perform block occupancy conflict detection. During the block occupancy conflict detection process, there are two types of block occupancy conflicts. One is free block occupancy, that is, the blocks occupied by a file are free in the global block occupancy bitmap. The detection of this type of block occupancy conflict only involves one query of the global block occupancy bitmap and is relatively simple. The other is duplicate block occupancy, that is, the same block is occupied by one or more files multiple times. The detection of this type of block occupancy conflict requires a full scan of the files and is relatively complex.

[0029] In some related technologies, for the detection of duplicate block occupancy, an interval tree is usually used for interval overlap detection. The interval tree can complete the interval query operation within a time complexity of O(log n + k), where O(log n) is the search time, corresponding to the time required to locate the possibly overlapping intervals in the interval tree, and k is the number of intervals overlapping with the target interval. However, this method occupies an average of 32 bytes per key, resulting in a large memory consumption in the scenario of a large number of intervals.

[0030] In some other related technologies, the detection of duplicate block occupancy can include: first using a Bloom filter for quick duplicate checking, and then using an interval tree for precise detection of suspected duplicate blocks. However, when the Bloom filter achieves a false positive rate of 1%, it occupies an average of 2 bytes per key, and 2TB of memory is required in the scenario of 1T keys, and the memory consumption is still large. In addition, sharding processing can also be performed. If the sharding granularity is finer, the amount of data that needs to be loaded each time is less, and the memory occupancy is lower, but the possibility of cross-shard queries increases, resulting in more disk accesses. On the contrary, the coarser the sharding granularity, the higher the memory occupancy, but the possibility of cross-shard queries decreases.

[0031] In a distributed file system, the metadata of a file belongs to a single controller, and the data of the file may belong to multiple controllers. Therefore, in a distributed file system, in addition to the above problems, when multiple controllers perform concurrent scans and encounter files with data across multiple controllers, it is necessary to notify the owning controller for duplicate checking. The cross-controller duplicate checking efficiency is low, affecting the conflict detection efficiency.

[0032] To solve the technical problem, an embodiment of the present application provides a method for detecting conflicts in a file system. The file system includes multiple control nodes, and the conflict detection method includes: in response to a conflict detection instruction for the file system, for any one of the multiple control nodes, performing a first-stage block conflict detection on a first data block belonging to any one of the control nodes according to the shared bitmap of the file system to obtain a first detection result, where the first detection result indicates that there are conflicting data blocks in the first data block, and the shared bitmap represents a bitmap that can be shared for access; and in response to the completion of the first-stage block conflict detection, sending a second data block belonging to the target control node to the target control node among the multiple control nodes, so that the target control node performs a second-stage block conflict detection on the second data block according to the shared bitmap to obtain a second detection result, where the second detection result indicates that there are conflicting data blocks in the second data block, and the target control node is different from any one of the control nodes. By using the shared bitmap, the control node can quickly identify locally the data blocks that may have conflicts, and only send the second data blocks that need to be processed across control nodes to the target control node, filtering out invalid communications and reducing the network transmission overhead. In addition, by first performing the first-stage block conflict detection on the first data block belonging to itself by the control node and then performing the second-stage block conflict detection on the second data block across ownership by the target control node, the separation of the local detection and cross-control detection processes is achieved, which helps to reduce the latency caused by cross-node communication and reduce the memory consumption.

[0033] Figure 1 The application scenario diagram of the conflict detection method for the file system according to the embodiment of the present application is shown.

[0034] As Figure 1 shown, the file system 100 according to this embodiment may include a control node A 110 and a control node B 120. When receiving a conflict detection instruction for the file system 100, the control node A 110 and the control node B 120 perform block conflict detection in parallel.

[0035] Specifically, the A control node 110 scans the first file data 111 belonging to it in the file system, then determines the first data blocks belonging to the A control node 110 in the first file data 111, such as data block A1 and data block A2, performs the first-stage block conflict detection on the first data blocks to obtain the first detection sub-result 112, and adds the second data blocks belonging to the B control node 120, such as data block A3, to the first pending list 113. At the same time, the B control node 120 scans the second file data 121 belonging to it in the file system, then determines the first data blocks belonging to the B control node 120 in the second file data 121, such as data block B1 and data block B2, performs the first-stage block conflict detection on the first data blocks to obtain the second detection sub-result 122, and adds the second data blocks belonging to the A control node 110, such as data block B3, to the second pending list 123.

[0036] After both the A control node 110 and the B control node 120 complete the first-stage block conflict detection, the first detection sub-result 112 and the second detection sub-result 122 form the first detection result of the first-stage block conflict detection. Then, the pending list is sent to the peer controller, so that the peer controller performs the second-stage block conflict detection on the second data blocks in the pending list. Specifically, the A control node 110 sends the first pending list 113 to the B control node 120, and the B control node 120 performs the second-stage block conflict detection on the second data blocks in the first pending list 113, such as data block A3. At the same time, the B control node 120 sends the second pending list 123 to the A control node 110, and the A control node 110 performs the second-stage block conflict detection on the second data blocks in the second pending list 123, such as data block B3. After both the A control node 110 and the B control node 120 complete the second-stage block conflict detection, the block conflict detection for the file system is completed.

[0037] It should be understood that Figure 1 the number of control nodes in

[0038] is only illustrative. According to the implementation requirements, any number of control nodes can be provided. Figure 1 The following will be based on Figures 2 to 6 the described scenario, and will describe in detail the conflict detection method for the file system of the application embodiment through

[0039] Figure 2 FIG. shows a flowchart of the conflict detection method for the file system according to the embodiment of the present application.

[0040] As Figure 2As shown, the conflict detection method of the file system in this embodiment includes operation S210 to operation S220. The file system in this embodiment includes multiple control nodes. Each control node can be controlled by a controller.

[0041] In operation S210, in response to a conflict detection instruction for the file system, for any one of the multiple control nodes, according to the shared bitmap of the file system, perform a first-stage block conflict detection on the first data block belonging to any one of the control nodes to obtain a first detection result.

[0042] The conflict detection instruction of the file system can be a command triggered in a distributed system for whether there are conflicting data blocks. Conflicting data blocks can be that the same data block is occupied by one or more files multiple times. The first detection result indicates that there are conflicting data blocks in the first data block.

[0043] A bitmap is a data structure composed of binary bits (0 or 1), and each bit represents a binary state: 0: represents states such as "no", "idle", "unoccupied", etc.; 1: represents states such as "yes", "occupied", "allocated", etc.

[0044] In the file system, each data block corresponds to 1 bit in the bitmap. For example, if the file system includes 8 data blocks, the corresponding bitmap can be bitmap: 01011001, where data blocks 2, 4, 5, and 8 are allocated.

[0045] The shared bitmap in the embodiment of the present application represents a bitmap that can be shared for access, that is, a bitmap that can be accessed by multiple control nodes in the file system. For example, the shared bitmap can be saved on a logical unit (LUN) that can be accessed by multiple control nodes.

[0046] It should be noted that the shared bitmap in the embodiment of the present application is the bitmap used for block conflict detection, that is, the initial bitmap state of each bit in the shared bitmap is the idle state, that is, each bit is 0. During the process of block conflict detection, the initial bitmap state of each bit in the shared bitmap is dynamically updated.

[0047] The first data block represents the data block belonging to the file of any one of the control nodes among the data blocks corresponding to the file of any one of the control nodes. For example, the data blocks corresponding to the file belonging to the first control node include data block a1 and data block a2, where data block a1 belongs to the first control node and data block a2 does not belong to the first control node, then data block a1 is the first data block and data block a2 is the second data block.

[0048] Performing first-stage block conflict detection on a first data block belonging to any control node according to the shared bitmap of the file system may include: judging, according to the bitmap status in the shared bitmap, whether the bitmap status corresponding to the first data block is the idle state. If it is the idle state, it indicates that the first data block is not a data block with conflicts, and update the bitmap status of the first data block to the used state. If it is the used state, it indicates that the first data block is a data block with conflicts.

[0049] In operation S220, in response to the completion of the first-stage block conflict detection, send the second data block belonging to the target control node to the target control node among the multiple control nodes, so that the target control node performs second-stage block conflict detection on the second data block according to the shared bitmap to obtain a second detection result.

[0050] Wherein, the second detection result indicates the data blocks with conflicts in the second data block.

[0051] The target control node is different from any control node, and the target control node may be the control node to which the second data block belongs. It should be noted that since the multiple control nodes perform block conflict detection in parallel, the target control node is not a fixed control node, but relative to a certain data block, it is the target control node relative to a certain data block.

[0052] For example, the data blocks corresponding to the file belonging to the first control node include data block a1 and data block a2. Among them, data block a1 belongs to the first control node, and data block a2 belongs to the second control node. Then data block a1 is the first data block, data block a2 is the second data block, and the second control node is the target control node relative to data block a2.

[0053] For another example, the data blocks corresponding to the file belonging to the second control node include data block b1 and data block b2. Among them, data block b1 belongs to the first control node, and data block b2 belongs to the second control node. Then data block b2 is the first data block, data block b1 is the second data block, and the first control node is the target control node relative to data block b1.

[0054] It should be noted that the target control node may include multiple ones. For example, the data blocks corresponding to the file belonging to the first control node include data block a1, data block a2 and data block a3. Among them, data block a1 belongs to the first control node, data block a2 belongs to the second control node, and data block a3 belongs to the third control node. Then data block a1 is the first data block, and both data block a2 and data block a3 are the second data blocks. Among them, the second control node is the target control node relative to data block a2, and the third control node is the target control node relative to data block a3.

[0055] According to an embodiment of the present application, multiple control nodes perform first-stage block conflict detection on a first data block belonging to any control node according to a shared bitmap of a file system to obtain a first detection result; then, after the first-stage block conflict detection is completed, a second data block belonging to a target control node among the multiple control nodes is sent to the target control node; at this time, the multiple control nodes simultaneously enter the second-stage block conflict detection to obtain a second detection result, and finally the block conflict detection is completed. By using the shared bitmap, the control nodes can quickly identify locally the data blocks that may have conflicts, and only send the second data blocks that need to be processed across control nodes to the target control node, filtering out invalid communications and reducing the network transmission overhead. In addition, by first performing the first-stage block conflict detection on the first data block belonging to itself by the control node, and then performing the second-stage block conflict detection on the second data block across ownership by the target control node, the separation of the processes of local detection and cross-control detection is realized, which helps to reduce the delay caused by cross-node communication and reduce the memory consumption.

[0056] According to an embodiment of the present application, during the conflict detection process, each controller first scans the files belonging to the current controller, directly performs the first-stage block conflict detection on the first data blocks belonging to the current controller included in the files, and adds the second data blocks belonging to other controllers included in the files to the pending list to wait for the second-stage block conflict detection. When all controllers have completed the first-stage block conflict detection, the second-stage block conflict detection is entered.

[0057] Figure 3 A schematic diagram of a conflict detection method according to an embodiment of the present application is shown.

[0058] Figure 3 The shown embodiment takes two control nodes as an example to illustrate the conflict detection method of the embodiment of the present application.

[0059] As Figure 3As shown, the conflict detection method of this embodiment may include: The A control node 110 and the B control node 120 respond to the conflict detection instruction 310 for the file system. The A control node 110 and the B control node 120 simultaneously scan the files belonging to the current control node. For example, the A control node 110 scans the first file data 111 belonging to the A control node 110, and the B control node 120 scans the second file data 121 belonging to the B control node 120. Then, the A control node 110 and the B control node 120 simultaneously perform the first-stage block conflict detection of operation S320. Specifically: The A control node 110 performs the first-stage block conflict detection of operation S320 on the first data blocks belonging to it in the first file data 111, such as data block A1 and data block A2, to obtain the first detection sub-result 112, and adds the second data blocks belonging to the B control node 120, such as data block A3, to the first pending list 113, waiting to perform the second-stage block conflict detection of operation S330. At the same time, the B control node 120 performs the first-stage block conflict detection of operation S320 on the first data blocks belonging to it in the second file data 121, such as data block B1 and data block B2, to obtain the second detection sub-result 122, and adds the second data blocks belonging to the A control node 110, such as data block B3, to the second pending list 123, waiting for the second-stage block conflict detection of operation S330.

[0060] After both the A control node 110 and the B control node 120 complete the first-stage block conflict detection, they send the pending lists to the peer control node. For example, the A control node 110 sends the first pending list 113 to the B control node 120, and the B control node 120 sends the second pending list 123 to the A control node 110. Then, the A control node 110 and the B control node 120 simultaneously perform the second-stage block conflict detection of operation S330. Specifically: The A control node 110 performs the second-stage block conflict detection of operation S330 according to the second pending list 123 to obtain the third detection sub-result 114; the B control node 120 performs the second-stage block conflict detection of operation S330 according to the first pending list 113 to obtain the fourth detection sub-result 124; then, the final detection result 350 is obtained according to the first detection sub-result 112, the second detection sub-result 122, the third detection sub-result 114, and the fourth detection sub-result 124.

[0061] According to the embodiment of the present application, there are multiple first data blocks; performing the first-stage block conflict detection on the first data blocks belonging to any control node according to the shared bitmap of the file system includes: determining the bitmap status of each of the multiple first data blocks according to the shared bitmap loaded into the memory, where the bitmap status represents the usage status of the first data block; determining the data blocks in conflict among the multiple first data blocks according to the bitmap status.

[0062] Within the file system, the storage space is divided into multiple blocks of a fixed size, such as 8K, 16K, 64K, 1M, etc. Each control node is responsible for managing a specific range of blocks. When appending to a file, the file system dynamically allocates data blocks according to the size of the service request and saves them in the metadata of the file.

[0063] In some embodiments, performing a first-stage block conflict detection on a first data block belonging to any control node according to the shared bitmap of the file system may include: loading the shared bitmap into memory, traversing all the first data blocks belonging to the current control node, querying the bitmap status of each first data block in the shared bitmap, and determining whether there is a conflict according to the bitmap status.

[0064] After loading the shared bitmap into memory, it can be directly queried without frequently accessing low-speed storage media such as disks, improving the efficiency of conflict detection and the system throughput. In addition, using the shared bitmap as a global status identifier among multiple control nodes ensures that all control nodes have a consistent understanding of the usage status of data blocks.

[0065] According to an embodiment of the present application, determining a conflicting data block among multiple first data blocks according to the bitmap status may include: for any one of the multiple first data blocks, in response to the bitmap status indicating that the usage status of any one of the first data blocks is the first state, determining that any one of the first data blocks is a conflicting data block.

[0066] The first state may be a state indicating that the first data block is allocated, and the first state may be represented by the identifier "1".

[0067] When the usage status is the first state, it indicates that the data block has been allocated to one of the files. At this time, if the data block is allocated to another file, it means that the data block is repeatedly occupied and belongs to a conflicting data block. For example, data block 1 has been allocated to file 1. At this time, the bitmap status corresponding to data block 1 is the allocated state; when performing block conflict detection on file 2, if file 2 also contains data block 1, the bitmap status obtained by querying the shared bitmap is the allocated state. At this time, data block 1 is recorded as a conflicting data block.

[0068] Figure 4 Shows a schematic diagram of determining a conflicting data block according to an embodiment of the present application.

[0069] As Figure 4As shown in the figure, the storage space 400 includes data blocks 1, 2, …, n. The first file data 111 scanned by the A control node 110 includes data blocks A1, A2, and A3. The second file data 121 scanned by the B control node 120 includes data blocks B1, B2, and B3. Among them, each data block records a starting address and an address length. Based on the starting address and address length recorded in each data block, it can be determined that data block A1 corresponds to data block 1, data block A2 corresponds to data block 2, both data block A3 and data block B1 correspond to data block 3, and both data block B2 and data block B3 correspond to data block n. Therefore, it can be determined that data blocks 3 and n are conflicting data blocks.

[0070] When the bitmap status indicates that the usage status of any first data block is the second status, update the bitmap status corresponding to any first data block in the shared bitmap to the first status.

[0071] The second status can represent the unallocated status of the first data block, and the first status can be represented by the identifier "0".

[0072] When the usage status is the first status, it indicates that the data block is unallocated, and the bitmap status of the data block can be directly updated.

[0073] When the usage status of the first data block is the second status, by updating the corresponding status in the shared bitmap to the first status, the allocation status of the data block can be clearly marked. During subsequent detection, it can be clearly known that the data block has been allocated, avoiding misjudgment during subsequent block conflict detection.

[0074] According to an embodiment of the present application, the above conflict detection method may further include: before sending the second data block belonging to the target control node to the target control node among multiple control nodes, persistently store the updated shared bitmap, where the target control node performs second-stage block conflict detection on the second data block based on the updated shared bitmap.

[0075] The target control node performs second-stage conflict detection based on the persistent shared bitmap, which can ensure that the bitmap information it obtains is in the latest state, reducing conflict misjudgment caused by network latency or asynchronous updates.

[0076] Figure 5 Shows a flowchart of the first-stage block conflict detection according to an embodiment of the present application.

[0077] It should be noted that the first-stage block conflict detection processes executed by multiple control nodes included in the file system are the same. Figure 5 Taking one of the control nodes, for example, the first control node, as an example, the detection process of the first-stage block conflict is described in detail.

[0078] As shown Figure 5 in the figure, the first-stage block conflict detection of this embodiment includes operations S510 to S590.

[0079] In operation S510, scan the files belonging to the first control node. Among them, the data of each file corresponds to at least one data block.

[0080] In operation S520, obtain one of the data blocks in the data blocks corresponding to the file to obtain the current data block.

[0081] In operation S530, determine whether the current data block belongs to the first control node. If so, execute operations S540 to S580; if not, execute operation S590.

[0082] In operation S540, load the shared bitmap into the memory.

[0083] In operation S550, determine the bitmap status of the current data block according to the shared bitmap.

[0084] In operation S560, determine whether the bitmap status is the first status. If so, execute operation S570; if not, execute operation S580.

[0085] In operation S570, update the bitmap status of the current data block in the shared bitmap to the second status.

[0086] In operation S580, determine that the current data block is a data block with a conflict.

[0087] In operation S590, add the current data block to the first pending list.

[0088] After all control nodes of the file system have completed the first-stage block conflict detection, send the pending list to the corresponding control node, and then enter the second-stage block conflict detection. The second-stage block conflict detection is similar to the first-stage block conflict detection, except that the data blocks in the second-stage block conflict detection are detected for the databases in the pending list. This will not be elaborated here.

[0089] According to the embodiment of the present application, the storage space of the file system includes multiple storage areas, each of the multiple storage areas is configured with an area bitmap, and the storage area includes multiple data blocks.

[0090] The storage space is the underlying management space composed of the underlying storage components of the file system, such as the storage space composed of persistent storage media such as mechanical hard disks, solid-state drives, and disks.

[0091] The capacity of each storage area can be in the GB level. For example, each storage area is 1GB. The capacity of the data block is usually 8K, 16K, 64K, 1M, etc.

[0092] Figure 6 Shows a schematic diagram of the division of storage space according to an embodiment of the present application.

[0093] As Figure 6 shown, the storage space 400 is divided into multiple storage areas of the same size and connected end to end, such as storage area 0, storage area 1,.... Each storage area also includes multiple data blocks. Each storage area reserves area bitmaps for block conflict detection, such as area bitmap 610 and area bitmap 620. The area bitmap can be located within the storage area or on a shared area associated with the storage area, such as a logical unit (lun), and the area bitmap is dynamically updated during block conflict detection.

[0094] It should be noted that each storage area belongs to a single control unit. In this case, the method of determining the first data block should be determined in combination with the storage area. For example, the file data belonging to the first control node corresponds to data block a1, and data block a1 belongs to storage area m. If storage area m belongs to the first control node, at this time, data block a1 is the first data block of the first control node; if storage area m does not belong to the first control node, at this time, data block a1 is the second data block of the first control node.

[0095] In some of these embodiments, performing the first-stage block conflict detection on the first data block belonging to any control node according to the shared bitmap of the file system may include: determining the target storage area among the multiple storage areas according to the storage area to which the first data block belongs; loading the target area bitmap of the target storage area into the memory, and performing block conflict detection on the first data block according to the target area bitmap loaded into the memory.

[0096] Dividing the shared bitmap of the file system into area bitmaps according to the storage area, and only loading the area bitmap of the target storage area into the memory, avoiding waste of memory resources caused by loading the full bitmap. In addition, only operating on the area bitmap of the target storage area reduces the data traversal range, reduces the time-consuming of conflict detection, and improves the response speed in a concurrent scenario.

[0097] According to an embodiment of the present application, any control node includes multiple threads; the above conflict detection method further includes: dividing the files belonging to any control node into multiple file sets according to a preset division rule; performing the first-stage block conflict detection on the first data block belonging to any control node according to the shared bitmap of the file system includes: multiple threads processing multiple file sets in parallel to perform the first-stage block conflict detection on the first data block belonging to any control node in the file set.

[0098] The preset partitioning rule can use hash sharding to partition the files scanned by any control node to obtain multiple shards, i.e., file sets, and each thread independently scans a single shard.

[0099] Partition the files managed by any control node into multiple file sets according to the preset rule, and through multi-threaded parallel processing of different sets, the execution time of the first-phase block conflict detection is shortened to 1 / N (N is the number of threads) of that of a single thread, which helps to improve the processing efficiency in a concurrent scenario. In addition, multi-threaded parallel processing avoids the blocking problem of a single thread, makes full use of the multi-core processor resources of the control node, and realizes the pipelining of the detection task.

[0100] According to an embodiment of the present application, the above conflict detection method further includes: for any one of the multiple threads, in response to any one of the threads completing the first-phase block conflict detection, updating the state of any one of the threads to a third state; in response to the states of all the threads being the third state, determining that the first-phase block conflict detection is completed.

[0101] The third state can be any state that can distinguish the completion of the first-phase block conflict detection. For example, it can be represented by any identifier. For example, "phase2" can be used to represent the third state.

[0102] By updating the thread state to the third state to mark the detection as completed, the control node can keep track of the task progress of each thread in real time, avoiding detection omissions or duplications. Taking the states of all threads being the third state as the determination condition for the detection to be completed ensures the integrity of the multi-threaded parallel detection task and improves the accuracy of the detection result.

[0103] In some embodiments, the files in the file system can be divided into multiple processing batches, and the first-phase block conflict detection and the second-phase block conflict detection are periodically executed according to the batches, and the file data of one batch is processed within each period. And within one period, each thread processes a fixed number of batches of files. Each processing period can specifically include the following operations:

[0104] Phase 1, detecting the data blocks belonging to the current control node: If the data block corresponding to the file belongs to the current control node, load the shared bitmap into the memory, and query whether the bitmap status of the data block in the bitmap is 0. If it is 0, it means there is no conflict, and update the bitmap status to 1. Otherwise, it means there is a conflict; if the data block belongs to the peer controller, add the file to the list to be processed and wait for phase 2 processing. After all the files in the current period are scanned, persist the shared bitmap in the memory, and then switch the current thread state to phase2.

[0105] Phase 2, detecting data blocks belonging to the peer control node: When the states of all threads are phase2, all control nodes enter Phase 2, scan the pending list, and perform block conflict detection on the data blocks in the file that belong to the peer controller, that is, load the shared bitmap into memory and query whether the bitmap status of the data block in the bitmap is 0. If it is 0, it means there is no conflict, and update the bitmap status to 1. Otherwise, it means there is a conflict. After scanning the pending list, persistently store the shared bitmap in memory and switch the state of the current thread to phase2_end. If the pending list is empty, directly switch the state of the current thread to phase2_end.

[0106] When the states of all threads are phase2_end, start the processing of the next cycle and switch the state of the current thread to phase1.

[0107] By dividing the full-scale scan into multiple batches and periodically according to batches, a certain number of files in each cycle helps reduce memory consumption.

[0108] By dividing the file into fixed-size processing batches and allocating an equal number of files to each thread, load balancing among multiple threads is achieved.

[0109] In addition, in Phase 1, the data blocks belonging to the local control node are processed preferentially, and the memory bitmap is used to quickly detect and update the status; in Phase 2, the data blocks of the peer control node are processed centrally, and the cross-node consistency is ensured through the status synchronization mechanism. The separation of the two phases reduces the cross-node communication frequency and improves the overall efficiency.

[0110] In addition, through the orderly conversion of the thread state (phase1 → phase2 → phase2_end) and global state synchronization, an automated detection process driven by the thread state is realized.

[0111] In summary, the embodiments of the present application realize high-performance, high-reliability, and scalability of conflict detection in a distributed environment through batch processing, two-phase detection, periodic persistence, and thread state-driven coordination, which helps improve the resource utilization rate and data security of the system and is suitable for block conflict detection in large-scale file systems.

[0112] Based on the above conflict detection method for the file system, the present application also provides a conflict detection device for the file system. The following will be combined with Figure 7 Describe this device in detail.

[0113] Figure 7 The structural block diagram of the conflict detection device for the file system according to the embodiment of the present application is shown.

[0114] As Figure 7As shown, the conflict detection device 700 of the file system in this embodiment includes a first block conflict detection module 710 and a second block conflict detection module 720.

[0115] The first block conflict detection module 710 is configured to, in response to a conflict detection instruction for the above file system, for any one of the above multiple control nodes, perform a first-stage block conflict detection on the first data blocks belonging to any one of the above control nodes according to the shared bitmap of the above file system, and obtain a first detection result, where the first detection result indicates that there are conflicting data blocks in the first data blocks, and the shared bitmap represents a bitmap that can be shared for access. In one embodiment, the first block conflict detection module 710 performs the operation S210 described above, which will not be elaborated here.

[0116] The second block conflict detection module 720 is configured to, in response to the completion of the first-stage block conflict detection, send the second data blocks belonging to the target control node to the target control node among the above multiple control nodes, so that the target control node performs a second-stage block conflict detection on the second data blocks according to the shared bitmap, and obtain a second detection result, where the second detection result indicates that there are conflicting data blocks in the second data blocks, and the target control node is different from any one of the above control nodes. In one embodiment, the second block conflict detection module 720 can be used to perform the operation S220 described above, which will not be elaborated here.

[0117] According to an embodiment of the present application, there are multiple first data blocks.

[0118] According to an embodiment of the present application, the first block conflict detection module 710 includes: a first determination sub-module and a second determination sub-module.

[0119] The first determination sub-module is configured to determine the bitmap status of each of the multiple first data blocks according to the shared bitmap loaded into the memory, where the bitmap status represents the usage status of the first data blocks.

[0120] The second determination sub-module is configured to determine the conflicting data blocks among the multiple first data blocks according to the bitmap status.

[0121] According to an embodiment of the present application, the second determination sub-module includes: a determination unit.

[0122] The determination unit is configured to, for any one of the multiple first data blocks, in response to the bitmap status indicating that the usage status of any one of the first data blocks is the first status, determine that any one of the first data blocks is a conflicting data block.

[0123] According to an embodiment of the present application, the second determination sub-module further includes: an update unit.

[0124] An update unit, configured to update, in response to the usage status of any one of the first data blocks being represented by the bitmap as the second state, the bitmap status corresponding to any one of the first data blocks in the shared bitmap to the first state.

[0125] According to an embodiment of the present application, the conflict detection device further includes: a persistent storage module.

[0126] The persistent storage module is configured to perform persistent storage on the updated shared bitmap before sending the second data block belonging to the target control node to the target control node among the multiple control nodes, where the target control node performs a second-stage block conflict detection on the second data block according to the updated shared bitmap.

[0127] According to an embodiment of the present application, the storage space of the file system includes multiple storage areas, each of the multiple storage areas is configured with an area bitmap, and the storage area includes multiple data blocks.

[0128] According to an embodiment of the present application, the first block conflict detection module 710 includes: a third determination sub-module and a block conflict detection sub-module.

[0129] The third determination sub-module is configured to determine a target storage area among the multiple storage areas according to the storage area to which the first data block belongs.

[0130] The block conflict detection sub-module is configured to load the target area bitmap of the target storage area into the memory, and perform block conflict detection on the first data block according to the target area bitmap loaded into the memory.

[0131] According to an embodiment of the present application, any one of the control nodes includes multiple threads.

[0132] According to an embodiment of the present application, the conflict detection device further includes: a partitioning module.

[0133] The partitioning module is configured to partition the files belonging to any one of the control nodes into multiple file sets according to a preset partitioning rule.

[0134] According to an embodiment of the present application, the first block conflict detection module includes: a parallel processing sub-module.

[0135] The parallel processing sub-module is configured to perform parallel processing on the multiple file sets by the multiple threads to perform a first-stage block conflict detection on the first data blocks belonging to any one of the control nodes in the file sets.

[0136] According to an embodiment of the present application, the conflict detection device further includes: an update module and a determination module.

[0137] An update module, configured to update the state of any one of the multiple threads to a third state in response to any one of the multiple threads completing the first-stage block conflict detection.

[0138] A determination module, configured to determine that the first-stage block conflict detection is completed in response to the states of the multiple threads all being the third state.

[0139] According to an embodiment of the present application, any multiple modules among the first block conflict detection module 710 and the second block conflict detection module 720 can be combined and implemented in one module, or any one of them can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present application, at least one of the first block conflict detection module 710 and the second block conflict detection module 720 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on substrate, a system on package, an application specific integrated circuit (ASIC), or can be implemented by any other reasonable means such as hardware or firmware through integrating or packaging circuits, or can be implemented in any one of the three implementation manners of software, hardware, and firmware or in any suitable combination of several of them. Alternatively, at least one of the first block conflict detection module 710 and the second block conflict detection module 720 can be at least partially implemented as a computer program module, and when the computer program module is run, it can execute corresponding functions.

[0140] Figure 8 FIG. shows a block diagram of an electronic device suitable for implementing a conflict detection method for a file system according to an embodiment of the present application.

[0141] As Figure 8 shown, the electronic device 800 according to an embodiment of the present application includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage section 808 into a random access memory (RAM) 803. The processor 801 can include, for example, a general microprocessor (such as a CPU), an instruction set processor, and / or a related chipset, and / or a dedicated microprocessor (such as an application specific integrated circuit (ASIC)), etc. The processor 801 can also include on-board memory for caching purposes. The processor 801 can include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present application.

[0142] In the RAM 803, various programs and data required for the operation of the electronic device 800 are stored. The processor 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. The processor 801 performs various operations of the method flow according to the embodiments of the present application by executing the programs in the ROM 802 and / or the RAM 803. It should be noted that the programs can also be stored in one or more memories other than the ROM 802 and the RAM 803. The processor 801 can also perform various operations of the method flow according to the embodiments of the present application by executing the programs stored in the one or more memories.

[0143] According to an embodiment of the present application, the electronic device 800 may further include an input / output (I / O) interface 805, and the input / output (I / O) interface 805 is also connected to the bus 804. The electronic device 800 may further include one or more of the following components connected to the input / output (I / O) interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a local area network card (LAN) card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output (I / O) interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed so that a computer program read from it can be installed into the storage section 808 as needed.

[0144] The present application also provides a computer-readable storage medium, which may be included in the device / device / system described in the above embodiments; or may exist separately without being assembled into the device / device / system. The above computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the method according to the embodiments of the present application is implemented.

[0145] According to an embodiment of the present application, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In the present application, the computer-readable storage medium may be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present application, the computer-readable storage medium may include one or more memories other than the above-described ROM 802 and / or RAM 803 and / or ROM 802 and RAM 803.

[0146] An embodiment of the present application also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product runs in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present application.

[0147] When the computer program is executed by the processor 801, it executes the above functions defined in the system / apparatus of the embodiment of the present application. According to an embodiment of the present application, the above-described systems, apparatuses, modules, units, etc. can be implemented by computer program modules.

[0148] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices and magnetic storage devices. In another embodiment, the computer program may also be transmitted and distributed in the form of a signal on a network medium, and be downloaded and installed through the communication part 809, and / or be installed from the removable medium 811. The program code contained in the computer program can be transmitted by any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0149] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or be installed from the removable medium 811. When the computer program is executed by the processor 801, it executes the above functions defined in the system of the embodiment of the present application. According to an embodiment of the present application, the above-described systems, devices, apparatuses, modules, units, etc. can be implemented by computer program modules.

[0150] In accordance with embodiments of the present application, program code for executing the computer programs provided by the embodiments of the present application can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, such as Java, C++, Python, the "C" language, or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., by using an Internet service provider to connect through the Internet).

[0151] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of code, and the above-mentioned module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutively represented blocks can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, as well as the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0152] Those skilled in the art can understand that the features described in the various embodiments of the present application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present application. In particular, without departing from the spirit and teachings of the present application, the features described in the various embodiments of the present application can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of the present application.

[0153] The above describes the embodiments of the present application. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present application. Although the embodiments are described separately above, this does not mean that the measures in each embodiment cannot be used advantageously in combination. Without departing from the scope of the present application, those skilled in the art can make various substitutions and modifications, and all such substitutions and modifications should fall within the scope of the present application.

Claims

1. A method for conflict detection in a file system, the file system comprising a plurality of control nodes, characterized in that, The conflict detection method includes: In response to a conflict detection instruction for the file system, for any one of the multiple control nodes, perform first-stage block conflict detection on the first data blocks belonging to the any one control node according to the shared bitmap of the file system, to obtain a first detection result, where the first detection result indicates the data blocks with conflicts in the first data blocks, and the shared bitmap represents a bitmap that can be shared for access; and In response to the completion of the first-stage block conflict detection, send the second data blocks belonging to the target control node to the target control node among the multiple control nodes, so that the target control node performs second-stage block conflict detection on the second data blocks according to the shared bitmap, to obtain a second detection result, where the second detection result indicates the data blocks with conflicts in the second data blocks, and the target control node is different from the any one control node.

2. The conflict detection method according to claim 1, characterized in that There are multiple first data blocks; the performing first-stage block conflict detection on the first data blocks belonging to the any one control node according to the shared bitmap of the file system includes: Determine the bitmap status of each of the multiple first data blocks according to the shared bitmap loaded into the memory, where the bitmap status represents the usage status of the first data block; Determine the data blocks with conflicts in the multiple first data blocks according to the bitmap status.

3. The conflict detection method according to claim 2, wherein The determining the data blocks with conflicts in the multiple first data blocks according to the bitmap status includes: For any one of the multiple first data blocks, in response to the bitmap status representing that the usage status of the any one first data block is the first status, determine that the any one first data block is a data block with a conflict.

4. The conflict detection method according to claim 3, wherein The conflict detection method further includes: In response to the bitmap status representing that the usage status of the any one first data block is the second status, update the bitmap status corresponding to the any one first data block in the shared bitmap to the first status.

5. The conflict detection method according to claim 4, wherein The conflict detection method further includes: Before sending the second data blocks belonging to the target control node to the target control node among the multiple control nodes, perform persistent storage on the updated shared bitmap, where the target control node performs second-stage block conflict detection on the second data blocks according to the updated shared bitmap.

6. The conflict detection method according to claim 1, wherein The storage space of the file system includes multiple storage areas, each of the multiple storage areas is configured with an area bitmap, and the storage area includes multiple data blocks; The performing first-stage block conflict detection on the first data blocks belonging to the any one control node according to the shared bitmap of the file system includes: Determine the target storage area among the multiple storage areas according to the storage area to which the first data block belongs; Load the target area bitmap of the target storage area into the memory, and perform block conflict detection on the first data blocks according to the target area bitmap loaded into the memory.

7. The conflict detection method according to claim 1, wherein The any one control node includes multiple threads; The conflict detection method further includes: Divide the files belonging to the any one control node into multiple file sets according to a preset division rule; Performing first-stage block conflict detection on the first data blocks belonging to any one of the control nodes according to the shared bitmap of the file system includes: The multiple threads process the multiple file sets in parallel to perform first-stage block conflict detection on the first data blocks belonging to any one of the control nodes in the multiple file sets.

8. The conflict detection method according to claim 7, wherein The conflict detection method further includes: For any one of the multiple threads, in response to the completion of the first-stage block conflict detection by the any one of the threads, updating the state of the any one of the threads to a third state; In response to the states of the multiple threads all being the third state, determining that the first-stage block conflict detection is completed.

9. An electronic device, comprising: One or more processors; A memory for storing one or more computer programs, Characterized in that the one or more processors execute the one or more computer programs to implement the steps of the conflict detection method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instruction is executed by the processor, the steps of the conflict detection method according to any one of claims 1 to 8 are implemented.

11. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instruction is executed by the processor, the steps of the conflict detection method according to any one of claims 1 to 8 are implemented.

Images