Data consistency guarantee method and device for distributed file system and medium

Abstract

The invention relates to the technical field of data processing, in particular to a data consistency guaranteeing method and device of a distributed file system and a medium, and the method comprises the steps that a reading node periodically reads operation logs from a writing node in batches and applies the operation logs to a local storage of the reading node; establishing a waiting reading queue and a waiting application queue for an upper-layer reading operation request, when the reading operation request with high data consistency is received, putting an execution thread of the reading operation request into the waiting reading queue, and migrating all threads in the waiting reading queue to the waiting application queue after operation log reading of a current period is completed; when it is monitored that all the operation logs read in the current period are applied to the read node for local storage, all the threads in the waiting application queue execute data reading operation of the read operation request; according to the invention, lock competition is eliminated, and high-consistency data reading with low delay, high throughput and high resource utilization rate is realized.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, device and medium for ensuring data consistency in a distributed file system. Background Technology

[0002] In distributed file systems employing log synchronization mechanisms, data consistency is a prerequisite for ensuring data quality. Current technologies typically rely on each read request thread to independently achieve strong consistency. That is, each read thread must first independently pull the latest log, then sequentially apply the log by competing for a global lock, and finally read the data. However, this approach has serious drawbacks in high-concurrency scenarios. First, multiple threads simultaneously pulling the same log will generate duplicate I / O operations, triggering I / O storms and wasting significant network and disk bandwidth. Second, all threads competing for the unique log application lock leads to intense lock contention, wasting CPU time on idle waiting, drastically reducing system throughput. Furthermore, the randomness of lock contention can cause some read requests to experience severe latency fluctuations due to prolonged lock waiting, resulting in extremely high long-tail latency and severely impacting system stability and service quality.

[0003] Therefore, there is an urgent need to provide a data consistency guarantee method that can eliminate lock contention and improve throughput and resource utilization. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a method, device, and medium for ensuring data consistency in a distributed file system. By setting up a dual-queue waiting mechanism, lock contention is eliminated, achieving strong consistency data reading with low latency, high throughput, and high resource utilization.

[0005] According to a first aspect of the present invention, a method for ensuring data consistency in a distributed file system is provided, comprising the following steps: By using a centralized log application pre-deployed locally on the read node, operation logs are periodically read in batches from the write node, and the read operation logs are applied in batches to the local storage of the read node.

[0006] On the local read node, a waiting queue for read operations and a waiting queue for applications are established for upper-layer read operation requests.

[0007] When a read operation request is received and the configured data consistency level is strong consistency, the thread executing the read operation request is placed in the waiting read queue. After the centralized log application completes the reading of the operation log for the current period, it broadcasts to wake up all threads in the waiting read queue so that the woken-up threads are migrated to the waiting application queue and suspended.

[0008] Once it is detected that the centralized log application has applied all the operation logs read in the current period to the local storage of the read node, a broadcast wakes up all threads in the waiting application queue, so that the awakened threads can perform the data reading operation of the read operation request based on the latest local storage of the read node.

[0009] According to a second aspect of the present invention, a non-transitory computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored therein, the at least one instruction or the at least one program being loaded and executed by a processor to implement the data consistency guarantee method of the distributed file system described above.

[0010] According to a third aspect of the present invention, an electronic device is provided, including a processor and the aforementioned non-transitory computer-readable storage medium.

[0011] The present invention has at least the following beneficial effects: This invention provides a data consistency guarantee method for a distributed file system. First, a centralized log application deployed locally on the read node periodically reads operation logs from the write node in batches and applies them to the local storage of the read node. On the read node, a waiting queue for read operations and a waiting queue for application operations are established for upper-layer read operation requests. When a read operation request with strong data consistency is received, the execution thread of the read operation request is placed in the waiting queue for read operations. After the centralized log application completes the reading of the operation logs for the current period, all threads in the waiting queue for read operations are moved to the waiting queue for application operations. When it is detected that the centralized log application has applied all the operation logs read in the current period to the local storage of the read node, all threads in the waiting queue for application operations execute the data reading operation of the read operation request. This invention eliminates lock contention by setting up a dual-queue waiting mechanism, achieving low latency, high throughput, and high resource utilization for strongly consistent data reading. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating a data consistency guarantee method for a distributed file system provided in an embodiment of the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] This invention provides a method for ensuring data consistency in a distributed file system, such as... Figure 1 As shown, the method includes the following steps: S1 periodically reads operation logs from the write node in batches through a centralized log application pre-deployed locally on the read node, and applies the read operation logs in batches to the local storage of the read node.

[0016] Specifically, the centralized log application is an independent background thread running within the read node, and serves as the sole functional entity within the read node responsible for performing log reading and log application.

[0017] Furthermore, the centralized log application works in a round-robin manner, and each round includes a batch reading phase and a batch application phase; wherein, in the batch reading phase, the centralized log application reads new operation logs sequentially from the end position of the previous round until the current end of the log stream; in the batch application phase, the centralized log application applies the new operation logs read in the current round to the local storage of the read nodes in sequence.

[0018] As described above, by establishing a single centralized log application, the log fetching and application operations that were originally executed concurrently by multiple read threads are unified and normalized into a single sequential I / O operation of the centralized log application. This fundamentally eliminates I / O storms and redundant calculations, normalizes network and disk I / O, and improves the overall throughput and resource utilization efficiency of the system through the design of the batch processing mechanism.

[0019] In a preferred embodiment, the dynamic batch read threshold corresponding to the batch read operation log from the write node meets the following conditions: Where T represents the dynamic batch read threshold, T min This is the preset minimum batch read threshold. This represents the floor function, where λ is the preset base threshold coefficient, L is the total number of newly added operation logs, and N is the number of read nodes in the cluster. base P is the baseline value for the total amount of operation logs. base F is the baseline value for the failure probability. maxR represents the maximum probability of failure for each read node within a historical unit of time, and R is the current log write rate of the write node. base ε is the baseline value for write speed, ε is the preset minimum value to prevent the denominator from being zero, and α, β, and γ are preset weighting coefficients, respectively.

[0020] As described above, the batch read threshold can be dynamically adjusted based on data conditions. This involves comprehensively considering multiple dimensions of indicators, including the amount of logs to be synchronized, node stability, and real-time write pressure. By dynamically adjusting the batch size of a single log synchronization, an intelligent balance between synchronization efficiency and system risk is achieved. This allows the system to proactively increase the batch size to improve throughput under high load and low risk, and automatically decrease the batch size to ensure rapid response and data security when the risk of failure increases. This enables the system to adapt to complex and ever-changing operating environments, significantly improving the resource utilization and operational stability of the distributed file system.

[0021] In another embodiment, step S1 is replaced by the following steps: P1, construct a one-write-many-read architecture; the one-write-many-read architecture includes a write node with unique write permissions and multiple read nodes, the write node and each read node share access to the same shared storage layer; the write node is locally configured with a B-tree data structure, and each read node is locally configured with a copy of the B-tree data structure; it can be understood that: all nodes share the underlying data blocks, but each maintains an independent B-tree data structure in memory to manage operation logs, where the operation logs do not record the content of the data blocks, but rather the operation records of the B-tree data structure.

[0022] Specifically, the shared storage layer includes a cache set consisting of several caches and at least one backup storage device; the shared storage layer supports dynamically attaching or detaching the backup storage device when the write node and read node are running; that is, since the shared storage layer is a storage space independent of the write node and read node, during the operation of the distributed file system, the administrator can add new storage devices to the shared storage layer to expand the capacity, or remove old, faulty, or replacement storage devices without stopping the system.

[0023] As described above, by constructing a unique write node and a shared storage layer, the data conflicts and complex coordination overhead caused by multi-node writes are avoided from the root. Furthermore, the configuration of B-tree data structure replicas enables each read node to have independent metadata indexing capabilities, providing a natural data consistency guarantee based on shared storage, which lays the foundation for subsequent efficient and differentiated data access.

[0024] P2, when the write node performs a file system write operation, stores the write operation content to the shared storage layer, and at the same time converts the write operation content into an operation log for the local B-tree data structure of the write node, and synchronizes the operation log to each read node through a centralized log application pre-deployed locally on each read node; it can be understood that the operation log is synchronized by reading the operation log through the log reading application thread of the read node.

[0025] Specifically, each operation log is assigned a globally unique log sequence number.

[0026] Specifically, the operation log for the local B-tree data structure of the write node refers to the recording of insertion, deletion, or modification operations on the local B-tree data structure of the write node. For example, when a write node writes data to the shared storage layer, it inserts the location index information of the written data into the B-tree data structure.

[0027] In one specific embodiment, synchronizing the operation log to each read node includes the following steps: P201. For any read node, the local log reading application thread of the read node periodically requests new operation logs from the write node whose log sequence number is higher than the locally applied sequence number of the read node.

[0028] P202 reads the requested new operation logs through the read node, and synchronizes the new operation logs to the local read node.

[0029] The above decouples data storage from operation log generation, requiring only the synchronization of lightweight B-tree operation logs instead of the complete data transmission in the traditional mode. This reduces the amount of data to be synchronized from GB or TB to KB or MB, and the latency from seconds to minutes to milliseconds, greatly reducing network transmission overhead and improving synchronization efficiency. Furthermore, the globally unique log sequence number provides a clear time order for the entire system, which is key to ensuring eventual data consistency and order.

[0030] Furthermore, the following steps are included after step P2: P3, at each read node, the received operation logs are applied to the local B-tree data structure copy of the read node in the order of the log sequence number through the preset log read application thread, so that the B-tree data structure copy is consistent with the state of the B-tree data structure of the write node; it can be understood that the read node performs the same operation in sequence on the local B-tree data structure according to the operation log content.

[0031] Specifically, applying the received operation logs to the local B-tree data structure replica of the read node in sequence according to the log sequence number further includes the following steps: P301 sorts the received operation logs according to their sequence numbers and removes duplicate operation logs, resulting in an ordered sequence of logs to be applied. In practice, network retransmission, inconsistent data states after fault recovery, and restarting of the log reading application thread after an abnormal crash can all lead to duplicate operation logs.

[0032] P302, apply each operation log in the ordered sequence to be applied to the local B-tree data structure copy of the read node one by one.

[0033] As described above, by setting up independent log reading application threads on each read node, logs are replayed asynchronously and in an orderly manner on each read node. This allows the local B-tree data structure replicas on each read node to gradually synchronize with the state of the write node, achieving eventual data consistency. Furthermore, by storing operation logs instead of storing complete data, the storage space occupied is significantly reduced.

[0034] Furthermore, the lifecycle of the log reading application thread is controlled by a preset state machine, including: In the initialization state, two reader waiting queues are created and initialized; this can be understood as: creating a waiting queue for reading and a waiting queue for application on the local read node, which is applicable to scenarios that require strong data consistency.

[0035] In the running state, it enters a main loop to repeatedly perform operations such as obtaining, processing and applying operation logs, and checks whether a stop command has been received or an error status has occurred after each loop.

[0036] Upon receiving a stop command or encountering an error, exit the main loop, perform resource cleanup, and switch to the exit state.

[0037] P4, for any read node, based on the current B-tree data structure replica on the read node's local machine, when the read node receives a read operation request, it adopts the corresponding read operation strategy according to the pre-configured data consistency level corresponding to the read operation request; it can be understood as: controlling the consistency level through data consistency configuration items.

[0038] The data consistency level is either strong consistency or eventual consistency. Strong consistency means that after new data is written, any subsequent read operation must be able to read the latest updated data, ensuring data consistency between the read node and the write node at the current moment. Eventual consistency means that after new data is written, all B-tree data structure replicas on all read nodes will eventually reach a consistent state after an indeterminate period of time. During this period, due to the existence of operation logs synchronized to the read nodes but not yet applied, the data read by the user is not the latest data currently written by the write node.

[0039] Specifically, the read operation request refers to requesting the read node to access the corresponding data block in the shared storage layer based on the read node's local B-tree data structure copy and return it to the user.

[0040] Furthermore, the step of adopting the corresponding read operation strategy based on the pre-configured data consistency level includes the following steps: P401, if the data consistency level is strong consistency, then determine whether the log sequence number corresponding to the data to be accessed by the read operation request has been applied to the local read operation; if yes, then directly perform the read operation from the shared storage layer based on the current B-tree data structure copy of the local read operation; if no, then wait for the operation log corresponding to the data to be accessed by the read operation request to be applied locally before performing the read operation from the shared storage layer based on the updated B-tree data structure copy of the local read operation. This can be understood as: locating the data storage location in the shared storage layer based on the index information in the B-tree data structure copy, and then performing data reading.

[0041] P402 If the data consistency level is eventual consistency, then read operations are performed directly from the shared storage layer based on the local copy of the current B-tree data structure.

[0042] As described above, the configurable consistency levels and corresponding read strategies provide flexibility for upper-layer applications. The strong consistency strategy ensures that the latest data is read by waiting for the log application, while the eventual consistency strategy allows direct reading of the local current B-tree data structure copy to obtain the lowest read latency. This design achieves an on-demand balance between performance and consistency guarantees.

[0043] S2, on the local read node, establishes a waiting queue for read operations and a waiting queue for applications for upper-layer read operation requests; this can be understood as: dividing all read threads on the local read node into two parts, placing them in the waiting queue for read and the waiting queue for applications respectively.

[0044] Specifically, the waiting read queue refers to the read threads that need to read the data corresponding to the latest operation log; it can be understood as: when a read operation request is received, the execution thread corresponding to the read operation request is put into the read thread in the waiting read queue and suspended.

[0045] Specifically, the waiting application queue refers to the queue used to place read threads that need to wait for the read operation logs to be applied.

[0046] As described above, by establishing logically separated dual waiting queues, the two waiting stages required for a strong consistency read operation are clearly defined and isolated. This transforms a many-to-many distributed coordination problem into a one-to-many centralized notification problem, providing a structural foundation for precise and orderly thread scheduling and notification mechanisms. This design allows system state changes, such as log read or log applied, to be clearly mapped to different waiting queues, providing an implementation basis for strong data consistency between read and write nodes.

[0047] S3, when a read operation request is received and the configured data consistency level is strong consistency, the execution thread of the read operation request is placed in the waiting read queue. After the centralized log application completes the reading of the operation log for the current period, it broadcasts to wake up all threads in the waiting read queue so that the woken-up threads are migrated to the waiting application queue and suspended.

[0048] As mentioned above, during high-concurrency reads, multiple concurrent read threads will perform repeated and parallel read operations on the same log area simultaneously or within a very short period of time. By designing a wait-to-read queue, the I / O amplification effect is reduced, disk and network bandwidth utilization is saved, and I / O efficiency is effectively improved.

[0049] S4, when it is detected that the centralized log application has applied all the operation logs read in the current period to the local storage of the read node, it broadcasts to wake up all threads in the waiting application queue, so that the woken threads can perform the data reading operation of the read operation request based on the latest local storage of the read node; it can be understood that the current read node has been synchronized to the data consistency state of this round, and the read thread that leaves the waiting application queue performs the subsequent actual data reading, that is, reads the data corresponding to the read operation request from the shared storage layer.

[0050] As mentioned above, when multiple read threads concurrently complete reading the logs written to the write node, a global lock or a complex distributed lock mechanism must be used to arbitrate which read thread can apply which log segment, which will cause lock contention and affect the quality and efficiency of data transmission. The dual waiting queue method described above ensures that the log application itself must be executed strictly serially, so that read requests either wait or read the data that has been fully applied, and will never read the intermediate state of "log has been read but not applied", thereby ensuring strong consistency and correctness of the data state.

[0051] Specifically, the awakened thread performs a data read operation based on the latest local storage of the read node, including: S401 uses the latest locally stored B-tree data structure of the read node as an index to locate the target data corresponding to the awakened thread in the shared storage layer; this can be understood as: the B-tree data structure is used to store the location information of the data.

[0052] S402 reads the located target data and sends it back to the corresponding user terminal.

[0053] In summary, this invention achieves lock-free, batch scheduling of read threads through two precise broadcast notifications: "log reading complete" and "log application complete." This mechanism completely liberates read threads from complex log processing and lock contention. Concurrent read threads relinquish their application rights and become pure waiters, eliminating the lock contention bottleneck under high concurrency. It achieves complete decoupling of system performance and the number of concurrent read operation requests. As read operation requests increase, the waiting queue only grows longer without increasing I / O burden or lock contention. At the same time, by eliminating the randomness of lock contention, system latency becomes highly predictable, effectively improving long-tail latency, ensuring stable service quality, and achieving strong data consistency between read and write nodes.

[0054] In another embodiment, the present invention also provides a testing method for the above-described distributed file system, comprising the following steps: Q1. Receive and parse the configuration file of the target test scenario corresponding to the distributed file system to be tested; the configuration file includes the parameters to be tested, the test atomic operation library, and the verification rules; in specific implementation, the configuration file is defined through declarative syntax. Declarative syntax is a configuration paradigm. Users only need to declare what goal they want to achieve or what state the system should be in, without needing to describe in detail how to achieve this goal step by step.

[0055] Preferably, the target test scenarios include, but are not limited to, data consistency testing, stress testing, high availability testing, and stability testing.

[0056] Specifically, the test atomic operation library includes a basic interface operation module for the distributed file system, a distributed coordination operation module, a fault injection module, and a stress load generation module. The distributed coordination operation module simulates node start-up and shutdown, network differentiation, and clock drift; the fault injection module actively injects preset I / O errors, process crashes, memory corruption, or network anomalies during the test; and the stress load generation module integrates or simulates high-concurrency, high-throughput I / O loads.

[0057] As described above, by decoupling test requirements from implementation through declarative configuration files, testers can define complex scenarios without writing a lot of unit test code, which greatly reduces the testing threshold and achieves flexibility and maintainability of test scenarios. The configuration file describes the parameters and verification rules corresponding to the target test scenario in a structured way, providing accurate input for subsequent automated processing.

[0058] Q2, by combining the values ​​of several parameters to be tested through Cartesian product to obtain several parameter combinations, and calling the test atomic operation library corresponding to the distributed file system to orchestrate and generate several test cases corresponding to each parameter combination; it can be understood that: the test atomic operation library refers to a collection of pre-encapsulated and reusable basic operation functions, where an atomic operation refers to a basic operation that is indivisible and serves as the smallest test unit.

[0059] For ease of understanding, the process of generating parameter combinations through the Cartesian product is described as follows: For example, the parameters to be tested include disk, mode, replic, and block_size. Each parameter to be tested has at least one value. The Cartesian product can list all possible combinations of parameter values. Each combination contains one value of each parameter to be tested. When disk corresponds to 3 values, mode corresponds to 1 value, replic corresponds to 2 values, and block_size corresponds to 2 values, a total of 3×1×2×2 parameter combinations can be obtained. Each parameter combination contains one value from disk, mode, replic, and block_size.

[0060] Furthermore, step Q2 also includes filtering the parsed parameters to be tested, as follows: Q201, Parse the predefined constraints between the parameters to be tested from the configuration file; the constraints include mutual exclusion, dependency and conditional relationships.

[0061] Q202: Iterate through several parameter combinations and, based on the mutual exclusion relationship, filter out all parameter combinations that contain mutually exclusive parameters to be tested.

[0062] Q203, based on the dependency relationship, verify the filtered parameter combination, and perform a correction operation on the target parameter combination that contains the dependent test parameter but does not contain the corresponding dependent test parameter; the correction operation includes: supplementing the dependent test parameter, or marking the target parameter combination as invalid and filtering it.

[0063] Q204, based on the aforementioned conditional relationship, the modified parameter combinations that do not meet the preset conditional logic are filtered again to obtain the screening results of the parameters to be tested.

[0064] As described above, by parsing and applying the constraint relationships between the parameters to be tested, invalid parameter combinations are intelligently filtered out and incomplete combinations are corrected. This significantly reduces the number of redundant test cases while ensuring the effectiveness of the test, simplifies the set of parameters to be tested, and significantly improves test execution efficiency and resource utilization.

[0065] Furthermore, step Q2 also includes filtering several parameter combinations, as follows: Q210, randomly select a parameter combination from the original set of parameter combinations and add it to a preset set of combinations to be filled; the initial state of the set of combinations to be filled is empty.

[0066] Q220, calculate the difference between each remaining parameter combination in the original parameter combination set and each parameter combination in the current set of combinations to be filled, and obtain the minimum difference corresponding to each remaining parameter combination; this can be understood as: the obtained minimum difference represents the current distance between its corresponding parameter combination and the current set of combinations to be filled.

[0067] Specifically, the formula for calculating the degree of difference is as follows: Where D(A, B) represents the difference between parameter combination A and parameter combination B, n is the total number of parameters to be tested, and A i and B i These are the values ​​of the i-th parameter to be tested in parameter combination A and parameter combination B, respectively. i It is the preset weight of the i-th parameter to be tested, δ(A) i B i ) is the difference indicator function, when A i ≠B i At that time, δ(A) i B i )=1, when A i =B i At that time, δ(A) i B i )=0.

[0068] Q230: Based on the minimum difference degree corresponding to each of the remaining parameter combinations, migrate the parameter combination with the largest minimum difference degree to the set of combinations to be filled, and repeat steps Q220-Q230 until the parameter combinations in the set of combinations to be filled reach the preset target, thus completing the screening of several parameter combinations; This can be understood as: in each iteration, add the parameter combination with the largest distance from the set of combinations to be filled to the set of combinations to be filled.

[0069] As described above, by prioritizing the selection of parameter combinations with the greatest differences from the original combinations and adding them to the set of combinations to be filled, the maximum dispersion and diversity of the final test case set in the parameter space is ensured. The maximum range of parameter combination possibilities is covered with the fewest number of test cases. This significantly improves testing efficiency while effectively avoiding redundancy in test scenarios and ensuring the rationality and coverage quality of test scenarios.

[0070] Q3 distributes the operation instructions in each test case to the write nodes and corresponding read nodes in the distributed file system for collaborative execution and monitors the test process; the operation instructions include data read and write operations and fault simulation operations; it can be understood that: if a node needs to be tested, then the test load is deployed on the corresponding node and the automated test cases are driven to be executed.

[0071] As described above, by distributing operation instructions to cluster nodes for collaborative execution, distributed read / write and fault scenarios are accurately simulated. This solves the problem that existing tests can only test the performance and stress of a single node and cannot meet the testing of distributed functions, especially advanced features of distributed file systems, such as read / write separation and data consistency.

[0072] Q4. When the test is completed, the data status of the distributed file system is verified according to the verification rules, and the test results are output; wherein, the data status verification includes data consistency verification.

[0073] Specifically, the steps for data consistency verification are as follows: Q401, when the node to be tested writes data, a unique data sequence number is added to the data header; this can be understood as: adding a data sequence number to the header of the written data block as an identifier for the data block.

[0074] Q402 stores the data carrying the data sequence number in a preset shared storage layer, and simultaneously sends the data sequence number to the node to be tested through the shared communication unit.

[0075] Q403: When the node under test reads data sequentially from the shared storage layer, the data sequence number in the data is extracted and compared with the data sequence number obtained through the shared communication unit in sequence. When the comparison is consistent, the node under test writes and the node under test read are considered to have data consistency.

[0076] As described above, by verifying data consistency according to preset verification rules, the core correctness attributes of the distributed system are objectively verified, eliminating the subjective error of human judgment. By maintaining data consistency between write nodes and read nodes, the reliable operation of the distributed file system and the accurate reading and querying of data can be maintained.

[0077] Furthermore, the method also includes the following steps: Q410: After performing a fault simulation operation, monitor the current role and status of all nodes in the distributed file system.

[0078] Q420 If, within a preset time period, the current role and status of all nodes are detected to conform to the expected state transition path, then verify whether there is only one master node and the roles of the other nodes are correct and point to the master node; this can be understood as: the state transition path includes at least: the original master node becomes unavailable, and a specified slave node eventually becomes a readable and writable master node.

[0079] Q430, if the verification passes, the failover process is deemed successfully verified and the test results are output.

[0080] As mentioned above, for a distributed file system with read-write master nodes and read-only slave nodes, after a fault test, it is necessary to immediately verify whether the role switching and data state transfer of each node are as expected, rather than checking whether the service has recovered after a traditional restart. This is to ensure that data transmission and subsequent data consistency verification are performed while ensuring that the node state is correct, thereby improving verification efficiency.

[0081] Preferably, the batch read threshold corresponding to the batch read operation log from the write node is adjusted through the following steps: Q01. After the test is completed, collect the input feature set and the corresponding data consistency verification results for each test case.

[0082] Specifically, the input feature set includes data features, system load features, and fault and topology features. Among them, data features include file size, data type, compression algorithm, and encryption mode; system load features include the number of concurrent write threads, read-write ratio, average IO depth, and cache hit strategy; fault and topology features include simulated fault type, affected nodes, master node switchover, and post-recovery verification latency.

[0083] Q02, by comparing the data consistency verification results corresponding to different test cases, identify the set of fault features associated with the deterioration of data consistency results from the corresponding input feature set; this can be understood as: taking different input features in different parameter combinations as pending fault features that affect different data consistency results, and then comparing whether each pending fault feature is also a pending fault feature in other parameter combinations. When the number of times each input feature is judged as a pending fault feature exceeds the threshold, it is confirmed as a fault feature.

[0084] Q03, during the operation of the distributed file system, the input characteristics of the data to be processed are monitored in real time; if the input characteristics of the current data are detected to match at least one fault feature in the identified fault feature set, the estimated fault probability of the current data is calculated, and when the estimated fault probability exceeds the preset probability threshold, the batch read threshold is dynamically lowered.

[0085] Specifically, the estimated failure probability of the current data is calculated by weighting each input feature that matches the failure feature and the weight of each input feature. The weight of each input feature can be calculated using historical data corresponding to the test cases, employing either the entropy weight method or gray-scale correlation analysis. The specific processes for calculating weights using the entropy weight method and gray-scale correlation analysis are well-known in this field and will not be elaborated upon here.

[0086] Embodiments of the present invention also provide a non-transitory computer-readable storage medium, which can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a method in the method embodiments, wherein the at least one instruction or the at least one program is loaded and executed by the processor to implement the data consistency guarantee method of the distributed file system provided in the above embodiments.

[0087] Embodiments of the present invention also provide an electronic device, including a processor and the aforementioned non-transitory computer-readable storage medium.

[0088] While specific embodiments of the invention have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of the invention. It should also be understood that various modifications can be made to the embodiments without departing from the scope and spirit of the invention. The scope of the invention is defined by the appended claims.

Claims

1. A method for ensuring data consistency in a distributed file system, characterized in that, The method includes the following steps: By using a centralized log application pre-deployed locally on the read node, operation logs are periodically read in batches from the write node, and the read operation logs are applied in batches to the local storage of the read node; On the local read node, a waiting read queue and a waiting application queue are established for upper-layer read operation requests. When a read operation request is received and the configured data consistency level is strong consistency, the thread executing the read operation request is placed in the waiting read queue. After the centralized log application completes the reading of the operation log for the current period, it broadcasts to wake up all threads in the waiting read queue so that the woken-up threads are migrated to the waiting application queue and suspended. Once it is detected that the centralized log application has applied all the operation logs read in the current period to the local storage of the read node, a broadcast wakes up all threads in the waiting application queue, so that the awakened threads can perform the data reading operation of the read operation request based on the latest local storage of the read node.

2. The data consistency guarantee method for a distributed file system according to claim 1, characterized in that, The centralized log application is an independent background thread running within the read node, and serves as the sole functional entity within the read node responsible for performing log reading and log application.

3. The data consistency guarantee method for a distributed file system according to claim 1, characterized in that, The centralized log application works in a round-robin fashion, and each round includes a batch reading phase and a batch application phase. During the batch reading phase, the centralized log application reads the newly added operation logs sequentially, starting from the end position of the previous round. During the batch application phase, the centralized log application application unit applies the newly added operation logs read in the current round to the local storage of the read nodes in sequence.

4. The data consistency guarantee method for a distributed file system according to claim 1, characterized in that, The dynamic batch read threshold for the batch read operation log from the write node meets the following conditions: Where T represents the dynamic batch read threshold, T min This is the preset minimum batch read threshold. This represents the floor function, where λ is the preset base threshold coefficient, L is the total number of newly added operation logs, and N is the number of read nodes in the cluster. base P is the baseline value for the total amount of operation logs. base F is the baseline value for the failure probability. max R represents the maximum probability of failure for each read node within a historical unit of time, and R is the current log write rate of the write node. base ε is the baseline value for write speed, ε is the preset minimum value to prevent the denominator from being zero, and α, β, and γ are preset weighting coefficients, respectively.

5. The data consistency guarantee method for a distributed file system according to claim 1, characterized in that, Step S1 is replaced by the following steps: Construct a write-multiple-read architecture; the write-multiple-read architecture includes a write node with unique write permissions and multiple read nodes, the write node and each read node share access to the same shared storage layer; the write node is locally configured with a B-tree data structure, and each read node is locally configured with a copy of the B-tree data structure; When the write node performs a file system write operation, the write operation content is stored in the shared storage layer. At the same time, the write operation content is converted into an operation log for the local B-tree data structure of the write node, and the operation log is synchronized to each read node through a centralized log application pre-deployed locally on each read node. A globally unique log sequence number is assigned to each operation log.

6. The data consistency guarantee method for a distributed file system according to claim 5, characterized in that, The awakened thread performs data read operations based on the latest local storage of the read node, including: The latest B-tree data structure stored locally on the read node is used as an index to locate the target data corresponding to the awakened thread in the shared storage layer; The located target data is read and fed back to the corresponding user terminal.

7. The data consistency guarantee method for a distributed file system according to claim 5, characterized in that, The operation log for the local B-tree data structure of the write node refers to the record of insertion, deletion, or modification operations on the local B-tree data structure of the write node.

8. A non-transitory computer-readable storage medium, wherein the storage medium stores at least one instruction or at least one program segment, characterized in that, The at least one instruction or the at least one program segment is loaded and executed by the processor to implement the data consistency guarantee method of the distributed file system as described in any one of claims 1-7.

9. An electronic device, characterized in that, Includes a processor and the non-transitory computer-readable storage medium as described in claim 8.

Images