Data acquisition component optimization method based on distributed storage all-in-one machine, medium and equipment

Abstract

The invention provides a data acquisition component optimization method based on a distributed storage all-in-one machine, a medium and equipment. A distributed lock and version control combined method is adopted. Specifically, the distributed lock mechanism can ensure that only one node performs write operation on certain data at the same moment, so that data conflict and write competition are avoided. In the version control, a version number is added to each piece of data, so that the system can detect and solve conflicts occurring in the data synchronization process. When one node prepares to write data, whether the version number of the data is consistent with the latest version or not is checked firstly, if not, the data of the latest version needs to be acquired again, and the data is written after being merged and changed. Therefore, the data consistency is ensured, and the stability and reliability of the system are improved. Through the method, the problem of data consistency in the optimization process of the data acquisition component can be effectively solved, so that the overall performance of the distributed storage all-in-one machine is improved.

Description

Technical Field

[0001] The present invention belongs to the field of data acquisition of distributed storage all-in-one machines, and in particular relates to a data acquisition component optimization method, medium and equipment based on a distributed storage all-in-one machine. Background Art

[0002] The application background of data acquisition component optimization technology in distributed storage integrated machine stems from the growing demand for efficient and reliable data processing in the era of big data. With the rapid development of information technology, the amount of data in various industries has increased exponentially, and the traditional single storage and processing methods can no longer meet the needs of modern data processing. Therefore, distributed storage integrated machine came into being. By distributing storage and computing resources on multiple nodes, parallel processing and efficient storage of data are achieved. However, an important challenge facing distributed storage systems is how to efficiently collect, transmit and process data to ensure high performance and high reliability of the system.

[0003] In this context, data acquisition component optimization technology is particularly important. Traditional data acquisition methods often have problems such as low transmission efficiency, high latency, and data loss, making it difficult to cope with the real-time processing needs of large-scale data. To this end, researchers have optimized data acquisition components by introducing parallel processing, multi-threading technology, and advanced data compression algorithms, significantly improving the efficiency and reliability of data transmission. Parallel processing technology allows the system to process multiple data streams simultaneously, avoiding the bottleneck problem of a single data stream, while multi-threading technology increases the speed of data acquisition by reasonably allocating system resources. In addition, the application of data compression algorithms reduces the volume of data transmission, speeds up data transmission, and reduces network bandwidth consumption.

[0004] With the continuous development of optimization technology, data acquisition components have not only been significantly improved in performance, but also have stronger adaptability and flexibility. Modern distributed storage systems usually need to process data of different types and formats. The optimized data acquisition components can automatically identify and adapt to different data types for efficient processing and storage. In addition, optimization technology also introduces intelligent monitoring and dynamic adjustment mechanisms, which can monitor various parameters in the data transmission process in real time, discover and solve potential problems in a timely manner, and ensure that the system can still run stably under high load.

[0005] In summary, the application background of data acquisition component optimization technology in distributed storage all-in-one machines is to meet the challenges of efficient and reliable data processing in the big data era. By introducing advanced technical means, the data acquisition process is fully optimized, which not only improves the efficiency and reliability of data transmission, but also enhances the adaptability and stability of the system. The application of this technology not only meets the needs of modern data processing, but also lays a solid foundation for the further development of distributed storage systems.

[0006] However, data consistency issues still exist in existing applications. Since data is transmitted and stored between multiple nodes, data synchronization may be delayed or inconsistent during the optimization process. This problem mainly manifests itself in that when multiple nodes perform data read and write operations at the same time, some nodes may experience data lag or data loss, which affects data consistency. This not only affects the accuracy of the data, but may also cause errors in the system when performing critical tasks, reducing the reliability of the system and the trust of users. Summary of the invention

[0007] In view of the deficiencies in the prior art, the present invention provides a method, medium and device for optimizing a data acquisition component based on a distributed storage all-in-one machine. The present invention adopts a method combining distributed locks and version control. The distributed lock mechanism can ensure that only one node performs a write operation on a certain data at the same time, thereby avoiding data conflicts and write competition. Version control enables the system to detect and resolve conflicts that occur during data synchronization by adding a version number to each copy of data. When a node is ready to write data, it first checks whether the version number of the data is consistent with the latest version. If not, it is necessary to re-acquire the latest version of the data, merge the changes and then write. This not only ensures data consistency, but also improves the stability and reliability of the system. Through this method, the data consistency problem that occurs during the optimization of the data acquisition component can be effectively solved, thereby improving the overall performance of the distributed storage all-in-one machine.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] In a first aspect, the present invention provides a method for optimizing a data acquisition component based on a distributed storage integrated machine, comprising:

[0010] When a node of the distributed storage machine needs to write data, it sends a lock request to the distributed lock service.

[0011] After receiving lock requests from multiple nodes, the lock service processes the requests in the order in which they were received;

[0012] The lock service allocates the lock to the node with the highest request in the queue and records the lock holding information;

[0013] After the node successfully acquires the lock, it obtains exclusive access to the data and performs data write operations;

[0014] After the data write operation is completed, the node sends a request to release the lock to the lock service. After receiving the request, the lock service reallocates the lock resource to the next requesting node in the queue and notifies the relevant nodes that the lock has been released.

[0015] Optionally, the lock request includes the identity information of the node and the identifier of the data to be locked, and each lock request has a unique identifier for distinguishing requests from different nodes.

[0016] Optionally, after receiving lock requests from multiple nodes, if the lock for the specific data requested by the node is not currently held by any other node, the lock service will immediately grant the lock to the first node that sends the lock request.

[0017] Optionally, after the lock service receives lock requests from multiple nodes, if the lock of the specific data requested by the node is already held by other nodes, the node in the request enters a waiting state, and the lock service records the lock requests of all waiting nodes and arranges them in order; when the node holding the lock releases the lock, the lock service grants the lock to the next waiting node in order.

[0018] Optionally, when the node is in the waiting state, it continuously communicates with the lock service to confirm the state change of the lock; once the lock service confirms that the node currently holding the lock has released the lock, it immediately notifies the next waiting node that the lock has been assigned to the node.

[0019] Optionally, before performing a write operation, the node reads a version number or a hash value of the data to determine whether the data has been modified by other nodes after the node acquires the lock.

[0020] Optionally, after confirming that the data status is correct, the node performs an actual write operation; the write operation includes updating, inserting and deleting data.

[0021] Optionally, after the write operation is completed, the node updates the state of the data, including increasing the version number of the data or updating the timestamp, so that other nodes can identify the latest data in subsequent operations.

[0022] In a second aspect, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the data acquisition component optimization method based on a distributed storage integrated machine as described in the first aspect.

[0023] In a third aspect, the present invention provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for optimizing data acquisition components based on a distributed storage integrated machine as described in the first aspect is implemented.

[0024] The beneficial effects of the present invention are:

[0025] 1. The present invention uses a simple timestamp or logical clock mechanism to solve the problem of data consistency in traditional methods, but cannot completely avoid data conflicts and competition, resulting in unstable system performance. The present invention uses a method combining distributed locks and version control to ensure that only one node writes data at the same time through a distributed lock mechanism, thereby avoiding data conflicts.

[0026] 2. By adding a version number to each piece of data, the system can detect and resolve conflicts that occur during data synchronization. If the data versions are inconsistent, the node will re-acquire the latest version of the data and merge it before writing it, thereby ensuring data consistency and improving system stability and reliability. The present invention effectively solves the data consistency problem that occurs during the optimization of data acquisition components and significantly improves the overall performance of the distributed storage all-in-one machine. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of a method for optimizing a data acquisition component based on a distributed storage all-in-one machine.

[0028] Figure 2 It is a flow chart of the improved C-RAFT algorithm. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0030] In one embodiment, the present invention proposes a data acquisition component optimization method based on a distributed storage integrated machine, such as Figure 1 As shown in the figure, a method combining distributed locks and version control is adopted to avoid data conflicts and competitions by ensuring the exclusivity of data write operations, thereby ensuring data consistency and system stability. The specific implementation process and its working principle are as follows.

[0031] 1. Obtain a distributed lock.

[0032] In a distributed storage system, in order to ensure that only one node can write specific data at a time, a distributed lock service is needed to synchronize data write operations. These services can ensure that when multiple nodes process data in parallel, data conflicts are avoided through the lock mechanism to ensure data consistency and integrity. In the specific implementation process, when a node is ready to write data, it first needs to send a request to the distributed lock service to apply for the data lock.

[0033] First, when a node is ready to write data, it sends a request to the distributed lock service. This request clearly indicates that the node needs to write specific data. After the request is sent to the lock service, the lock service will process the request. In the distributed lock service, each request has a unique identifier, which is used to distinguish requests from different nodes.

[0034] After receiving the request, the distributed lock service will allocate the locks in the order of the requests. If the lock for the specific data is not currently held by any other node, the lock service will immediately grant the lock to the first node that sent the request. After obtaining the lock, this node can write the data. After the write operation is completed, the node needs to actively release the lock so that other nodes can continue to obtain the lock and perform the corresponding operation.

[0035] If the lock service finds that the lock of the specific data is already held by other nodes when receiving the request, the node in the request needs to enter a waiting state. The lock service will record all node requests waiting to obtain the lock and arrange them in order. When the node holding the lock releases the lock, the lock service will grant the lock to the next waiting node in order. In this way, the fair distribution of locks is ensured and possible resource competition and data conflicts are avoided.

[0036] While waiting for the lock, the node needs to continuously communicate with the lock service to confirm the lock status change. Once the lock service confirms that the node currently holding the lock has released the lock, the lock service will immediately notify the next waiting node that the lock has been assigned to it. After receiving the lock assignment notification, the node can start writing data.

[0037] Through such a distributed lock mechanism, it can be ensured that in a distributed storage system, only one node can write specific data at any time, thereby effectively avoiding data conflicts. In addition, the design of distributed lock services is usually highly available and fault-tolerant, so that the entire system can still operate normally even when some nodes fail. This is mainly due to the fact that the distributed lock service uses mechanisms such as master-slave replication and data consistency algorithms to ensure that the lock status can be consistent across multiple nodes.

[0038] 2. Release the distributed lock.

[0039] In a distributed storage system, when a node successfully acquires a distributed lock, it can safely perform data write operations. The specific steps of data write operations include checking the current data status, performing write operations, and updating the data status. These steps ensure the accuracy and consistency of the data and prevent data conflicts.

[0040] First, before a node performs a write operation, it must check the current state of the data. To achieve this, the node usually reads the version number or hash value of the data. This information can help the node determine whether the data has been modified by other nodes after it acquires the lock. In this way, the node can ensure that the data it operates is the latest, thereby avoiding errors caused by operating old data.

[0041] After confirming that the data status is correct, the node can perform actual write operations. The specific content of the write operation may include updating, inserting or deleting data. The specific types of these operations depend on business needs. In the case of data update, the node will write the new data value to the specified location; in the case of data insertion, the node will add the new data to the data collection; in the case of data deletion, the node will remove the specified data from the data collection. Regardless of the type of write operation, the node needs to ensure the accuracy and integrity of the operation.

[0042] After the write operation is completed, the node needs to update the state of the data. This step includes increasing the version number of the data or updating the timestamp so that other nodes can identify the latest data in subsequent operations. Increasing the version number or updating the timestamp helps to identify the latest state of the data and prevent other nodes from using old data in subsequent operations. In this way, the node can ensure the consistency and correctness of the data.

[0043] The distributed lock mechanism plays a vital role in the entire data writing process. Distributed locks ensure the exclusivity of write operations, that is, at a certain moment, only one node can modify specific data. This design effectively prevents multiple nodes from modifying the same data at the same time, thereby preventing data conflicts and inconsistencies. This guarantee of exclusivity enables the distributed storage system to maintain data consistency and integrity in a high-concurrency environment.

[0044] Here, the present invention also proposes an improved RAFT algorithm process, such as Figure 2 As shown, the data acquisition component used to optimize the distributed storage integrated machine can solve the data consistency problem and improve the system performance and reliability.

[0045] The C-RAFT (Consensus-RAFT) algorithm is an improvement on the RAFT consensus algorithm, focusing on optimizing leader election, log replication, and lease mechanisms. Its goal is to provide higher performance and reliability in high concurrency and high failure rate environments. The specific steps are as follows:

[0046] <1> System initialization.

[0047] In the C-RAFT algorithm, system initialization is the first step of improvement. All nodes are initialized to the follower state at startup and a random election timeout is generated. This time is usually between 150 milliseconds and 300 milliseconds. This randomized election timeout setting avoids multiple nodes initiating elections at the same time, thereby reducing the probability of election conflicts and ensuring the efficiency of the election process and the stability of the system.

[0048] <2> Leader election.

[0049] When the election of a follower node times out, it immediately becomes a candidate and sends a voting request to all other nodes. The voting request contains the current term, the candidate's node ID, the index of the last log entry, and the term information. After receiving the voting request, other nodes will check their own log index and term. If the candidate's log is newer than itself and the term is not less than the current term, vote for the candidate immediately. This fast election confirmation mechanism ensures that the leader can be selected in the shortest time, reducing the system downtime during the election.

[0050] <3> Parallel log replication.

[0051] After the leader election is completed, the new leader is responsible for processing client requests and recording log entries. In order to improve the efficiency of log replication, the leader adopts a parallel log replication strategy. Specifically, the leader sends log entries to all follower nodes in parallel through multithreading. While waiting for the follower nodes to confirm the log entries, the leader can continue to process new client requests. This parallel processing method not only improves the efficiency of log replication, but also ensures high throughput and low latency of the system.

[0052] <4> Incremental log replication.

[0053] The C-RAFT algorithm also introduces an incremental log replication mechanism. When the leader detects that a follower node's log is lagging behind, it only sends the missing log entries of the node instead of sending all logs from the beginning. This incremental log replication strategy greatly reduces network communication overhead, speeds up log synchronization, and ensures efficient operation of the system in a high-concurrency environment.

[0054] <5> Dynamic lease management.

[0055] In order to enhance the flexibility and efficiency of lock management, the C-RAFT algorithm has made improvements to the lease mechanism. When processing client lock requests, the leader will dynamically adjust the lease time according to the current system load and client needs, and record the lease information in the log to ensure that all follower nodes can synchronize this information. When the lock held by the client is about to expire, the leader automatically sends a renewal prompt to the client. The client sends a renewal request before the lease expires, and the leader updates the lease time and replicates the change to all follower nodes. This automatic renewal mechanism avoids deadlock problems and improves system reliability.

[0056] <6> Fault detection and recovery.

[0057] The C-RAFT algorithm introduces a pre-election mechanism to improve the system's fault detection and recovery capabilities. During normal operation, the leader periodically sends pre-election messages to all nodes to confirm the health status of the current leader. If a node does not receive a pre-election message for a long time, it will actively ask other nodes about the status of the current leader and initiate a new election after confirming that the leader has failed. This pre-election mechanism ensures that the system can quickly select a new leader after a leader fails, reducing system downtime.

[0058] <7> Hierarchical lock management.

[0059] In order to improve the scalability and concurrency of the distributed lock mechanism, the C-RAFT algorithm adopts a hierarchical lock management strategy. The system divides lock management into global locks, regional locks, and local locks. The global lock is responsible for lock management across the entire system to ensure the consistency of global data. The regional lock is responsible for lock management within a specific area to reduce the load of the global lock service. The local lock is responsible for lock management within a single node to improve the concurrency of operations within the node. Through hierarchical lock management, the load of a single lock service can be effectively reduced, and the scalability and concurrency of the system can be improved.

[0060] Leader election is the core of the RAFT algorithm. In order to improve the efficiency of the election, the C-RAFT algorithm introduces the following optimization measures: When the randomized election times out, each node generates a random election timeout at initialization to avoid most nodes initiating elections at the same time and reduce the probability of election conflicts. After receiving the candidate's voting request, the node immediately checks its own log index and term. If the candidate's log is newer than its own and its term is not less than the current term, it immediately votes for the candidate. After receiving the votes of most nodes, the candidate node immediately becomes the leader and sends a leader confirmation message.

[0061] To improve the efficiency of log replication, the C-RAFT algorithm adopts the following strategy: after the leader receives the client request and records the log entry, it immediately sends the log entry to all follower nodes in parallel. While waiting for the follower node to confirm, the leader continues to process new client requests. During the subsequent incremental log replication, when the leader detects that the log of a follower node is lagging behind, it only sends the missing log entries of the node instead of sending all the logs from the beginning. This strategy not only speeds up log replication, but also reduces network traffic.

[0062] The lease mechanism is an important means to improve the flexibility and efficiency of lock management. The C-RAFT algorithm has made the following improvements to the lease mechanism: within the dynamic lease time adjustment, when the leader processes the client lock request, it dynamically adjusts the lease time according to the current system load and client demand, and records the lease information to ensure that all follower nodes synchronize the information. The automatic renewal mechanism is that when the lock held by the client is about to expire, the leader automatically sends a renewal prompt to the client. The client sends a renewal request before the lease expires, and the leader updates the lease time and replicates it to all follower nodes.

[0063] During normal operation, the leader periodically sends pre-election messages to all nodes to confirm the health status of the current leader. If a node does not receive a pre-election message for a long time, it will actively ask other nodes about the status of the current leader and initiate an election after confirming that the leader is invalid.

[0064] In order to further improve the performance of the distributed lock mechanism, the C-RAFT algorithm introduces a hierarchical lock management strategy. The system divides lock management into global locks, regional locks, and local locks: Global locks are responsible for lock management across the entire system to ensure global data consistency. Regional locks are responsible for lock management within a specific area to reduce the load on the global lock service. Local locks are responsible for lock management within a single node to improve the concurrency of operations within the node.

[0065] System initialization is when all nodes are started and initialized to the follower state, a random election timeout is generated, and a timer countdown is started. Leader election is when a follower node's election times out, it immediately becomes a candidate, sends a voting request to all nodes, and attaches its own log index and term information. Election confirmation requires other nodes to check their own log index and term immediately after receiving the voting request. If the candidate's log is newer than its own and its term is not less than the current term, then vote for the candidate. Leader recommendation requires the candidate node to become the leader immediately after receiving the votes of most nodes, and send a leader confirmation message to all nodes. During log replication, after receiving the client request, the leader records the log entry and sends it to all follower nodes in parallel, and waits for the confirmation of most follower nodes before committing the log and updating the status. For lease management, when allocating locks, the leader dynamically adjusts the lease time according to the current system load and client demand, and records the lease information. When the lease is about to expire, it automatically sends a renewal reminder to the client, and after receiving the renewal request, it updates the lease time and replicates it to all follower nodes. During the operation of the system, the leader periodically sends pre-election messages to all nodes to confirm the health status of the leader. If a node does not receive the pre-election message for a long time, it will actively ask the status of the leader of other nodes and initiate an election after confirming that the leader is invalid.

[0066] A distributed consensus algorithm based on the RAFT protocol is used to reduce the delay in acquiring and releasing locks. Reasonable lock timeouts are set according to specific business needs and system performance to avoid long-term occupation of lock resources. By dividing lock management into multiple levels, the load on a single lock service is reduced, improving the scalability and concurrency of the system.

[0067] In order to implement a distributed consensus algorithm based on the RAFT protocol to optimize the lock acquisition and release delays, reasonably set the lock timeout, and reduce the load of a single lock service through hierarchical lock management to improve the scalability and concurrency of the system, the following is a detailed text description of the specific operation plan and parameter configuration.

[0068] Implementation of distributed consensus algorithm based on RAFT protocol: In a distributed system, all nodes are started and initialized as followers, and a random election timeout is generated to avoid most nodes initiating elections at the same time and reduce the probability of election conflicts. The election timeout of each node is generally set between 150 milliseconds and 300 milliseconds. Randomized election timeout can avoid large-scale conflicts during the election process and ensure the stability of the system.

[0069] When the election of a follower node times out, it immediately becomes a candidate and sends a voting request to all other nodes. The voting request includes the current term, the candidate's node ID, the index of the last log entry, and term information. After receiving the voting request, other nodes will check their own log index and term. If the candidate's log is newer than itself and the term is not less than the current term, vote for the candidate immediately. This fast election confirmation mechanism ensures the efficiency of the election process and reduces the time required for elections.

[0070] After the leader election is completed, the new leader is responsible for processing client requests and recording log entries. In order to improve the efficiency of log replication, the leader will adopt a parallel log replication strategy, that is, sending log entries to all follower nodes in parallel through multi-threading. The leader can continue to process new client requests while waiting for confirmation from the follower nodes. In addition, when the leader detects that the log of a follower node is lagging behind, it only sends the missing log entries of the node instead of sending all logs from the beginning. This incremental log replication strategy not only speeds up log replication, but also effectively reduces network traffic and improves the overall performance of the system.

[0071] Lock timeout setting: In the distributed lock mechanism, in order to prevent the lock resources from being occupied for a long time, the lock timeout needs to be set reasonably. Lock timeout means that after the lock is held for a certain period of time, the system automatically releases the lock to ensure efficient use of lock resources. According to specific business needs and system performance, the lock timeout can be set to 1000 milliseconds. When the node successfully acquires the lock, it records the current timestamp and automatically releases the lock when the timeout period arrives. If the node fails to complete the operation within the timeout period, it needs to reapply for the lock. This mechanism can effectively prevent system performance degradation caused by node failure or long-term occupation of lock resources.

[0072] Hierarchical lock management: To improve the scalability and concurrency of the distributed lock mechanism, a hierarchical lock management strategy can be adopted. Hierarchical lock management reduces the load of a single lock service by dividing lock management into global locks, regional locks, and local locks.

[0073] The global lock is responsible for lock management across the entire system to ensure the consistency of global data. The regional lock is responsible for lock management within a specific area, reducing the load on the global lock service. The local lock is responsible for lock management within a single node, improving the concurrency of operations within the node. Through hierarchical lock management, the load on a single lock service can be effectively reduced, and the scalability and concurrency of the system can be improved. For example, when a node needs to acquire a lock for a resource, it first applies for a global lock from the global lock service, then applies for a regional lock from the regional lock service, and finally applies for a local lock in the local node. This ensures that the lock acquisition process is efficient and orderly, reducing the load on the lock service.

[0074] Parameter configuration and tuning: To ensure efficient operation of the system, parameter configuration and tuning are required according to the actual application environment. The election timeout is generally set between 150 milliseconds and 300 milliseconds to ensure fast leader election under network delays. The lock timeout is set to 1000 milliseconds to avoid long-term lock resource occupation. The number of parallel threads for log replication is adjusted according to the system load and network bandwidth to ensure the efficiency of parallel replication. Through reasonable parameter configuration and tuning, the performance and reliability of the system can be further improved.

[0075] By introducing a distributed consensus algorithm based on the RAFT protocol, the acquisition and release delays of distributed locks can be significantly optimized. Reasonable setting of lock timeouts can prevent lock resources from being occupied for a long time. Hierarchical lock management can reduce the load of a single lock service and improve the scalability and concurrency of the system. These optimization measures ensure the stable operation of distributed storage systems in high concurrency and high failure rate environments, meeting the requirements of modern distributed systems for high performance and high reliability.

[0076] By introducing the C-RAFT algorithm, the performance and reliability of the data collection component of the distributed storage machine can be significantly improved. The C-RAFT algorithm effectively solves the data consistency problem by optimizing the leader election, log replication, and lease mechanisms, and improves the system's concurrency performance and fault tolerance. The hierarchical lock management strategy further improves the scalability of the system, enabling it to operate stably in high-concurrency and high-failure rate environments. These innovative measures provide a solid foundation for the development of distributed storage systems.

[0077] 3. Performance optimization.

[0078] In order to further improve the performance of the distributed lock mechanism, a variety of optimization measures can be adopted, including optimizing the lock service implementation, setting the lock timeout reasonably, adopting the lease mechanism and hierarchical lock management. These optimization measures can effectively improve the performance and reliability of distributed locks and ensure the stable operation of distributed storage systems in high-concurrency environments.

[0079] The use of a lease mechanism is also an important way to improve the performance of distributed locks. The lease mechanism improves the reliability and performance of the system by ensuring the validity of the lock within a specific time. Under the lease mechanism, the lock holder obtains a lease when acquiring the lock, which specifies the validity period of the lock. During the lease period, the lock holder can operate on the data, and when the lease period expires, the lock is automatically released.

[0080] First, when a node is ready to acquire a distributed lock, it needs to request a lease from the lock service. This request includes not only the application for the lock, but also the duration of the lease, that is, the time the lock is held. In the distributed lock service, each lock request is accompanied by a lease time, which indicates the validity of the lock during the lease time. After receiving the request, the lock service will decide whether to grant the lease based on the current state of the lock. If the lock is currently idle, the lock service will grant the lock to the requesting node and assign a lease at the same time. During the lease time, the node has exclusive access to the data.

[0081] Once a node obtains a lock and its lease, it can perform data operations during the validity period of the lease. During this period, the node needs to periodically send renewal requests to the lock service to extend the validity period of the lease. The frequency and time interval of renewal requests need to be adjusted according to the specific system settings, and renewal requests are usually initiated some time before the lease expires. After receiving the renewal request, the lock service will update the validity period of the lease to ensure that the node continues to hold the lock. During the renewal process, if the node fails to send a renewal request in time, or the renewal request fails due to network problems, the lease may expire, triggering the lease expiration handling mechanism.

[0082] Lease expiration is a key link in the lease mechanism. When the lock service finds that a lease has expired and no renewal request has been received, it will automatically release the lock. At this time, the lock service will mark the lock status as available and notify other nodes waiting to acquire the lock. In this way, the system can automatically recycle lock resources that have been occupied for a long time to prevent deadlock problems caused by node failures or network problems. In addition, the lock service also needs to maintain a timeout mechanism to regularly check the status of each lease to ensure that expired locks are released in time.

[0083] In terms of lease expiration processing, when the lease expires or a node failure causes the lease to expire, the lock service needs to take corresponding measures to ensure the stability of the system and the consistency of the data. First, the lock service needs to notify other nodes waiting to acquire the lock so that they can try to reacquire the lock. Second, the lock service needs to ensure that during the process of lease expiration, no multiple nodes hold the same lock at the same time. To this end, the lock service usually designs a lock retry mechanism so that nodes can reapply for locks in an orderly manner after the lease expires to ensure the exclusivity of the lock.

[0084] By introducing the lease mechanism, the distributed lock service can effectively manage the life cycle of the lock and improve the fault tolerance and performance of the system. In practical applications, the lease mechanism can not only prevent deadlock problems, but also improve the concurrency and reliability of the system. In the specific implementation process, it is necessary to reasonably set and optimize the lease time, renewal frequency and timeout mechanism in combination with the specific needs and environment of the system.

[0085] The lease mechanism is widely used in distributed systems. In addition to distributed locks, it can also be used in scenarios such as distributed caching and consistency maintenance. In distributed caching systems, the lease mechanism can ensure the validity and consistency of cached data and prevent cached data from becoming outdated or invalid. In consistency maintenance scenarios, the lease mechanism can help the system maintain data consistency between different nodes and prevent data inconsistency problems.

[0086] In the actual deployment and operation process, the effectiveness of the lease mechanism also needs to be ensured through monitoring and tuning. System administrators need to regularly check the operating status of the lock service, analyze the usage of the lease, and make adjustments and optimizations based on actual conditions. In addition, through log and monitoring tools, the status and renewal of the lease can be monitored in real time, and possible problems can be discovered and handled in a timely manner to ensure the stable operation of the system.

[0087] In large-scale distributed systems, adopting a hierarchical lock management strategy is an effective way to improve the performance of the lock mechanism. Hierarchical lock management reduces the load of a single lock service by dividing lock management into multiple levels, thereby improving the scalability and concurrency of the system. Under the hierarchical lock management strategy, the system divides lock management into different levels, such as global locks, regional locks, and local locks. Global locks are responsible for lock management across the entire system, regional locks are responsible for lock management within a specific area, and local locks are responsible for lock management within a single node. This hierarchical management method can effectively reduce the load of a single lock service and improve the scalability and concurrency performance of the system.

[0088] The performance and reliability of the distributed lock mechanism can be significantly improved by optimizing lock service implementation, setting lock timeouts appropriately, adopting lease mechanisms and hierarchical lock management, etc. These optimization measures can ensure the stable operation of the distributed storage system in a high-concurrency environment, improve the scalability and concurrency performance of the system, and thus meet the requirements of modern distributed applications for high performance, high availability, and high reliability.

[0089] In another embodiment, the present invention proposes a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the data acquisition component optimization method based on a distributed storage integrated machine as described in the aforementioned embodiment.

[0090] In another embodiment, the present invention proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the data acquisition component optimization method based on the distributed storage integrated machine as described in the above embodiment is implemented.

[0091] In the embodiments disclosed in the present application, the computer storage medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. The computer storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of computer storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0092] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0093] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A method for optimizing data acquisition components based on a distributed storage integrated machine, characterized in that: include: When a node of the distributed storage machine needs to write data, it sends a lock request to the distributed lock service. After receiving lock requests from multiple nodes, the lock service processes the requests in the order in which they were received; The lock service allocates the lock to the node with the highest request in the queue and records the lock holding information; After the node successfully acquires the lock, it obtains exclusive access to the data and performs data write operations; After the data write operation is completed, the node sends a request to release the lock to the lock service. After receiving the request, the lock service reallocates the lock resource to the next requesting node in the queue and notifies the relevant nodes that the lock has been released.

2. The method for optimizing data acquisition components based on a distributed storage integrated machine according to claim 1, characterized in that: The lock request includes the identity information of the node and the identifier of the data to be locked. Each lock request has a unique identifier to distinguish requests from different nodes.

3. The method for optimizing data acquisition components based on a distributed storage integrated machine according to claim 1, characterized in that: After receiving lock requests from multiple nodes, if the lock for the specific data requested by the node is not currently held by any other node, the lock service will immediately grant the lock to the first node that sends the lock request.

4. The method for optimizing data acquisition components based on a distributed storage integrated machine according to claim 1, characterized in that: After the lock service receives lock requests from multiple nodes, if the lock of the specific data requested by the node is already held by other nodes, the node in the request enters a waiting state, and the lock service records the lock requests of all waiting nodes and arranges them in order; when the node holding the lock releases the lock, the lock service grants the lock to the next waiting node in order.

5. The method for optimizing data acquisition components based on a distributed storage integrated machine according to claim 4, characterized in that: When the node is in the waiting state, it continuously communicates with the lock service to confirm the state change of the lock; once the lock service confirms that the node currently holding the lock has released the lock, it immediately notifies the next waiting node that the lock has been assigned to the node.

6. The method for optimizing data acquisition components based on a distributed storage integrated machine according to claim 1, characterized in that: Before performing a write operation, the node reads the version number or hash value of the data to determine whether the data has been modified by other nodes after the node acquires the lock.

7. The method for optimizing data acquisition components based on a distributed storage integrated machine according to claim 1, characterized in that: After confirming that the data status is correct, the node performs actual write operations; the write operations include updating, inserting and deleting data.

8. The method for optimizing data acquisition components based on a distributed storage integrated machine according to claim 1, characterized in that: After the write operation is completed, the node updates the state of the data, including increasing the version number of the data or updating the timestamp, so that other nodes can identify the latest data in subsequent operations.

9. A computer-readable storage medium storing a computer program, characterized in that: The computer program enables the computer to execute the data acquisition component optimization method based on a distributed storage all-in-one machine as described in any one of claims 1-8.

10. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for optimizing a data acquisition component based on a distributed storage integrated machine as described in any one of claims 1 to 8 is implemented.

Images