System operating methods, apparatus, equipment, storage media and program products

Abstract

This application relates to a system operation method, apparatus, device, storage medium, and program product. The method includes: upon determining that a first service node cannot continue running a distributed file system, sending an operation instruction to a second service node, instructing the second service node to obtain first file operation data from a management node, thereby loading and running the distributed file system on the second service node; upon receiving an acquisition request from the second service node, sending the first file operation data to the second service node; the first file operation data is uploaded to the management node by the first service node during the operation of the distributed file system; the operation instruction further instructs the second service node to determine, based on the first file operation data, the target shared storage space storing the target operation data corresponding to the first file operation data and request the closure of the target shared storage space. This method can avoid file data access errors.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a system operation method, apparatus, device, storage medium, and program product. Background Technology

[0002] A distributed file system allows clients to access data files over a network. Typically, a distributed file system runs on a single server node, but to improve system reliability, multiple server nodes can be configured. This allows the distributed file system to be switched to another server node if the current one becomes unavailable. When the server node running the distributed file system changes, all clients need to be notified of the new server node so that each client can access the distributed file system's files through the new server node next time.

[0003] However, if there are too many clients, the process of notifying all clients of the new service node takes some time. At the same time, some clients may believe that the distributed file system is running on the original service node, while others may believe that the distributed file system is running on the new service node. In this case, if both types of clients access the same file data in the distributed file system at the same time, it will lead to an access error. Summary of the Invention

[0004] Therefore, it is necessary to provide a system operation method, apparatus, device, storage medium, and program product that can avoid file data access errors in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a system operation method. For managing nodes, the method includes:

[0006] If it is determined that the first service node cannot continue to run the distributed file system, a run instruction is sent to the second service node. The run instruction is used to instruct the second service node to obtain the first file running data from the management node, and to load the distributed file system on the second service node according to the first file running data and then run the distributed file system.

[0007] Upon receiving a request from the second service node, the system sends the first file execution data to the second service node; the first file execution data is uploaded to the management node by the first service node during the operation of the distributed file system.

[0008] The execution instruction also instructs the second service node, after the distributed file system is successfully loaded, to determine the target shared storage space for storing the target operation data corresponding to the first file's execution data based on the first file's execution data, and to request the closure of the target shared storage space.

[0009] In one embodiment, sending a run instruction to the second service node includes:

[0010] If it is determined that the first service node cannot continue to run the distributed file system, the current logical clock is determined and used as the target logical clock; an execution instruction with the target logical clock is generated according to the target logical clock; the execution instruction is sent to the second service node; wherein, if the second file execution data is generated during the operation of the distributed file system on the second service node, the target logical clock is used for the second service node to bind the second file execution data with the target logical clock and send it to the management node.

[0011] In one embodiment, the method further includes:

[0012] The system receives the latest first file running data and the first logical clock sent by the first service node; the latest first file running data is used to request the management node to store the latest first file running data; if it is determined that the first logical clock is not the target logical clock, the system refuses to store the latest first file running data and sends a stop command to the first service node; wherein, the stop command is used to instruct the first service node to stop running the distributed file system.

[0013] In one embodiment, the method further includes:

[0014] Receive a first notification message sent by the second service node; if the first notification message indicates that the second service node has successfully loaded the distributed file system, send a system view to the client and the first service node; wherein the system view is used to describe at least the distributed file system currently running on the second service node.

[0015] In one embodiment, the system view is used to instruct the first service node to stop running the distributed file system upon receiving the system view.

[0016] In one embodiment, the first file operation data includes at least one of a first file list and a second file list; the first file list includes multiple first block identifiers, and the block shared space corresponding to the first block identifier stores at least one file operation information; the second file list includes multiple second block identifiers, and the block shared space corresponding to the second block identifier stores at least one data snapshot.

[0017] Secondly, this application also provides a system operating apparatus. For managing nodes, the apparatus includes:

[0018] The first sending module is used to send a running instruction to the second service node when it is determined that the first service node cannot continue to run the distributed file system. The running instruction is used to instruct the second service node to obtain the first file running data from the management node, and to load the distributed file system on the second service node according to the first file running data and then run the distributed file system.

[0019] The second sending module is used to send the first file execution data to the second service node upon receiving an acquisition request from the second service node; the first file execution data is uploaded to the management node by the first service node during the process of running the distributed file system;

[0020] The execution instruction also instructs the second service node, after the distributed file system is successfully loaded, to determine the target shared storage space for storing the target operation data corresponding to the first file's execution data based on the first file's execution data, and to request the closure of the target shared storage space.

[0021] In one embodiment, the first sending module is specifically used for:

[0022] If it is determined that the first service node cannot continue to run the distributed file system, the current logical clock is determined and used as the target logical clock; an execution instruction with the target logical clock is generated according to the target logical clock; the execution instruction is sent to the second service node; wherein, if the second file execution data is generated during the operation of the distributed file system on the second service node, the target logical clock is used for the second service node to bind the second file execution data with the target logical clock and send it to the management node.

[0023] In one embodiment, the device further includes a first receiving module for:

[0024] The system receives the latest first file running data and the first logical clock sent by the first service node; the latest first file running data is used to request the management node to store the latest first file running data; if it is determined that the first logical clock is not the target logical clock, the system refuses to store the latest first file running data and sends a stop command to the first service node; wherein, the stop command is used to instruct the first service node to stop running the distributed file system.

[0025] In one embodiment, the device further includes a second receiving module for:

[0026] Receive a first notification message sent by the second service node; if the first notification message indicates that the second service node has successfully loaded the distributed file system, send a system view to the client and the first service node; wherein the system view is used to describe at least the distributed file system currently running on the second service node.

[0027] In one embodiment, the system view is used to instruct the first service node to stop running the distributed file system upon receiving the system view.

[0028] In one embodiment, the first file operation data includes at least one of a first file list and a second file list; the first file list includes multiple first block identifiers, and the block shared space corresponding to the first block identifier stores at least one file operation information; the second file list includes multiple second block identifiers, and the block shared space corresponding to the second block identifier stores at least one data snapshot.

[0029] Thirdly, this application also provides a computer device, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described in any of the first aspects above.

[0030] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first aspects above.

[0031] Fifthly, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the method described in any one of the first aspects above.

[0032] The aforementioned system operation method, apparatus, device, storage medium, and program product, upon determining that the first service node cannot continue running the distributed file system, sends an operation instruction to the second service node. This operation instruction instructs the second service node to obtain first file operation data from the management node, load the distributed file system on the second service node based on the first file operation data, and then run the distributed file system. Upon receiving an acquisition request from the second service node, the system sends the first file operation data uploaded to the management node during the first service node's operation of the distributed file system to the second service node. Furthermore, the operation instruction instructs the second service node, after successfully loading the distributed file system, to determine the target shared storage space storing the target operation data corresponding to the first file operation data based on the first file operation data, and request the closure of the target shared storage space. This allows the second service node to close the target shared storage space upon loading the distributed file system, preventing the first service node from accessing the target shared storage space and thus preventing it from running. This avoids the situation where two service nodes are running the distributed file system simultaneously, preventing file data access errors. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is an application environment diagram of the system operation method in one embodiment;

[0035] Figure 2 This is a flowchart illustrating the system operation method in one embodiment;

[0036] Figure 3 This is a schematic diagram of the service node operation in one embodiment;

[0037] Figure 4 This is a schematic diagram of the architecture for accessing a distributed file system in one embodiment;

[0038] Figure 5 This is a flowchart illustrating the process of sending a run command in one embodiment;

[0039] Figure 6 This is a schematic diagram illustrating the storage of the first file list and the second file list in one embodiment;

[0040] Figure 7 This is a schematic diagram of the data update process in one embodiment;

[0041] Figure 8 This is a flowchart illustrating the process of sending a stop command to the first service node in one embodiment;

[0042] Figure 9 This is a schematic diagram of system switching operation in one embodiment;

[0043] Figure 10 This is a structural block diagram of the system operating device in one embodiment;

[0044] Figure 11 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0045] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be understood that many specific details are set forth in the following description in order to provide a full understanding of this application, but this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0047] A distributed file system allows clients to access data files over a network. Typically, a distributed file system runs on a single server node, but to improve system reliability, multiple server nodes can be configured. This allows the distributed file system to be switched to another server node if the current one becomes unavailable. When the server node running the distributed file system changes, all clients need to be notified of the new server node so that each client can access the distributed file system's files through the new server node next time.

[0048] However, if there are too many clients, the process of notifying all clients of the new service node takes some time. At the same time, some clients may believe that the distributed file system is running on the original service node, while others may believe that the distributed file system is running on the new service node. In this case, if both types of clients access the same file data in the distributed file system at the same time, it will lead to an access error.

[0049] The system operation method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, the management node 101 is connected to multiple service nodes 102 and at least one client 103. Any service node can act as a first service node, and when the first service node is running, one of the other service nodes can act as a second service node. The management node, service nodes, and clients can be services running on computer devices; in other words, the management node, service nodes, and clients can be computer devices. These computer devices can be, but are not limited to, various personal computers, laptops, tablets, or servers, etc. The servers can be implemented using independent servers or a server cluster composed of multiple servers.

[0050] In one embodiment, such as Figure 2 As shown, a system operation method is provided, which can be applied to Figure 1 Taking management node 101 as an example, the following steps are included:

[0051] Step 201: If it is determined that the first service node cannot continue to run the distributed file system, send a run instruction to the second service node.

[0052] Distributed file systems, such as DBFS (Database File System), are not fully illustrated here. The term "file system" will be used in the following text.

[0053] When the file system is running on the first service node, the client can access the file system based on the first service node and process the files in the file system, such as performing open operations, expansion operations, and write operations on the files, without specific limitations.

[0054] Optionally, the first service node can be implemented by StreamFS MDS. StreamFS MDS is the streaming file system metadata service node, responsible for loading the metadata service of the file system namespace, responding to metadata access and update requests transmitted by the StreamFS client, and ensuring the semantics of one-write-many-read. StreamFSClient is the semantic interface SDK for the StreamFS append-only file system, used to map file semantics to shared storage space, providing file access interfaces such as supporting file operations like appending, reading, deleting, and renaming.

[0055] Here, StreamFS is an append-only distributed file system within the storage engine, based on a shared storage subsystem (chunk subsystem). An append-only distributed file system is a special type of distributed file system whose core characteristic is that data can only be written in append-only mode, and does not support traditional in-place modification operations. StreamFS allows clients to connect to a StreamFS client and access resources within StreamFS over the network, providing a write-multiple-read shared access method for the same file. StreamFS provides eventually consistent data access services; in a write-multiple-read scenario, non-local read clients may not be able to read the latest data, but will eventually be able to read the latest data. In the distributed engine, StreamFS, as the underlying file system for the key-value database, leverages its powerful horizontal scaling capabilities to support hundreds of billions of large files, thus enabling ultra-large-scale clusters and distributed key-value databases with hundreds of billions of records.

[0056] In one alternative implementation, the first service node operates on a lease; when the lease expires, the first service node can no longer run the distributed file system.

[0057] In one alternative implementation, when the first service node loses network connection and is disconnected from the management node, the management node considers that the first service node can no longer run the distributed file system.

[0058] Dependency lease expiration is a technique used in distributed systems to ensure data consistency and prevent resource contention. Most distributed file systems rely on lease expiration for operation. In a distributed file system, a lease is a time-limited authorization that allows a holder exclusive access to a resource for a certain period. When the lease expires, if the holder does not renew, the system will automatically reclaim the resource; this is known as "lease expiration."

[0059] However, in distributed systems, when a process is in an "uninterruptible sleep" (D) state while waiting for I / O operations, it may fail to respond immediately to termination signals or requests, resulting in a split-brain scenario and loss of connection with other nodes. For example, due to network or node failures, the first service node may lose connection with other nodes, becoming isolated. This can lead to data inconsistency and service unavailability. Furthermore, in related technologies, to prevent split-brain scenarios in service nodes running distributed file systems, a system restart is required. However, restarting the system requires restarting all services on the system, resulting in high execution costs. Moreover, if containers or virtual machines are used to implement microservices, extreme latency cannot be guaranteed.

[0060] For example, refer to Figure 3 The diagram illustrates the operation of service nodes. Service nodes 1 through 2 are connected to the management node. During the process of service node 1 running the file system, if a network failure occurs, resulting in a network partition, the management node considers the lease of service node 1 invalid and instructs service node 2 to begin loading and running the file system data. The lease system requires that the process of service node 1 must exit at this time. However, in systems such as Linux, when a process is in the D state while waiting for I / O returns, it cannot exit. At this point, both service node 1 and service node 2 are running the file system, and both are accessing the shared storage space.

[0061] In this embodiment, when the management node determines that the first service node can no longer run the distributed file system, it sends a run instruction to the second service node. The run instruction instructs the second service node to obtain the first file running data from the management node, and then load and run the distributed file system on the second service node based on the first file running data.

[0062] The data for the first file operation is uploaded to the management node during the process of the first service node running the distributed file system. This is equivalent to migrating the file system to the second service node for operation. By providing multiple service nodes, the high reliability of the file system operation is ensured.

[0063] Step 202: Upon receiving the acquisition request sent by the second service node, send the first file execution data to the second service node.

[0064] After receiving the run instruction, the second service node sends a retrieval request to the management node. The management node receives the retrieval request and sends the first file's execution data to the second service node. Thus, the second service node loads the distributed file system based on the first file's execution data.

[0065] The execution instruction also instructs the second service node, after the distributed file system is successfully loaded, to determine the target shared storage space for storing the target operation data corresponding to the first file's execution data based on the first file's execution data, and to request the closure of the target shared storage space.

[0066] Each service node can access the shared storage space while running the file system. During operation, the first service node stores target operation data in the target shared storage space and obtains the identifier of the target shared storage space storing the target operation data as the first file operation data, which is then uploaded to the management node. In this way, the second service node can access the target storage space based on the first file operation data to run the file system.

[0067] In this embodiment, the shared storage space can be implemented by a chunk subsystem. The chunk subsystem is a subsystem that provides chunks, and operations such as requesting, accessing, and updating chunks can be performed through the chunk subsystem's SDK (Software Development Kit).

[0068] The chunk subsystem comprises multiple chunks, each with a unique identifier. Each chunk provides a byte stream space that supports append-only writes, similar to virtual address memory write-on-write functionality. Optionally, each chunk can be configured with a default maximum storage capacity of 128MB and provides chunk operation interfaces for reading, appending, closing, deleting, and querying length.

[0069] Correspondingly, the aforementioned target shared storage space can be at least one target chunk in the chunk subsystem. For example, during the operation of the first service node, after a client performs an operation on a file and generates operation information, this operation information is used as the target operation data. If the first service node requests chunk1 from the shared storage space to store the target operation data, the target operation data is stored in chunk1, and the identifier of chunk1 is used as the first file's execution data and uploaded to the management node. For instance, when an open operation is performed on file 1, the open operation information is used as the target operation data. Furthermore, when the first service node requests a new chunk, the identifier of the new chunk is uploaded to the management node.

[0070] Therefore, requesting the closure of the target shared storage space means closing the chunks that are being written to. Since the first service node is currently using these chunks, closing them prevents the first service node from continuing to write, thus preventing it from running the file system. This ensures that only the second service node is running the file system. Consequently, when multiple clients access the file system, only those accessing it through the second service node can successfully access it, avoiding file data processing errors.

[0071] The above system operation method, upon determining that the first service node cannot continue running the distributed file system, sends a run instruction to the second service node. This run instruction instructs the second service node to obtain the first file execution data from the management node, load the distributed file system on the second service node based on the first file execution data, and then run the distributed file system. Upon receiving the request from the second service node, the system sends the first file execution data uploaded to the management node during the first service node's operation of the distributed file system to the second service node. Furthermore, the run instruction instructs the second service node, after successfully loading the distributed file system, to determine the target shared storage space storing the target operation data corresponding to the first file execution data, and request to close the target shared storage space. This ensures that the second service node closes the target shared storage space upon loading the distributed file system, preventing the first service node from accessing it and thus preventing it from running. This avoids the situation where two service nodes are running the distributed file system simultaneously, preventing file data access errors.

[0072] In one embodiment, the management node can be implemented by the streamfs master. The streamfs master is the streaming file system management cluster, responsible for functions such as file system mounting, namespace creation, namespace loading, view management, and cluster configuration persistence.

[0073] For example, a StreamFS master includes multiple cluster nodes, and the management node can be one of these cluster nodes.

[0074] Optionally, multiple cluster nodes can be run as master nodes, with one of the master nodes connecting to each service node while it is running. If the master node fails, a new master node is selected from the other master nodes to connect to each service node.

[0075] For example, please refer to Figure 4 This is an exemplary schematic diagram of an architecture for accessing a distributed file system. Here, there are service nodes 1 (mds1) to service node n (mdsn), and management nodes 1 to management node m. Clients access the file system from the working service node 1 through the streamfs interface. Furthermore, Figure 4 It also shows the paths to other data updates.

[0076] In one embodiment, such as Figure 5 A flowchart illustrating the process of sending a run command is shown. Sending the run command to the second service node includes:

[0077] Step 501: If it is determined that the first service node cannot continue to run the distributed file system, determine the current logical clock and use the current logical clock as the target logical clock.

[0078] The management node maintains a globally unique logical clock.

[0079] Optionally, a 64-bit binary representation of the logical clock can be used, which ensures that time overflow will not occur. That is, the logical clock is long enough to prevent easy reset, ensuring the long-term operation of the file system.

[0080] During the operation of the file system, the logical clock is continuously updated. If it is determined that the first service node can no longer run the distributed file system, the management node determines the current logical clock as the target logical clock. For example, if the current logical clock is logical clock 35, then logical clock 35 will be used as the target logical clock.

[0081] Step 502: Generate a running instruction with the target logical clock based on the target logical clock, and send the running instruction to the second service node.

[0082] In other words, the execution command with the target logical clock is sent to the second service node.

[0083] Optionally, the execution command includes a target logical clock and indication information. The second service node receives and parses the execution command to obtain the target logical clock and indication information. It stores the target logical clock and, upon obtaining the indication information, sends a retrieval command to the management node to execute the file system.

[0084] In the case where the second file execution data is generated during the operation of the distributed file system on the second service node, the target logical clock is used for the second service node to bind the second file execution data with the target logical clock and then send it to the management node.

[0085] In other words, during the process of running the file system, each service node needs to send the latest file execution data to the management node after each update. For example, the first service node uploads the latest first file execution data after each update, and similarly, the second service node uploads the latest second file execution data after each update. This allows other service nodes to promptly obtain the file execution data from the management node and load the file system.

[0086] In one embodiment, the first file operation data includes at least one of a first file list and a second file list; the first file list includes multiple first block identifiers, and the block shared space corresponding to the first block identifier stores at least one file operation information; the second file list includes multiple second block identifiers, and the block shared space corresponding to the second block identifier stores at least one data snapshot.

[0087] In one embodiment, the second file execution data includes a third file list and a fourth file list; the third file list includes multiple third block identifiers, and the block shared space corresponding to the third block identifier stores at least one file operation information; the fourth file list includes multiple fourth block identifiers, and the block shared space corresponding to the fourth block identifier stores at least one data snapshot.

[0088] The chunk shared space is the same as the chunk mentioned above.

[0089] The following description uses the example of the first file execution data including at least one of the first file list and the second file list to illustrate file execution data. The updating of the second file execution data is similar and will not be repeated here.

[0090] The first service node requests a chunk shared space (chunk1) from the shared storage space to store file operation information. After successful request, the identifier of chunk1 is used as the first chunk identifier and added to the first file list. The first file list is then uploaded to the management node.

[0091] Thus, after receiving the client's first operation request for the target file, the first service node performs the first operation on the target file, generates a log of the first operation, and obtains the first operation information (same as the file operation information above). This first operation information is written to chunk1. When chunk1 is full, the first service node requests a shared storage space chunk2 from the shared storage space to store the file operation information. After successful request, the identifier of chunk2 is used as the first block identifier and added to the first file list. If the first file list is updated at this point, the updated first file list, including the chunk1 and chunk2 identifiers, is uploaded to the management node.

[0092] The first service node also persists file operation information in memory in the form of logs, and periodically persists the data in memory to the shared storage space by generating snapshots. The file operation information corresponding to the generated snapshots can be deleted to free up the shared storage space. Therefore, the first service node also needs to apply for block shared space for storing snapshots.

[0093] For example, the first service node requests a chunk shared space (chunk3) from the shared storage space for storing snapshots. After successful request, the identifier of chunk3 is used as the second chunk identifier and added to the second file list. The second file list is then uploaded to the management node.

[0094] After writing file operation information into the shared space of the first block identifier (e.g., chunk1), this file operation information is also stored in memory. When multiple file operation information entries are stored in memory, a snapshot of these entries is periodically generated and stored in chunk3. At this point, the file operation information in chunk1 is deleted to free up space in chunk1. This achieves persistence of file operation information, and because data in memory can be modified, it provides data modification functionality. The freed-up chunk1 can then continue to store other file operation information.

[0095] Understandably, during file system operation, multiple snapshots are continuously generated. When multiple snapshots are written to chunk3, filling it up, the first service node requests shared storage space chunk4 from the shared storage space to store the data snapshots. After successful request, the identifier of chunk4 is used as the second chunk identifier and added to the second file list. If the second file list is updated at this point, the updated second file list, including the chunk3 and chunk4 identifiers, is uploaded to the management node.

[0096] In other words, each time the first file list and / or the second file list are updated, new first file execution data is obtained and uploaded to the management node.

[0097] For example, such as Figure 6 The diagram illustrates the storage of the first and second file lists. `meta` represents a data snapshot, and `log` represents file operation information. Optionally, the second file list includes an authoritative list and a non-authoritative list. In the non-authoritative list, the data snapshots stored in the chunks corresponding to the chunk identifiers are incomplete, while the data snapshots stored in the chunks corresponding to the chunk identifiers in the authoritative list are complete. Furthermore, Figure 7 This is a schematic diagram of the data update process corresponding to the description above.

[0098] As mentioned above, the target logical clock is stored in the second service node. Similarly, when the first service node is instructed to load the file system, the management node also obtains the logical clock at that time as the first logical clock and sends it to the first service node for storage.

[0099] Optionally, the first logical clock precedes the target logical clock, which means that the service node runs the file system before the second service node.

[0100] For example, if the logical clock at that time is logical clock 6, then logical clock 6 will be used as the first logical clock. At this time, the management node will use logical clock 6 as the detection logical clock and record it.

[0101] Taking logical clock 6 as the first logical clock and logical clock 35 as the second logical clock as the second logical clock, during the normal operation of the file system on the first service node, the management node uses logical clock 6 as the detection logical clock and records it. When the first service node uploads updated first file execution data, it sends the updated first file execution data and the first logical clock together to the management node. Thus, when the management node receives the data, it first determines whether the first logical clock is the recorded detection logical clock; if so, it stores the updated first file execution data.

[0102] Similarly, during the normal operation of the file system on the first service node, the management node uses and records logical clock 35 as the detection logical clock. When the second service node uploads updated second file execution data, it sends the updated second file execution data and the target logical clock to the management node. Thus, when the management node receives the data, it first determines whether the target logical clock is the detection logical clock; if so, it stores the updated second file execution data.

[0103] By using a logical clock to identify the file execution data uploaded by each service node, the management node can identify whether the currently received data was sent by the second service node that is allowed to run the file system based on the logical clock. This ensures that the data of the second service node is stored while the data of the first service node does not need to be stored, thus guaranteeing the correctness of the file execution data stored in the management node.

[0104] The following section continues to explain the process by which the second service node loads the distributed file system based on the data from the first file.

[0105] In one embodiment, the process of the second service node loading the distributed file system based on the data from the first file includes:

[0106] Read at least one file operation information in the block shared space corresponding to the first block identifier, and execute the file operation action corresponding to the at least one file operation information; and read at least one data snapshot in the block shared space corresponding to the second block identifier, and store at least one data snapshot in the second service node; wherein, the data snapshot describes the information of the data stored in memory at the target time.

[0107] Optionally, each data snapshot is stored locally on the second service node. Alternatively, the second service node stores each data snapshot in the block shared space corresponding to the requested fourth block identifier.

[0108] For example, if at least one file operation information in the block shared space corresponding to the first block identifier includes operation information for opening file 1 and operation information for expanding file 1, then the second service node will run the operations of opening file 1 and expanding file 1.

[0109] This allows the second service node to reasonably replace the first service node and continue running the file system, ensuring the integrity of the file system migration.

[0110] Since it has been determined that the first service node can no longer run the distributed file system, the management node also needs to ensure the authenticity and reliability of the first service node's timely shutdown. The process of controlling the first service node to stop running the file system is described below.

[0111] In an optional embodiment, such as Figure 8 A flowchart illustrating the process of sending a stop command to the first service node is shown. The method also includes:

[0112] Step 801: Receive the latest first file running data and first logical clock sent by the first service node.

[0113] Among them, the latest first file running data is used to request the management node to store the latest first file running data.

[0114] Optionally, if the first service node is deemed unable to continue running the file system due to a network disconnection, the first service node will still upload data to the management node after the network is restored.

[0115] Step 802: If it is determined that the first logical clock is not the target logical clock, refuse to store the latest first file running data and send a stop command to the first service node.

[0116] The stop command is used to instruct the first service node to stop running the distributed file system.

[0117] As described above, the management node assumes the second service node is running a file system, and it records the target logical clock as the detection logical clock. Therefore, upon receiving the latest first file running data and the first logical clock from the first service node, and determining that the first logical clock is not the target logical clock, the management node does not store the latest first file running data.

[0118] Furthermore, the management node now knows that the first service node is still running the file system, but needs to stop it. Therefore, it can send a stop command to the first service node.

[0119] The first service node stops running the distributed file system after receiving the stop command.

[0120] This ensures that the file system runs on only one server node, preventing access errors.

[0121] In another optional embodiment, the method further includes: receiving a first notification message sent by a second service node; and, if it is determined that the first notification message indicates that the second service node has successfully loaded the distributed file system, sending a system view to the client and the first service node.

[0122] In other words, after the second service node successfully loads the distributed file system, it sends the first notification message to the management node to inform the management node that it has successfully loaded the file system and is continuing to run.

[0123] Optionally, the system receives a first notification message sent by the second service node that is bound to the target logical clock, thereby determining, based on the target logical clock, that the first notification message is from the second service node.

[0124] The system view is used to describe at least the distributed file system currently running on the second service node.

[0125] Optionally, since the service node runs on a server, the system view can also be used to describe the identifier of the server running the second service node. This allows the management node to connect to the server based on its identifier, thereby communicating with the second service node.

[0126] In this way, after receiving the system view, the client can know that it will subsequently access the file system through the second service node.

[0127] Alternatively, the system view can be used to instruct the first service node to stop running the distributed file system upon receiving the system view. That is, the management node also sends the system view to all other service nodes except the second service node. In this case, the first service node, upon receiving the system view, determines that the current file system is running on the second service node and then stops running the distributed file system.

[0128] In this way, the first service node can be notified in multiple ways to stop the file system in a timely manner, which improves the flexibility of ensuring that the first service node stops the file system in a timely manner.

[0129] For ease of understanding, the following describes the system operation method provided in this application in its entirety using a complete embodiment.

[0130] Please refer to Figure 9 The diagram illustrates the system switchover process. A global logical clock is maintained in the management node. It is configured that the first service node, mds1, will run the file system when logical clock 6 arrives. Therefore, when logical clock 6 arrives, logical clock 6 and the run command are sent to mds1, and mds1 runs the distributed file system. At this time, the first file list in the distributed file system includes the identifier for chunk2, and the second file list is empty.

[0131] The currently updated first file list and logical clock 6 are bound together and sent to the management node for storage. The first file list includes an identifier for chunk2, indicating that file operation data for a specific file has been written to chunk2. For example, the log entry for opening file 1 is a file operation record written to chunk2. Furthermore, a chunk can write logs for multiple operations until it is full, at which point a new chunk is requested from the chunk subsystem.

[0132] Subsequently, the management node records the detected logical clock as logical clock 6. It is assumed that on logical clock 6, mds1 runs a distributed file system and allows mds1 to periodically upload the first file list to the management node.

[0133] When the management node determines that it has disconnected from mds1 or that the lease of mds1 has expired, it uses the current logical time 35 as the detection logical clock and sends the logical time 35 and the running instructions to the second service node mds3, so that mds3 runs the distributed file system. At this time, the management node only recognizes the distributed file system running on mds3 at logical time 35.

[0134] After receiving the execution command, MDS3 retrieves the existing first and second file lists from the management node. It then obtains file operation information and data snapshots from the chunks corresponding to the chunk identifiers in both lists, stores the data snapshots, and executes the operations described in the file operation information. Finally, it requests the chunk subsystem to close the chunks in both the first and second file lists.

[0135] After the distributed system runs successfully, MDS3 informs the management node that it has run successfully. At this time, the management node sends the updated system view of the distributed file system to each client and each service node.

[0136] During MDS3 operation, new chunks can be requested from the chunk subsystem for writing file operation information and data snapshots. Additionally, updated third and / or fourth file lists can be bound to logical clock 35 and sent to the management node. After storing the updated third and / or fourth file lists, the management node returns a successful storage notification to MDS3.

[0137] When the MDS1 lease expires and stops running, the distributed file system fails to stop running due to factors such as being in state D, thus allowing the distributed system to continue running.

[0138] At this time, while both mds1 and mds3 are running the file system, because mds3 requests the chunk subsystem to close chunk2 when running the system, mds1 fails to update the metadata using chunk2. Therefore, mds1 continues to request chunk5 from the chunk subsystem to write file operation data and update the first file list. The updated first file list and logical clock 6 are then sent to the management node.

[0139] After receiving the data from mds1, the management node determines that logical clock 6 is different from the current detection logical clock (i.e., logical clock 35), and then sends an unload request to mds1. After receiving the unload request, mds1 stops running the distributed file system.

[0140] In addition, if mds1 receives a system view sent by the management node and determines that mds3 is running the system based on the system view, it will actively unload the system.

[0141] This application provides a system switching method implemented through a combination of multiple mechanisms and strategies. Based on this, metadata is loaded to provide consistent services and simplify the data loading logic. First, the space of the old time-based file system can be closed and its space requests can be prohibited. Second, file system metadata loading only begins after the new time-based file system successfully closes the target shared space requested by the old time-based file system. Third, the old time-based file system cannot use its existing space or request new space, and can only provide read services. Fourth, after the view of the new time-based file system is diffused, the old time-based file system also exits the service provision. In summary, this ensures that the old time-based file system is unloaded in a timely manner and ceases operation, thereby preventing data access errors caused by multiple file systems running simultaneously.

[0142] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0143] Based on the same inventive concept, this application also provides a system operation apparatus for implementing the system operation method described above. The solution provided by this apparatus is similar to the solution described in the above method; therefore, the specific limitations in one or more system operation apparatus embodiments provided below can be found in the limitations of the system operation method described above, and will not be repeated here.

[0144] In one embodiment, such as Figure 10 As shown, a system operating device 1000 is provided, comprising: a first transmitting module 1001 and a second transmitting module 1002, wherein:

[0145] The first sending module 1001 is used to send a running instruction to the second service node when it is determined that the first service node cannot continue to run the distributed file system. The running instruction is used to instruct the second service node to obtain the first file running data from the management node, and to load the distributed file system on the second service node according to the first file running data and then run the distributed file system.

[0146] The second sending module 1002 is used to send the first file execution data to the second service node upon receiving an acquisition request from the second service node; the first file execution data is uploaded to the management node by the first service node during the process of running the distributed file system;

[0147] The execution instruction also instructs the second service node, after the distributed file system is successfully loaded, to determine the target shared storage space for storing the target operation data corresponding to the first file's execution data based on the first file's execution data, and to request the closure of the target shared storage space.

[0148] In one embodiment, the first sending module 1001 is specifically used for:

[0149] If it is determined that the first service node cannot continue to run the distributed file system, the current logical clock is determined and used as the target logical clock; an execution instruction with the target logical clock is generated according to the target logical clock; the execution instruction is sent to the second service node; wherein, if the second file execution data is generated during the operation of the distributed file system on the second service node, the target logical clock is used for the second service node to bind the second file execution data with the target logical clock and send it to the management node.

[0150] In one embodiment, the device further includes a first receiving module for:

[0151] The system receives the latest first file running data and the first logical clock sent by the first service node; the latest first file running data is used to request the management node to store the latest first file running data; if it is determined that the first logical clock is not the target logical clock, the system refuses to store the latest first file running data and sends a stop command to the first service node; wherein, the stop command is used to instruct the first service node to stop running the distributed file system.

[0152] In one embodiment, the device further includes a second receiving module for:

[0153] Receive a first notification message sent by the second service node; if the first notification message indicates that the second service node has successfully loaded the distributed file system, send a system view to the client and the first service node; wherein the system view is used to describe at least the distributed file system currently running on the second service node.

[0154] In one embodiment, the system view is used to instruct the first service node to stop running the distributed file system upon receiving the system view.

[0155] In one embodiment, the first file operation data includes at least one of a first file list and a second file list; the first file list includes multiple first block identifiers, and the block shared space corresponding to the first block identifier stores at least one file operation information; the second file list includes multiple second block identifiers, and the block shared space corresponding to the second block identifier stores at least one data snapshot.

[0156] Each module in the aforementioned system operating device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0157] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 11 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores system operation data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a system operation method.

[0158] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0159] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0160] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0161] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0162] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0163] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0164] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A system operation method, characterized in that, For managing nodes, the method includes: If it is determined that the first service node cannot continue to run the distributed file system, a run instruction is sent to the second service node; the run instruction is used to instruct the second service node to obtain the first file running data from the management node, and to load the distributed file system on the second service node according to the first file running data and then run the distributed file system. Upon receiving a request from the second service node, the first file execution data is sent to the second service node; the first file execution data is uploaded to the management node during the process of the first service node running the distributed file system. The execution instruction further instructs the second service node, after the distributed file system is successfully loaded, to determine the target shared storage space for storing the target operation data corresponding to the first file execution data based on the first file execution data, and to request the closure of the target shared storage space.

2. The method according to claim 1, characterized in that, Sending the execution command to the second service node includes: If it is determined that the first service node can no longer run the distributed file system, the current logical clock is determined and the current logical clock is used as the target logical clock; Generate running instructions with the target logic clock based on the target logic clock; The execution command is sent to the second service node; In the case where second file execution data is generated during the operation of the distributed file system by the second service node, the target logical clock is used for the second service node to bind the second file execution data with the target logical clock and then send it to the management node.

3. The method according to claim 2, characterized in that, The method further includes: The system receives the latest first file running data and the first logical clock sent by the first service node; the latest first file running data is used to request the management node to store the latest first file running data. If it is determined that the first logical clock is not the target logical clock, the latest first file running data will not be stored, and a stop command will be sent to the first service node; The stop instruction is used to instruct the first service node to stop running the distributed file system.

4. The method according to claim 1, characterized in that, The method further includes: Receive the first notification information sent by the second service node; If it is determined that the first notification information indicates that the second service node has successfully loaded the distributed file system, a system view is sent to the client and the first service node. The system view is used at least to describe the distributed file system currently running on the second service node.

5. The method according to claim 4, characterized in that, The system view is used to instruct the first service node to stop running the distributed file system upon receiving the system view.

6. The method according to any one of claims 1 to 5, characterized in that, The first file execution data includes at least one of the first file list and the second file list; The first file list includes multiple first block identifiers, and the block shared space corresponding to the first block identifier stores at least one file operation information; The second file list includes multiple second block identifiers, and at least one data snapshot is stored in the block shared space corresponding to the second block identifier.

7. A system operating device, characterized in that, For managing nodes, the device includes: The first sending module is used to send a running instruction to the second service node when it is determined that the first service node cannot continue to run the distributed file system; the running instruction is used to instruct the second service node to obtain the first file running data from the management node, and to load the distributed file system on the second service node according to the first file running data and then run the distributed file system. The second sending module is used to send the first file execution data to the second service node upon receiving an acquisition request from the second service node; the first file execution data is uploaded to the management node by the first service node during the operation of the distributed file system; The execution instruction further instructs the second service node, after the distributed file system is successfully loaded, to determine the target shared storage space for storing the target operation data corresponding to the first file execution data based on the first file execution data, and to request the closure of the target shared storage space.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images