File sharing management method, device and system

Abstract

Embodiments of the present specification provide a shared file management method, device and system, the method comprising: generating an index of data and version information of the index for a write operation of the data, and sending the index and the version information corresponding to the index to a slave node of a master node; receiving a response message fed back by the slave node in a case where the slave node loads the index; in a case where target data needs to be processed, comparing the version information of the index carried in the response message with the current latest version information of the index of the master node; and in a case where the version information of the index carried in the response message is consistent with the current latest version information of the index of the master node, determining an operation on the target data based on shared demand information of the data carried in the response message. Since the response message is fed back according to the index, the version information and the shared demand information are carried in the response message, which reuses a communication link for feeding back the index, and a separate communication link is not needed, thereby reducing the complexity of the system.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of data processing technology, and in particular to a shared file management method. Background Technology

[0002] In a distributed system, a group of independent computers presents a unified system to the user. The system possesses various physical and logical resources, can dynamically allocate tasks, and these distributed resources exchange information through computer networks. In a distributed system based on shared storage, physical replication is used for master-slave data synchronization. Physical replication only requires constructing an index on the master replica and then synchronizing the entire index file to the slave replicas, saving CPU resources from repeatedly building the index on the slave replicas. Being based on shared storage means that files already persisted to disk on the master replica can be directly read and loaded from the shared storage on the slave replicas.

[0003] In current distributed systems based on shared storage, physical replication is used to synchronize data between "master replicas" and "slave replicas," which requires solving the problem of shared file management. However, the current shared file management mechanisms are complex and consume a lot of communication resources, necessitating a new solution. Summary of the Invention

[0004] In view of this, embodiments of this specification provide two methods for managing shared files. One or more embodiments of this specification also relate to a shared file management device, a system, a computing device, a computer-readable storage medium, and a computer program, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a shared file management method is provided, applied to a master node, comprising:

[0006] For write operations on the data, an index of the data and version information of the index are generated, and the index and the corresponding version information are sent to the slave node of the master node. The index is used to enable the slave node to provide read services for the data based on the index.

[0007] The system receives a response message from the slave node after loading the index, wherein the response message carries the version information of the index and information on the data sharing requirements.

[0008] When it is necessary to process target data, the version information of the index carried in the response message is compared with the latest version information of the master node index.

[0009] If the version information of the index carried in the response message is consistent with the latest version information of the master node index, the operation on the target data is determined based on the data sharing requirement information carried in the response message.

[0010] According to a second aspect of the embodiments of this specification, a shared file management method is provided, applied to a slave node, comprising:

[0011] Receive the index from the master node of the slave node;

[0012] Once the index is loaded, a response message is sent back to the master node, which carries the version information of the index and information on the data sharing requirements.

[0013] According to a third aspect of the embodiments of this specification, a shared file management device is provided, applied to a master node, comprising:

[0014] The data sending module is configured to generate an index of the data and version information of the index for write operations on the data, and send the index and the version information corresponding to the index to the slave node of the master node, wherein the index is used to enable the slave node to provide read services for the data based on the index;

[0015] The message receiving module is configured to receive a response message from the slave node after the index has been loaded, wherein the response message carries the version information of the index and information on the data sharing requirements.

[0016] The information comparison module is configured to compare the version information of the index carried in the response message with the latest version information of the master node index when the target data needs to be processed.

[0017] The data processing module is configured to determine the operation on the target data based on the data sharing requirement information carried in the response message, provided that the version information of the index carried in the response message is consistent with the latest version information of the master node index.

[0018] According to a fourth aspect of the embodiments of this specification, a shared file management device is provided, applied to a slave node, comprising:

[0019] The data receiving module is configured to receive the index from the master node;

[0020] The message feedback module is configured to send the response message back to the master node when the index loading is complete.

[0021] According to a fifth aspect of the embodiments of this specification, a system is provided, comprising:

[0022] A master node that implements the above-described shared file management method, and a slave node that implements the above-described shared file management method, wherein the master node and the slave node share storage.

[0023] According to a sixth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0024] Memory and processor;

[0025] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above-described shared file management method.

[0026] According to a seventh aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the shared file management method described above.

[0027] According to a sixth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described shared file management method.

[0028] This specification provides a shared file management method and apparatus. The shared file management method includes: generating an index of the data and version information of the index for write operations on the data; sending the index and the corresponding version information to a slave node of a master node, wherein the index enables the slave node to provide read services for the data based on the index; receiving a response message from the slave node after loading the index, wherein the response message carries the version information of the index and data sharing requirement information; when target data needs to be processed, comparing the version information of the index carried in the response message with the latest version information of the master node's index; if the version information of the index carried in the response message is consistent with the latest version information of the master node's index, determining the operation on the target data based on the data sharing requirement information carried in the response message. By receiving the response message from the slave node and operating on the target data according to the version information and sharing requirement information carried in the response message, since the response message is based on the index feedback, carrying the version information and sharing requirement information in the response message reuses the communication link for index feedback, eliminating the need for a separate communication link and reducing system complexity. Attached Figure Description

[0029] Figure 1 This is a schematic diagram illustrating a scenario of a shared file management method provided in one embodiment of this specification;

[0030] Figure 2a This is a flowchart illustrating a shared file management method provided in one embodiment of this specification;

[0031] Figure 2b This is an architecture diagram of a shared file management method provided in one embodiment of this specification;

[0032] Figure 3 This is a flowchart of another shared file management method provided in one embodiment of this specification;

[0033] Figure 4 This is a processing timing diagram of a system provided in one embodiment of this specification;

[0034] Figure 5 This is a flowchart illustrating the file deletion process of a shared file management method according to one embodiment of this specification;

[0035] Figure 6 This is a schematic diagram of the structure of a shared file management device provided in one embodiment of this specification;

[0036] Figure 7 This is a schematic diagram of another shared file management device provided in one embodiment of this specification;

[0037] Figure 8 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0038] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0039] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0040] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0041] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0042] Primary replica, also known as read-write replica, refers to a storage system where data is redundantly stored as multiple copies, one of which serves as the primary replica, or read-write replica, providing write and read capabilities. The primary replica synchronizes data to the secondary replicas through physical or logical replication.

[0043] A slave copy, also called a read-only copy, refers to a storage system where data is redundantly stored in multiple copies for backup. The copy that only provides read capability is called a slave copy, or read-only copy.

[0044] Shared storage: Unlike local disk storage, shared storage is a file system that allows multiple machines on a network to read and write the same data.

[0045] Logical replication: In the master-slave replication model, written documents are sent to the master replica and the slave replica respectively, and the indexes are constructed independently on the master and slave replicas in a synchronization method.

[0046] Physical replication: In the master-slave replication model, only the documents to be written are sent to the master replica, the index is built on the master replica, and then the entire built index file is synchronized to the slave replica to avoid duplicate building on the slave replica.

[0047] Primary: Main copy.

[0048] Replica: From a replica.

[0049] WAL: Write-Ahead Logging, a write-ahead logging system.

[0050] Persistence: is the mechanism for converting program data between persistent and transient states.

[0051] In the current scheme, the primary replica maintains a reference count for shared files. The reference count is incremented by 1 before physical copying and decremented by 1 afterward. When the reference count reaches 0, the file is physically deleted. After successfully loading a file from the replica, the primary replica needs to be notified to decrement the reference count, requiring an additional communication link and increasing system complexity.

[0052] Based on this, this specification provides two methods for managing shared files, and also relates to two devices for managing shared files, a system, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0053] See Figure 1 , Figure 1 The illustration shows a scenario diagram of a shared file management method according to an embodiment of this specification, which includes a master node, slave nodes, and shared storage.

[0054] When the master node receives a new file write request, it first generates a corresponding index, which includes version information. Based on the index, it generates the file, stores it in shared storage, and then sends the index to the slave nodes.

[0055] When a slave node receives an index, it loads the corresponding file from the shared storage based on the index and sends a response message to the master node for that index. The response message carries version information and shared file information.

[0056] By receiving response messages from slave nodes, the target data is manipulated based on the version information and sharing requirement information carried in the response messages. Since the response messages are based on index feedback, carrying version information and sharing requirement information in the response messages reuses the communication link for index feedback, eliminating the need for a separate communication link and reducing system complexity.

[0057] See Figure 2a , Figure 2a A flowchart of a shared file management method according to an embodiment of this specification is shown. The method is applied to a master node in a system, wherein the system may be a distributed system, and the master node can be understood as a node that stores master replicas. The method specifically includes the following steps.

[0058] Step 202: Generate an index for the data and version information of the index for the write operation of the data, and send the index and the corresponding version information to the slave node of the master node, wherein the index is used to enable the slave node to provide read service for the data based on the index.

[0059] Here, data can be data stored in distributed nodes, such as data in a database. Indexes can be indexes generated for the data, and correspondingly, index version information is the new version information generated each time the index is modified. For example, if new data A is generated, a new index V is generated. Slave nodes can be the nodes where replicas reside. Read services can be services that read data.

[0060] In practical applications, see Figure 2b , Figure 2b This diagram illustrates an architecture of a shared file management method according to an embodiment of this specification. Each time a user performs a new data write operation, the data is first written to a buffer in the master node's memory. An index file (segment) is then generated based on the data in the buffer and stored in the cache. An index snapshot of the index file is also generated, version information is added to the snapshot, and the snapshot is sent to the slave node via physical replication. This allows the slave node to obtain the snapshot of the index file and retrieve new file changes. Furthermore, users accessing from the master node can access the shared storage or the index file on the master node through a read-write engine, while users accessing from the slave node can access the shared storage or the index file on the slave node through a read-only engine. Additionally, when data is being written to the buffer, a new log entry is simultaneously added to the data write log to record the written data. This data write log can be a Write-Ahead Log (WAL).

[0061] For example, when newly written data is received in the buffer, refresh generates an index file, creating files f1 and f2, and generates an index snapshot of the current index file, with version information v1. The v1 version index snapshot is then sent to the slave node.

[0062] This embodiment generates an index snapshot, generates version information for the index snapshot, and sends the index snapshot and version information to the slave node, so that the slave node can generate a response message for the index snapshot and version information, which facilitates information transmission through the response message.

[0063] In one possible implementation, after the write operation on the data generates an index of the data and version information of the index, the method further includes:

[0064] Submit the index to make it effective, and write the data to the shared storage based on the index.

[0065] Shared storage can be the same as the terminology used to describe shared storage, which can be accessed by both slave nodes and master nodes.

[0066] In practical applications, see Figure 2b After the index file is generated, a commit operation is required before the data can be written to shared storage. The commit operation essentially persists the index file to shared storage.

[0067] For example, when newly written data is received in the buffer, refresh generates an index file, creating files f1 and f2, and also generates an index snapshot of the current index file with version information v1, creating files f1 and f2. The v1 version of the index is then committed, and f1 and f2 can then be written to shared storage.

[0068] Furthermore, after generating the index, the current version of the index can be associated with the file to be generated.

[0069] In practical applications, relationships can be established through mapping.

[0070] For example, when newly written data is received in the buffer, refresh generates an index file, creating files f1 and f2, and generates an index snapshot of the current index file with version information v1. Files f1 and f2 are then generated, and a new mapping {f1:v1, f2:v1} is created.

[0071] In the embodiments described in this specification, data can only be written to shared storage after the index submission operation is performed, thus avoiding the impact of short-term deletion operations on the read and write performance of shared storage.

[0072] Step 204: Receive a response message from the slave node after loading the index, wherein the response message carries the version information of the index and information on the data sharing requirements.

[0073] In this context, "index loading complete" can be understood as the completion of operations on the corresponding data based on the index. The response message can be a feedback message from the slave node regarding the received index. For example, after receiving the index, the slave node generates a response message "Index received".

[0074] In practical applications, after receiving an index, the slave node processes it, such as loading files from shared storage based on the index. After loading the index, it generates a response message containing the index version information and data sharing requirements. This allows the master node to know which version of the index the slave node has loaded and what data is being shared, enabling the master node to process the data based on the response message.

[0075] For example, when the master node receives newly written data in the buffer, it refreshes the index file, generating files f1 and f2, and creates an index snapshot of the current index file. This snapshot's version information is v1, also generating files f1 and f2. The v1 version of the index is committed, and then f1 and f2 can be written to shared storage. The v1 version index snapshot is then sent to the slave node. Upon receiving the v1 version index snapshot, the slave node loads the corresponding files f1 and f2 as needed; that is, it copies files f1 and f2 from shared storage to its own memory. After completing the copying of files f1 and f2, the loading of the files corresponding to the v1 version index snapshot is complete. This allows it to generate a response message for the index: "v1 version index snapshot successfully loaded; files f1 and f2 need to be shared," and return this response message to the master node. The master node then receives the response message for the v1 version index snapshot: "v1 version index snapshot successfully loaded; files f1 and f2 need to be shared."

[0076] In this embodiment of the specification, the master node can obtain the status of files in the slave nodes by receiving response messages carrying version information of the index and information on the data sharing requirements, so as to operate on the files without using a separate communication link, thus reducing the complexity of the system.

[0077] Step 206: If target data needs to be processed, compare the version information of the index carried in the response message with the latest version information of the master node index.

[0078] The target data can be data that has been deleted or modified.

[0079] In practical applications, the version information of the index obtained from the response message can tell us which version of the index was loaded by the slave node. This allows us to determine which data to operate on in subsequent steps.

[0080] For example, the master node can receive a response message for the v1 version index snapshot: "The v1 version index snapshot has been successfully loaded. Files f1 and f2 need to be shared." Upon receiving a deletion command for files f1 and f2, the master node confirms the version information of the index snapshots for files f1 and f2. At this point, the version information of the index snapshots for files f1 and f2 is v2, which is greater than the v1 version index snapshot in the slave node.

[0081] Step 208: If the version information of the index carried in the response message is consistent with the latest version information of the master node index, determine the operation on the target data based on the data sharing requirement information carried in the response message.

[0082] In practical applications, when it is necessary to perform operations such as data deletion or modification, the target data can be manipulated based on the received response message.

[0083] For example, upon receiving a deletion command for files f1 and f2, the version information of the index snapshots of files f1 and f2 is confirmed. After confirmation, the operation on files f1 and f2 is performed according to the response message.

[0084] The embodiments in this specification operate on files by using the version information of the index and the information on the need for data sharing carried in the response message, without the need for a separate communication link, thus reducing the complexity of the system.

[0085] One possible approach also includes:

[0086] If the version information of the index carried in the response message is less than the latest version information of the master node index, a deletion mark is added to the target data.

[0087] The deletion flag can be used to mark the data to be deleted.

[0088] In practical applications, the master node may also receive a delete command or a data merge command for a specific piece of data. However, if it finds that the version information of the index for that data in the master node is greater than the version information of the index in the slave node, then the data will not be deleted; only a delete flag will be added. In other words, in cases where index deletion or merging is required, only a deletion flag is added.

[0089] Continuing with the previous example, the master node will receive a response message for the v1 version index snapshot: "The v1 version index snapshot has been successfully loaded; files f1 and f2 need to be shared." Upon receiving a deletion command for files f1 and f2, the master node confirms the version information of the index snapshots for files f1 and f2. At this point, the version information of the index snapshots for files f1 and f2 is v2, which is greater than the v1 version index snapshot in the slave node. Therefore, files f1 and f2 are not deleted; only deletion markers are added to files f1 and f2.

[0090] Furthermore, if the data is not synchronized to the slave node, the data can be deleted directly.

[0091] In practical applications, the master node may also receive a deletion command for a specific piece of data, or a data merging command. If it is confirmed that the data does not exist in the slave node, the data can be deleted directly.

[0092] For example, if a deletion command is received for files f1 and f2, and it is found that no slave node shares files f1 and f2, then files f1 and f2 can be deleted directly.

[0093] The embodiments in this specification determine whether a file can be deleted by using the version information of the index carried in the response message, which can avoid accidental file deletion and improve security.

[0094] In one feasible approach, when target data needs to be processed, the version information of the index carried in the response message is compared with the latest version information of the master node index. If the version information of the index carried in the response message matches the latest version information of the master node index, the operation on the target data is determined based on the data sharing requirement information carried in the response message, including:

[0095] Based on the data sharing requirement information carried in the response message, determine whether the target data is in use; if so, add a deletion mark to the target data.

[0096] The phrase "data in use" can be understood as a service program querying the data.

[0097] In practical applications, the master node may also receive deletion or merging commands for a specific piece of data. However, if it finds that the version information of the index for that data in the master node is equal to the version information of the index in the slave node, it will further determine whether the data is currently in use. If the data is in use, it will not be deleted, but only a deletion marker will be added. In other words, in cases where index deletion or merging is required, only a marker deletion is performed to ensure that the file is not used by any slave node before the actual physical deletion is executed.

[0098] Continuing with the previous example, the master node can receive a response message for the v1 version index snapshot: "The v1 version index snapshot has been successfully loaded; files f1 and f2 need to be shared." Upon receiving a deletion command for files f1 and f2, the master node confirms the version information of the index snapshots for files f1 and f2. At this point, the version information of the index snapshots for files f1 and f2 is v1, which is the same as the version information of the v1 version index snapshot in the slave node. This determines whether files f1 and f2 are being used correctly; that is, it checks whether there is query traffic for files f1 and f2. If there is still query traffic for files f1 and f2, it can be determined that files f1 and f2 are being used, and therefore files f1 and f2 are not deleted; only a deletion mark is added to files f1 and f2.

[0099] The embodiments in this specification determine whether a file can be deleted by using the sharing request information carried in the response message, which can avoid accidental file deletion and improve security.

[0100] In one possible implementation, the shared file management method of the embodiments of this specification further includes:

[0101] Based on the data sharing requirement information carried in the response message, determine whether the target data is currently in use. If not, delete the target data according to the deletion operation for the target data.

[0102] In practical applications, the master node may also receive a deletion command or a data merging command for a certain piece of data. However, if it finds that the version information of the index of the data in the master node is equal to the version information of the index in the slave node, it will continue to determine whether the data is being used. If the data is not being used, it will be deleted.

[0103] For example, the master node can receive a response message for the v1 version index snapshot: "The v1 version index snapshot has been successfully loaded." Upon receiving a deletion command for files f1 and f2, the master node confirms the version information of the index snapshots for files f1 and f2. At this point, the version information of the index snapshots for files f1 and f2 is v1, which is the same as the version information of the v1 version index snapshot in the slave node. Therefore, it determines whether files f1 and f2 are being used correctly; that is, it checks if files f1 and f2 exist in the response message. If files f1 and f2 do not exist, it can be determined that files f1 and f2 are not being used, and files f1 and f2 are deleted.

[0104] The embodiments in this specification update the file sharing requirement information on the slave node before deleting the file, which can avoid accidental file deletion and improve security.

[0105] In one possible implementation, the shared file management method of the embodiments of this specification further includes:

[0106] After adding a deletion mark to the target data, an updated response message is received. Based on the sharing requirement information carried in the updated response message, it is determined whether the data with the deletion mark is in use. If not, the data with the deletion mark is deleted.

[0107] In practical applications, the master node may also receive deletion or data merging commands for a specific piece of data. However, if it finds that the version information of the data's index on the master node is equal to the version information of the index on the slave node, it will further determine whether the data is currently in use. If the data is in use, it will not be deleted, but only a deletion flag will be added. Only after waiting for a new response message and ensuring that the file is no longer in use by any slave node will the actual physical deletion be performed.

[0108] For example, the master node might receive a response message for the v1 version index snapshot: "The v1 version index snapshot has been successfully loaded. Files f1 and f2 need to be shared." Upon receiving a deletion command for files f1 and f2, the master node confirms the version information of the index snapshots for f1 and f2. If the version information of the index snapshots for f1 and f2 is v2, which is greater than the v1 version index snapshot on the slave node, then files f1 and f2 are not deleted; only a deletion marker is added. The master node then receives a response message for the v2 version index snapshot: "The v2 version index snapshot has been successfully loaded." The master node checks if any files marked for deletion exist. If files f1 and f2 exist among the marked files, but the response message indicates that files f1 and f2 no ​​longer exist, then files f1 and f2 can be deleted, and they are removed from the master node's memory and shared storage.

[0109] This specification provides a shared file management method and apparatus. The shared file management method includes: generating an index of the data and version information of the index for write operations on the data; sending the index and the corresponding version information to a slave node of a master node, wherein the index enables the slave node to provide read services for the data based on the index; receiving a response message from the slave node after loading the index, wherein the response message carries the version information of the index and data sharing requirement information; when target data needs to be processed, comparing the version information of the index carried in the response message with the latest version information of the master node's index; if the version information of the index carried in the response message is consistent with the latest version information of the master node's index, determining the operation on the target data based on the data sharing requirement information carried in the response message. By receiving the response message from the slave node and operating on the target data according to the version information and sharing requirement information carried in the response message, since the response message is based on the index feedback, carrying the version information and sharing requirement information in the response message reuses the communication link for index feedback, eliminating the need for a separate communication link and reducing system complexity.

[0110] This specification also provides a shared file management method, applied to slave nodes of a master node, see [link to documentation]. Figure 3 , Figure 3 A flowchart illustrating another shared file management method is shown, including:

[0111] Step 302: Receive the index from the master node;

[0112] Step 304: Once the index loading is complete, send the response message back to the master node. The response message carries the version information of the index and information on the data sharing requirements.

[0113] For example, the master node refreshes and generates an index snapshot (version v1), creating files f1 and f2. It commits the v1 version of the index and then writes f1 and f2 to shared storage. The master node then sends the v1 version index snapshot to the slave nodes. Upon receiving the v1 version index snapshot, the slave nodes load the corresponding files f1 and f2 as needed, i.e., they copy files f1 and f2 from shared storage to their own memory. After copying files f1 and f2, the loading of the files corresponding to the v1 version index snapshot is complete. The master node then generates a response message: "v1 version index snapshot successfully loaded; shared files f1 and f2 required," and returns this response message to the master node.

[0114] One possible approach also includes:

[0115] Based on the index, determine the data to be read; if the data to be read exists in the shared storage, retrieve the data to be read from the shared storage.

[0116] If the data to be read does not exist in the shared storage, the data to be read is obtained from the master node.

[0117] The data to be read can be the data that the slave node needs to load based on the index.

[0118] In practical applications, file retrieval rules can be set for slave nodes. For example, physical replication can be configured to send files in memory only over the network. For files already persisted to shared storage, slave replicas can load them directly from shared storage. Files in memory are sent to slave replicas over the network, reducing the pressure on shared storage metadata writes.

[0119] For example, when a slave node receives a v2 version index snapshot, it loads the file f3 corresponding to the v2 version index snapshot as needed. Since file f3 is not persisted to shared storage, it can be pulled directly from the master node's memory.

[0120] For example, when a slave node receives a v1 version index snapshot, it loads the files f1 and f2 corresponding to the v1 version index snapshot as needed. Since files f1 and f2 have been persisted to shared storage, they can be directly pulled from shared storage.

[0121] This specification provides a shared file management method and apparatus. The shared file management method includes: generating an index of the data and version information of the index for write operations on the data; sending the index and the corresponding version information to a slave node of a master node, wherein the index enables the slave node to provide read services for the data based on the index; receiving a response message from the slave node after loading the index, wherein the response message carries the version information of the index and data sharing requirement information; when target data needs to be processed, comparing the version information of the index carried in the response message with the latest version information of the master node's index; if the version information of the index carried in the response message is consistent with the latest version information of the master node's index, determining the operation on the target data based on the data sharing requirement information carried in the response message. By receiving the response message from the slave node and operating on the target data according to the version information and sharing requirement information carried in the response message, since the response message is based on the index feedback, carrying the version information and sharing requirement information in the response message reuses the communication link for index feedback, eliminating the need for a separate communication link and reducing system complexity.

[0122] This specification also provides a system comprising: a master node that applies the file sharing management method as described in any of the above embodiments, and a slave node that applies the file sharing management method as described in any of the above embodiments.

[0123] For example, the system can be a distributed storage system, including: a master node and slave nodes for shared storage;

[0124] The master node is configured to generate an index and version information of the data for write operations, and send the index and its corresponding version information to its slave nodes. The index enables the slave nodes to provide read services for the data. The master node receives a response message from the slave nodes after loading the index. This response message carries the index version information and data sharing requirements. If target data needs to be processed, the master node compares the index version information carried in the response message with the latest version information of the master node's index. If the index version information in the response message matches the latest version information of the master node's index, the master node determines the operation on the target data based on the data sharing requirements carried in the response message.

[0125] The slave node is configured to receive the index and the version information corresponding to the index from the master node, and to send the response message back to the master node when the index is loaded.

[0126] In practical applications, two rules for file deletion can be set. A file marked for deletion must meet either of the following conditions to still be shared; otherwise, it can be physically deleted. First, the version information of the index snapshot of the file marked for deletion is greater than the version information of the index snapshot already loaded by any slave node, indicating that it will still be used in subsequent physical copying processes. Second, the file marked for deletion will be used by any slave node and is not in the slave node's memory, meaning that the slave node needs to read shared storage to load the file.

[0127] In one feasible approach, see [link to relevant documentation] Figure 4 , Figure 4 A timing diagram of the system's processing is shown. For example... Figure 4 As shown, the processing procedure includes:

[0128] Step S1: When the master node receives newly written data in its buffer, the master node refreshes to generate index files, creating files f1 and f2, as well as a snapshot of the current version of the index. The version information of this snapshot is v1, and a mapping {f1:v1, f2:v1} is generated.

[0129] Step S2: The master node commits the v1 version index, and then f1 and f2 can be written to the shared storage.

[0130] Step S3: The master node performs physical replication, including: the master node sends a v1 version index snapshot to the slave node.

[0131] In step S4, upon receiving the v1 version index snapshot, the slave node loads the files f1 and f2 corresponding to the v1 version index snapshot as needed; that is, it copies files f1 and f2 from shared storage to the slave node's own memory. After completing the copying of files f1 and f2, the loading of the files corresponding to the v1 version index snapshot is complete.

[0132] Step S5: The slave node generates a response message for the index: "The v1 version index snapshot has been successfully loaded. Files f1 and f2 need to be shared." This response message is returned to the master node. The master node receives the response message for the v1 version index snapshot: "The v1 version index snapshot has been successfully loaded. Files f1 and f2 need to be shared." Since the response message declares the sharing of files f1 and f2, f1 and f2 are in a shared state and cannot be deleted.

[0133] In step S6, if newly written data is received in the master node's buffer, the master node refreshes to generate an index file, file f3, and a snapshot of the current version of the index, with version information v2. A mapping is then generated: {f1:v2,f2:v2,f3:v2}.

[0134] In step S7, the master node receives the instruction to delete file f3. At this time, it judges the version information of the index snapshots of the master node and the slave node. Since the version information of the current index snapshot of the master node is v2, which is greater than the version information v1 of the index snapshot that the slave node has already loaded, it can only mark f3 for deletion, that is, add f3 to the list of pending deletion files.

[0135] Step S8: The master node performs physical replication, including: the master node sends a v2 version index snapshot to the slave node.

[0136] Step S9: When the slave node receives the v2 version index snapshot, it loads the file f3 corresponding to the v2 version index snapshot as needed. Since the file f3 is not persisted to the shared storage, it can be transferred over the network and directly pulled from the master node's memory.

[0137] In step S10, after the slave node completes the copying of file f3, it has completed loading the file corresponding to the v2 version index snapshot and generates a response message for the index: "The v2 version index snapshot has been successfully loaded. Files f1 and f2 need to be shared." The slave node then returns this response message to the master node.

[0138] In step S11, the master node receives the instruction to delete files f1 and f2. Based on the sharing declaration in the most recently received response message, it determines whether files f1 and f2 are used on the slave node. Since there are still query requests on the slave node that will use f1 and f2, it can only mark f1 and f2 for deletion, that is, add f1 and f2 to the list of files waiting to be deleted.

[0139] Step S12: The query for files f1 and f2 in the node ends, meaning that files f1 and f2 have been used.

[0140] In step S13, the slave node uses the physical replication process to send a new response message to the master node: "The v2 version index snapshot has been successfully loaded. There are no files to share."

[0141] In step S14, upon receiving the response message, the master node determines that the version information of the index snapshot from the replica is v2 and there are no shared files. It then removes files f1, f2, and f3 from the list of files to be deleted and deletes files f1, f2, and f3.

[0142] This specification provides a system. By receiving response messages from slave nodes, the system operates on target data based on the version information and sharing requirement information carried in the response messages. Since the response messages are based on index feedback, carrying version information and sharing requirement information in the response messages reuses the communication link for index feedback, eliminating the need for a separate communication link and reducing system complexity.

[0143] The following is in conjunction with the appendix Figure 5 Taking the application of the shared file management method provided in this specification in file deletion as an example, the shared file management method will be further explained. Among other things, Figure 5 The diagram illustrates a file deletion process of a shared file management method according to an embodiment of this specification. The master node receives a response message from the physical copy and triggers an attempt to delete the file, which specifically includes the following steps.

[0144] Step 502: The master node traverses the list of files waiting to be deleted.

[0145] Step 504: The master node determines whether a replica exists for the current index. If yes, proceed to step 506. If no, proceed to step 508.

[0146] Step 506: The master node determines whether the version information of the current index snapshot is greater than the version information of the index snapshot of any slave replica. If not, proceed to step 510. If yes, use the next index encountered as the current index and proceed to step 504.

[0147] Step 508: Delete the current index. After deleting the current index, use the next index encountered during traversal as the current index and execute step 504.

[0148] Step 510: The master node determines whether any slave replica is still using the current index. If yes, it uses the next index encountered during traversal as the current index and proceeds to step 504. If not, it proceeds to step 508.

[0149] By receiving response messages from slave nodes, the target data is manipulated based on the version information and sharing requirement information carried in the response messages. Since the response messages are based on index feedback, carrying version information and sharing requirement information in the response messages reuses the communication link for index feedback, eliminating the need for a separate communication link and reducing system complexity.

[0150] Corresponding to the above method embodiments, this specification also provides embodiments of a shared file management device. Figure 6 A schematic diagram of a shared file management device according to one embodiment of this specification is shown. Figure 6 As shown, the device includes:

[0151] The data sending module 602 is configured to generate an index of the data and version information of the index for write operations on the data, and send the index and the version information corresponding to the index to the slave node of the master node, wherein the index is used to enable the slave node to provide read services for the data based on the index;

[0152] The message receiving module 604 is configured to receive a response message from the slave node after the index has been loaded, wherein the response message carries version information of the index and information on the data sharing requirements.

[0153] The information comparison module 606 is configured to compare the version information of the index carried in the response message with the latest version information of the master node index when target data needs to be processed.

[0154] The data processing module 608 is configured to determine the operation on the target data based on the data sharing requirement information carried in the response message, provided that the version information of the index carried in the response message is consistent with the latest version information of the master node index.

[0155] In one possible implementation, the data sending module 602 is further configured as follows:

[0156] Submit the index to make it effective, and write the data to the shared storage based on the index.

[0157] In one possible implementation, the data processing module 608 is further configured as follows:

[0158] If the version information of the index carried in the response message is less than the latest version information of the master node index, a deletion mark is added to the target data.

[0159] In one possible implementation, the data processing module 608 is further configured as follows:

[0160] Based on the data sharing requirement information carried in the response message, determine whether the target data is in use; if so, add a deletion mark to the target data.

[0161] In one possible implementation, the data processing module 608 is further configured as follows:

[0162] Based on the data sharing requirement information carried in the response message, determine whether the target data is currently in use. If not, delete the target data according to the deletion operation for the target data.

[0163] In one possible implementation, the data processing module 608 is further configured as follows:

[0164] Based on the data sharing requirement information carried in the response message, determine whether the target data is currently in use. If not, delete the target data according to the deletion operation for the target data.

[0165] In one possible implementation, the data processing module 608 is further configured as follows:

[0166] After adding a deletion mark to the target data, an updated response message is received. Based on the sharing requirement information carried in the updated response message, it is determined whether the data with the deletion mark is in use. If not, the data with the deletion mark is deleted.

[0167] In one possible implementation, the data processing module 608 is further configured as follows:

[0168] Determine the sharing requirement information of the data marked with deletion in the updated response message;

[0169] Based on the sharing requirement information corresponding to the data marked with deletion, determine whether the data marked with deletion is currently in use.

[0170] This specification provides a shared file management method and apparatus. The shared file management apparatus includes: generating an index of the data and version information of the index for write operations on the data; sending the index and the corresponding version information to a slave node of a master node, wherein the index enables the slave node to provide read services for the data based on the index; receiving a response message from the slave node after loading the index, wherein the response message carries the version information of the index and data sharing requirement information; when target data needs to be processed, comparing the version information of the index carried in the response message with the latest version information of the master node's index; if the version information of the index carried in the response message is consistent with the latest version information of the master node's index, determining the operation on the target data based on the data sharing requirement information carried in the response message. By receiving the response message from the slave node and operating on the target data according to the version information and sharing requirement information carried in the response message, since the response message is based on the index feedback, carrying the version information and sharing requirement information in the response message reuses the communication link for index feedback, eliminating the need for a separate communication link and reducing system complexity.

[0171] Corresponding to the above method embodiments, this specification also provides embodiments of a shared file management device. Figure 7 A schematic diagram of another shared file management device provided in one embodiment of this specification is shown. Figure 7 As shown, the device includes:

[0172] Data receiving module 702 is configured to receive the index from the master node;

[0173] The message feedback module 704 is configured to send the response message to the master node when the index loading is complete.

[0174] In one possible implementation, the message feedback module 704 is also configured as follows:

[0175] Based on the index, determine the data to be read; if the data to be read exists in the shared storage, retrieve the data to be read from the shared storage.

[0176] If the data to be read does not exist in the shared storage, the data to be read is obtained from the master node.

[0177] This specification provides a shared file management method and apparatus, wherein the shared file management apparatus includes: receiving an index from a master node, and, upon completion of index loading, sending a response message back to the master node. By sending a response message to the master node, the master node can operate on the target data based on the version information and sharing requirement information carried in the response message. Since the response message is based on the index, carrying the version information and sharing requirement information in the response message reuses the communication link for index feedback, eliminating the need for a separate communication link and reducing system complexity.

[0178] The above is an illustrative scheme of a shared file management device according to this embodiment. It should be noted that the technical solution of this shared file management device and the technical solution of the shared file management method described above belong to the same concept. For details not described in detail in the technical solution of the shared file management device, please refer to the description of the technical solution of the shared file management method described above.

[0179] Figure 8 A structural block diagram of a computing device 800 according to one embodiment of this specification is shown. The components of the computing device 800 include, but are not limited to, a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and a database 850 is used to store data.

[0180] The computing device 800 also includes an access device 840, which enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 840 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0181] In one embodiment of this specification, the above-described components of the computing device 800 and Figure 8 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 8 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0182] The computing device 800 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 800 can also be a mobile or stationary server.

[0183] The processor 820 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described shared file management method.

[0184] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the shared file management method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the shared file management method described above.

[0185] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the shared file management method described above.

[0186] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the shared file management method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the shared file management method described above.

[0187] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described shared file management method.

[0188] The above is an illustrative example of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the shared file management method described above belong to the same concept. Details not described in detail in the computer program's technical solution can be found in the description of the technical solution of the shared file management method described above.

[0189] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0190] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0191] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0192] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0193] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A shared file management method, applied to a master node, comprising: For write operations on the data, an index of the data and version information of the index are generated, and the index and the corresponding version information are sent to the slave node of the master node. The index is used to enable the slave node to provide read services for the data based on the index. The system receives a response message from the slave node after loading the index, wherein the response message carries the version information of the index and information on the data sharing requirements. When it is necessary to process target data, the version information of the index carried in the response message is compared with the latest version information of the master node index. If the version information of the index carried in the response message is consistent with the latest version information of the master node index, the operation on the target data is determined based on the data sharing requirement information carried in the response message.

2. The method according to claim 1, further comprising: Submit the index to make it effective, and write the data to the shared storage based on the index.

3. The method according to claim 1, further comprising: If the version information of the index carried in the response message is less than the latest version information of the master node index, a deletion mark is added to the target data.

4. The method according to claim 1, wherein determining the operation on the target data based on the data sharing requirement information carried in the response message includes: Based on the data sharing requirement information carried in the response message, determine whether the target data is in use; if so, add a deletion mark to the target data.

5. The method according to claim 1, wherein determining the operation on the target data based on the data sharing requirement information carried in the response message includes: Based on the data sharing requirement information carried in the response message, determine whether the target data is currently in use. If not, delete the target data according to the deletion operation for the target data.

6. The method according to claim 3 or 4, further comprising: After adding a deletion mark to the target data, an updated response message is received. Based on the sharing requirement information carried in the updated response message, it is determined whether the data with the deletion mark is in use. If not, the data with the deletion mark is deleted.

7. The method according to claim 6, wherein determining whether data with a deletion mark is being used includes: Determine the sharing requirement information of the data marked with deletion in the updated response message; Based on the sharing requirement information corresponding to the data marked with deletion, determine whether the data marked with deletion is currently in use.

8. The method according to claim 1, further comprising, after generating the index of the data and the version information of the index for the write operation on the data: Establish a mapping relationship between the index of the data and the version information of the index.

9. A shared file management method, applied to a slave node, comprising: Receive the index from the master node of the slave node; Once the index is loaded, a response message is sent back to the master node, which carries the version information of the index and information on the data sharing requirements.

10. The method of claim 9, further comprising: Based on the index, determine the data to be read; if the data to be read exists in the shared storage, retrieve the data to be read from the shared storage. If the data to be read does not exist in the shared storage, the data to be read is obtained from the master node.

11. A system comprising: A master node employing the shared file management method as described in any one of claims 1-8, and a slave node employing the file sharing management method as described in any one of claims 9-10, wherein the master node and the slave node share storage.

12. A shared file management device, applied to a master node, comprising: The data sending module is configured to generate an index of the data and version information of the index for write operations on the data, and send the index and the version information corresponding to the index to the slave node of the master node, wherein the index is used to enable the slave node to provide read services for the data based on the index; The message receiving module is configured to receive a response message from the slave node after the index has been loaded, wherein the response message carries the version information of the index and information on the data sharing requirements. The information comparison module is configured to compare the version information of the index carried in the response message with the latest version information of the master node index when the target data needs to be processed. The data processing module is configured to determine the operation on the target data based on the data sharing requirement information carried in the response message, provided that the version information of the index carried in the response message is consistent with the latest version information of the master node index.

13. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the file sharing management method according to any one of claims 1 or 8 or any one of claims 9 or 10.

14. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the file sharing management method of any one of claims 1 or 8 or 9 or 10.

Images