Data processing method, target working node in distributed system and distributed system

Abstract

The embodiment of the invention provides a data processing method, a target working node in a distributed system and the distributed system.The data processing method is applied to the target working node in the distributed system and comprises the steps that under the condition that an access instruction for a target partition is received, a to-be-verified access label of the target partition is obtained; a condition access request of the target partition is generated according to the to-be-verified access label, the target partition is accessed based on the condition access request, the to-be-verified access label is used for being matched with a target access label, and whether the target working node has the access permission of the target partition or not is determined based on a matching result; the target access tag is an access tag currently stored in the file system, and the conditional access request is used for indicating that access is allowed under the condition that the target working node has the access permission of the target partition. And whether the target working node has the access permission of the target partition is verified by using the to-be-verified access label, so that the partition data consistency is ensured.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of computer technology, and in particular, to a data processing method, a target worker node in a distributed system, a distributed system, a computing device, a computer-readable storage medium, and a computer program product. BACKGROUND

[0002] With the rapid development of computer technology, more and more partition-based distributed systems begin to be restructured, using file system cloud products as storage bases to realize storage and computing separation and serverless. A distributed data system realizes horizontal expansion through partitioning, and a partition refers to a data subset. In order to realize high-availability partition service, when an abnormal worker node of a partition occurs, the master node will migrate the partition to other worker nodes, and blocking the operation of the abnormal node on the partition data is the key to guaranteeing the consistency of the partition data.

[0003] At present, the operation of the abnormal node on the partition data is blocked by renaming the root directory of the partition, but the master node needs to generate a new root directory for the partition each time the partition is migrated, which makes the implementation and operation of the distributed system more complex. Therefore, there is an urgent need for an effective technical solution to solve the above problems. SUMMARY

[0004] Therefore, the embodiments of the present specification provide a data processing method. One or more embodiments of the present specification also relate to a target worker node in a distributed system, a distributed system, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects in the prior art.

[0005] According to a first aspect of the embodiments of the present specification, a data processing method is provided, applied to a target worker node in a distributed system, and the method comprises:

[0006] In the case of receiving an access instruction for a target partition, an access tag to be verified of the target partition is obtained, wherein the target partition is any partition in the distributed system for storing data by a file system;

[0007] A conditional access request of the target partition is generated according to the access tag to be verified, and the target partition is accessed based on the conditional access request, wherein the access tag to be verified is used to match a target access tag, and based on the matching result, it is determined whether the target worker node has access permission of the target partition, the target access tag is a currently stored access tag in the file system, and the conditional access request is used to indicate that access is allowed in the case that the target worker node has access permission of the target partition.

[0008] According to a second aspect of the embodiments of the present specification, a target worker node in a distributed system is provided, comprising:

[0009] an obtaining module configured to, in a case where an access instruction for a target partition is received, obtain a to-be-verified access tag of the target partition, wherein the target partition is any partition in the distributed system used by a file system to store data;

[0010] an access module configured to generate a conditional access request of the target partition according to the to-be-verified access tag, and access the target partition based on the conditional access request, wherein the to-be-verified access tag is used to match a target access tag, a result of the matching is used to determine whether the target worker node has access permission of the target partition, the target access tag is a currently stored access tag in the file system, and the conditional access request is used to indicate that access is allowed in a case where the target worker node has access permission of the target partition.

[0011] According to a third aspect of the embodiments of the present specification, a distributed system is provided, comprising a file system and at least two worker nodes;

[0012] a target worker node configured to, in a case where an access instruction for a target partition is received, obtain a to-be-verified access tag of the target partition, wherein the target worker node is any one of the at least two worker nodes, and the target partition is any partition in the distributed system used by a file system to store data; generate a conditional access request of the target partition according to the to-be-verified access tag, and access the file system based on the conditional access request;

[0013] the file system configured to receive the conditional access request, match the to-be-verified access tag with a target access tag, determine whether the target worker node has access permission of the target partition based on a result of the matching, and allow access to the target partition in a case where the target worker node has access permission of the target partition, wherein the target access tag is a currently stored access tag in the file system.

[0014] According to a fourth aspect of the embodiments of the present specification, a computing device is provided, comprising:

[0015] a memory and a processor;

[0016] the memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above data processing method.

[0017] According to a fifth aspect of the embodiments of the present specification, a computer readable storage medium is provided, which stores computer programs / instructions, which, when executed by a processor, implement the steps of the above data processing method.

[0018] According to a sixth aspect of the embodiments of the present specification, a computer program product is provided, which includes computer programs / instructions, which, when executed by a processor, implement the steps of the above data processing method.

[0019] One embodiment of the present specification provides a data processing method applied to a target work node in a distributed system, in the case of receiving an access instruction for a target partition, obtaining a to-be-verified access tag of the target partition, wherein the target partition is any partition in the distributed system for storing data by a file system; generating a conditional access request of the target partition according to the to-be-verified access tag, and accessing the target partition based on the conditional access request, wherein the to-be-verified access tag is used to match a target access tag, and based on the matching result, it is determined whether the target work node has access permission of the target partition, the target access tag is a currently stored access tag in the file system, and the conditional access request is used to indicate that access is allowed in the case that the target work node has access permission of the target partition.

[0020] In the above method, the target work node generates a conditional access request carrying a to-be-verified access tag when accessing the target partition, and accesses the target partition in the file system based on the conditional access request. The file system can match the to-be-verified access tag in the conditional access request with the target access tag currently stored in the file system to determine whether the target work node has access permission of the target partition, and access is allowed in the case that the target work node has access permission of the target partition. In this way, since the access permission of the target partition changes when the partition migration occurs, the target access tag currently stored in the file system will change accordingly. When the target work node accesses the target partition, the to-be-verified access tag carried in the conditional access request is matched with the target access tag currently stored in the file system to determine whether the target work node has access permission of the target partition. Only the work node with access permission of the target partition can access the target partition in the file system, which blocks the operation of the old abnormal node on the partition data, guarantees the consistency of the partition data, does not need to repeatedly change the root directory of the partition, simplifies the implementation and operation and maintenance of the distributed system, and facilitates the distributed system to implement the storage-computing separation architecture and serverless. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is an application scenario diagram of a data processing method provided by one embodiment of the present specification;

[0022] Figure 2 is a flowchart of a data processing method provided by one embodiment of the present specification;

[0023] Figure 3 is a process flowchart of a first data processing method in a distributed system provided by one embodiment of the present specification;

[0024] Figure 4 is a process flowchart of a second data processing method in a distributed system provided by one embodiment of the present specification;

[0025] Figure 5 is a process flowchart of a third data processing method in a distributed system provided by one embodiment of the present specification;

[0026] Figure 6 is a structural schematic diagram of a target working node in a distributed system provided by one embodiment of the present specification;

[0027] Figure 7 is a structural schematic diagram of a distributed system provided by one embodiment of the present specification;

[0028] Figure 8 is a structural block diagram of a computing device provided by one embodiment of the present specification. DETAILED DESCRIPTION

[0029] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present specification. However, the present specification can be practiced without the specific details, other than in the examples, and it is understood that the scope of the present specification is not limited to the details below.

[0030] The terms used in one or more embodiments of the present specification are merely for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present specification. The singular forms "a", "an" and "the" used in one or more embodiments of the present specification and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of the present specification means and includes any or all possible combinations of one or more associated listed items.

[0031] It should be understood that, although the terms first, second, etc. can be employed in describing various information in one or more embodiments of the present specification, the information should not be limited to these terms. These terms are only used to distinguish one type of information from another type of information. For example, without departing from the scope of one or more embodiments of the present specification, first can also be referred to as second, and similarly, second can also be referred to as first. Depending on the context, the word "if' as used herein can be interpreted as "when" or "upon" or "in response to determining".

[0032] In addition, it should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in one or more embodiments of the present specification are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0033] First, the nomenclature involved in one or more embodiments of the present specification is explained.

[0034] Conditional interface: the interface of a software or service is a set of functions representing a set of operations. Some conditional parameters can be included in the interface parameters, and when the specified conditions are matched inside the software or service, the operation is executed, otherwise the operation is refused. The interface with conditional parameters is called conditional interface.

[0035] Metadata: mainly information describing the properties of data, used to support functions such as indicating storage location, historical data, resource lookup, file recording, etc.

[0036] Distributed system: a system composed of multiple nodes for storing and accessing data, usually composed of master nodes and worker nodes. Data sets are divided into multiple data subsets according to intervals or hash values. The master node manages the meta information of the data subsets, such as data subset division information, the worker node where the data subset is located, etc. The worker node manages multiple data subsets and provides the ability to store and access data in them.

[0037] Rename function: its function is to rename a file. With this function, the file moving function can be realized. Changing the drive letter of the complete path of a file realizes the moving of the file, that is, the path of the file can be modified by using this function.

[0038] With the evolution of technology architecture and product mode such as "storage computing separation", "cloud product serverless", more and more partition-based distributed systems begin to be restructured, using file system cloud products as storage base to realize storage computing separation and serverless, such as cloud data warehouse, cloud message queue, etc.

[0039] Distributed systems realize horizontal expansion through partitioning, and data sets are divided into multiple data subsets according to intervals or hash values, commonly known as partitions, that is, a partition refers to a data subset. For example, the data of a single message queue is divided into multiple partitions, and the data of a single database table is divided into multiple partitions.

[0040] The master node manages the meta information of the data subset, such as data subset division information, the working node where it is located, etc., and the working node manages multiple data subsets and provides the ability to store and access data in them.

[0041] In order to realize storage computing separation and serverless, the data corresponding to the partitions of the distributed system is usually stored in the cloud file system service, such as the file system HDFS (Hadoop Distributed File System). In order to ensure the consistency of the data of a single partition, a single working node is generally used to update the partition, which is called a write node, and in order to improve the ability to read the partition, multiple working nodes are generally used to read the partition, which is called a read node.

[0042] In a distributed system, hardware failure, software problems, etc. of the working node cannot be avoided. When the working node has hardware failure, software problems, etc., the customer cannot access the partition managed by the working node, and the service is unavailable, affecting customer use. In order to realize high-availability partition service, when the partition service process fails, the master node will migrate the partition from the failed working node to other normal working nodes.

[0043] The specific migration process can be as follows: the master node detects that the original write node W of the partition P has failed; the master node finds a normal node W' as the new write node of the partition P; the master node initiates a request to the original write node W to unload the partition P, requiring W to no longer serve the partition P and cannot update the data of the partition P; the master node initiates a request to the new write node W' to load the partition P, requiring W' to serve the partition P and can update the data of the partition P; the working node W unloads the partition P; the working node W' loads the partition P.

[0044] In a distributed system, due to network partition, execution order, etc., the write nodes W and W' may write data of the partition P at the same time, which will destroy the data consistency of the partition P. In order to ensure the data consistency of the partition P, during the partition migration process, the original write node W needs to be blocked from accessing the data of the partition P. The data of the partition P is stored on a file system, and the data of the file system includes metadata and data. A key part of blocking the original write node W from accessing the data of the partition P is to block it from updating the metadata of the file system.

[0045] In a possible implementation, the distributed system usually uses a rename function (renaming function) of a file system to block the original write node W from updating the metadata of the file system during the partition migration process. This scheme includes two parts. The first part is a partition directory structure. The directory structure of the partition P on the file system needs to be designed in advance. For example, / data / P / epoch / represents the root directory of the data of the partition P, where epoch is a monotonically increasing number in the root directory, representing the number of times of partition migration. The second part is a scheme for blocking the abnormal node from updating the metadata of the file system. During the partition migration process, the new write node W' obtains two file system paths according to the original epoch E0 and the new epoch E1, that is, / data / P / E0 and / data / P / E1. The new write node W' calls the file system rename(" / data / P / E0", " / data / P / E1") function to rename the root directory of the partition P to the new path. Since the original write node W only knows the epoch E0 path of the partition P, when the original write node W accesses the data of the partition P, / data / P / E0 is still used as the root directory. Since the root directory of the partition P has been renamed, the original write node W will fail when updating the metadata of the file system, that is, the original write node W is blocked from updating the metadata of the file system.

[0046] However, in the above scheme, the root directory of the partition is dynamically changed. The master node needs to generate a new root directory for the partition each time the partition is migrated, and modify the root directory of the partition. This not only blocks the abnormal write node from updating the metadata of the file system, but also blocks the read node from reading the metadata of the file system. After the partition migration, the read node needs to query the master node or the file system to determine the root directory of the partition, and then read the data in the partition. In addition, since the root directory of the partition is dynamically changed, when investigating a problem, the current root directory of the partition needs to be queried from the system constantly, which makes the implementation and operation of the distributed system more complex.

[0047] To address the aforementioned issues, in one optional implementation of this specification, a method for blocking abnormal nodes from updating file system metadata based on a conditional interface is proposed. By configuring an access tag on the target partition to indicate whether the target worker node has access permissions to the target partition, operations on the partition data by the old abnormal node can be blocked, ensuring partition data consistency. This eliminates the need to repeatedly change the root directory of the partition, simplifies the implementation and operation of the distributed system, and facilitates the implementation of a storage-compute separation architecture and serverless architecture in the distributed system.

[0048] This specification provides a data processing method, and also relates to a target worker node in a distributed system, a distributed system, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail in the following embodiments.

[0049] See Figure 1 , Figure 1 A schematic diagram illustrating an application scenario of a data processing method provided according to an embodiment of this specification is shown.

[0050] Figure 1 The system includes a client 102 and a distributed system 104. The distributed system 104 includes a master node 1042, multiple worker nodes 1044, and a file system 1046. The master node 1042 manages the metadata of data subsets, such as data subset partitioning information and the worker node it belongs to. The worker nodes 1044 manage multiple data subsets and provide the ability to store and access the data within them. The file system 1046 is a distributed file system that provides storage services for the data corresponding to the partitions of the distributed system.

[0051] In practice, the user issues an access command through client 102. This access command can be, for example, a request to create new data, a request to read data, or a request to modify data. Client 102 sends this access command to distributed system 104. Distributed system 104 determines the corresponding target partition for this access command, and worker node 1044 provides the corresponding access service based on this target partition. The service logic of this worker node is as follows: Upon receiving an access command for the target partition, it obtains the access tag to be verified for the target partition; it generates a conditional access request for the target partition based on the access tag to be verified; and it accesses the target partition based on the conditional access request. The access tag to be verified is used to match the target access tag, and based on the matching result, it determines whether the worker node has access rights to the target partition. The conditional access request indicates that access to the target partition is permitted if the worker node has access rights to the target partition.

[0052] If the current working node of the target partition is abnormal, the master node migrates the target partition, and the update working node taking over the target partition can update the target access label in the file system. For the update working node, the to-be-verified access label carried in the conditional access request initiated by the update working node is used by the target access label in the updated file system, which matches the target access label currently stored in the file system. Therefore, the update working node has access permission of the target partition and can successfully access the target partition. For the current working node that is abnormal, the to-be-verified access label carried in the conditional access request initiated by the current working node is an old access label, and the target access label currently stored in the file system has been updated by the update working node. That is, for the current working node that is abnormal, the to-be-verified access label carried in the conditional access request does not match the target access label currently stored in the file system, the current working node that is abnormal does not have access permission of the target partition, and cannot access the target partition, thereby realizing blocking of the abnormal node to access the target partition in the file system.

[0053] The client 102 can include a browser, an application (APP), or a web application such as a Hyper Text Markup Language 5 (H5) application, or a light application (also known as a small program, a lightweight application), or a cloud application, and the like. The end-side device can be developed based on a software development kit (SDK) of a corresponding service provided by a server side, such as a real-time communication (RTC) SDK. The end-side device can be deployed in an electronic device and needs to be run in dependence on a device or an APP in the device. The electronic device can have a display screen and support information browsing, and the like, and can be a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, and the like. Various other types of applications can also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant communication tools, mailbox clients, social platform software, and the like.

[0054] Referring to Figure 2 , Figure 2 A flowchart of a data processing method according to one embodiment of the present specification is shown, which is applied to a target working node in a distributed system, and specifically includes the following steps 202-204.

[0055] Step 202: In the case of receiving an access instruction for a target partition, obtaining a to-be-verified access label of the target partition, wherein the target partition is any partition of a file system in a distributed system for storing data.

[0056] In actual implementation, the distributed system includes a master node, at least two worker nodes, and a file system, which is a distributed file system, and is configured to provide storage services for data corresponding to a plurality of partitions, to realize storage and computing separation and serverless, and one worker node serves one corresponding partition and provides the ability to store and access data in the partition. The target worker node is any worker node in the distributed system that wants to access the target partition.

[0057] It should be noted that, since the write instruction changes the data in the partition, if the worker nodes before and after the migration can update the data in the partition, the data consistency will be destroyed, and the read instruction reads the data in the partition and does not update the data, and multiple nodes simultaneously reading will not destroy the data consistency, so the read instruction can not be conditionally checked, that is, the target worker node before and after the migration or other worker nodes can read the data of the target partition. That is, the target worker node in the embodiment is any write node, and the access instruction is a write instruction. Of course, for the read instruction, conditionally checking can also be performed using the access tag, and the data of the target partition is allowed to be read after the checking is passed. The embodiments of the present application do not limit this, that is, the access instruction can be a read instruction in addition to the write instruction, and the target worker node can also be a read node.

[0058] In the embodiment, the to-be-checked access tag can be used to check whether the target worker node that wants to access the target partition has access permission of the target partition. In the case where the target worker node receives an access instruction triggered by a user through a client, the target worker node can determine the target partition to be accessed by the access instruction, obtain the to-be-checked access tag of the target partition, and subsequently generate a conditional access request based on the to-be-checked access tag to check the to-be-checked access tag based on the target access tag currently stored in the file system, to determine whether the target worker node that wants to access the target partition is a worker node currently responsible for serving the target partition, that is, to determine whether the target worker node that wants to access the target partition has access permission of the target partition.

[0059] The access permission of the target partition can be affected by partition migration, for example, worker node A has access permission of partition P, if partition P is migrated from worker node A to worker node B, then worker node B has access permission of partition P, and worker node A does not have access permission of partition P. Therefore, the partition migration information can be used as the to-be-checked access tag to indicate the migration of the target partition, so as to check whether the target worker node has access permission of the target partition based on the partition migration information. Alternatively, in actual implementation, the node identifier of the worker node serving the target partition can also be directly used as the to-be-checked access tag, and subsequently it is checked whether the to-be-checked node identifier of the target partition is the target node identifier stored in the file system, so as to check whether the target worker node has access permission of the target partition.

[0060] Specifically, each partition in the file system corresponds to an access path, that is, each partition has a corresponding root directory, and the working node can access the corresponding partition according to the root directory. The to-be-verified access tag can be an extended attribute tag set on the root directory of the target partition. And / or, the to-be-verified access tag is a flag file or flag directory created for the target partition.

[0061] It should be noted that the directory structure of the root directory of the partition on the file system is / data / P / , and the data root directory of the partition is / data / P / no matter how many times the partition is migrated. P is the partition identifier, which does not change in the partition life cycle. In this embodiment, an extended attribute tag can be configured on the root directory of the partition, and the extended attribute tag is used as the to-be-verified access tag. For example, an extended attribute (Extended Attributes) epoch representing the number of partition migrations can be set on the directory / data / P / . Alternatively, a flag file or flag directory, for example, / data / P / lock, can be created for the target partition, and the file or directory is used as the to-be-verified access tag for subsequent conditional verification.

[0062] In the embodiments of the present specification, in the case where the access instruction for the target partition is received, the to-be-verified access tag of the target partition can be acquired, and the to-be-verified access tag can be carried in the subsequent conditional access request, so as to determine whether the target working node to be accessed to the target partition is the working node newly taking over the target partition, to avoid the working nodes before and after the migration updating the data of the target partition at the same time, and to ensure data consistency.

[0063] In an optional implementation of the embodiment, the to-be-verified access tag includes to-be-verified tag content; and the to-be-verified access tag of the target partition is acquired by:

[0064] The to-be-verified tag content of the target partition is read from the local.

[0065] In actual implementation, the to-be-verified access tag can include a tag title and to-be-verified tag content. The to-be-verified tag content will change continuously with the migration of the partition. The to-be-verified tag content can be a numerical value representing the number of partition migrations, or a character or word identifying the working node currently serving the target partition.

[0066] It should be noted that the target working node can read the to-be-verified tag content of the target partition from the local. The to-be-verified tag content will change continuously with the migration of the partition. Through the to-be-verified tag content under the tag title which is dynamically updated based on the migration of the partition, it is identified whether the target working node accessing the target partition is the working node newly taking over the target partition, so as to perform subsequent access verification.

[0067] In an optional implementation of the embodiment, the target access tag includes target tag content; the distributed system further includes a master node, and the method further includes:

[0068] receiving first updated tag content issued by the master node, wherein the first updated tag content is sent by the master node in the case that the current working node of the target partition is abnormal;

[0069] querying target tag content of the target access tag from a file system;

[0070] updating the target tag content in the file system according to the first updated tag content.

[0071] In actual implementation, if the current working node of the target partition is abnormal and cannot provide services of the target partition, the master node selects a working node as an updated working node from other normal working nodes of the distributed system, and then issues the first updated tag content to the updated working node. That is to say, if the target working node initiating the access request is the updated working node in the embodiment, the target working node can receive the first updated tag content issued by the master node, and then can update the target tag content in the file system by using the first updated tag content. The target working node is a working node having access permission of the target partition.

[0072] The first updated tag content is generated and issued by the master node based on partition migration information. For example, if the first updated tag content represents the number of times of partition migration, the first updated tag content is increased by one each time of migration. If the first updated tag content is a character or a word, the first updated tag content indicates the working node after migration each time of migration.

[0073] For example, it is assumed that the target working node initiating the access request is an updated working node W', the access tag is epoch, and the epoch value issued by the master node to the updated working node W' is epoch_new. The updated working node W' queries the epoch value of the target partition in the file system, which is represented by epoch_old, and updates the epoch_old in the file system based on the epoch_new.

[0074] In the embodiments of the present specification, if the target worker node initiating the access request in the present embodiment is the update worker node, the target worker node can receive the first update label content issued by the master node, and when accessing the target partition, the to-be-verified label content obtained from the local is the first update label content newly issued by the master node, the first update label content matches the target label content in the file system, at this time, the target worker node has the access right of the target partition, which ensures that the update worker node can successfully access the target partition; and if the target worker node initiating the access request in the present embodiment is not the update worker node, the target worker node cannot receive the first update label content issued by the master node, and when accessing the target partition, the to-be-verified label content obtained is the old label content stored in the local, the old label content does not match the target label content in the file system, at this time, the target worker node does not have the access right of the target partition, and cannot access the target partition, which blocks other worker nodes except the update worker node from accessing the target partition, and ensures the data consistency.

[0075] In an optional implementation of the present embodiment, updating the target label content in the file system according to the first update label content comprises:

[0076] In the case that the first update label content and the target label content meet the update condition, the first update label content is used to replace the target label content in the file system.

[0077] In the case that the first update label content and the target label content do not meet the update condition, the update is prohibited.

[0078] It should be noted that if the current worker node of the target partition is abnormal and cannot provide the service of the target partition, the master node will select one of the other normal worker nodes of the distributed system as an update worker node, the master node will issue the first update label content to the update worker node, and if the update worker node does not respond for a long time, the master node will select the next update worker node and issue the second update label content. The first update label content is the label content issued by the master node to the current update worker node, and the second update label content is the label content issued by the master node to the next update worker node.

[0079] In actual application, in a distributed system, there can be more work nodes actually participating in the process, and a global order needs to be determined through comparison. The master node can be configured with the global order of each work node, which can be configured based on the performance, type, usage, and other attribute parameters of each work node. In the case of an exception of the current work node of the target partition, the master node can determine an updated work node based on the global order. If the updated work node does not respond for a long time, the master node determines the next updated work node based on the global order of each work node, and notifies the next updated work node to take over the target partition.

[0080] Specifically, if the master node issues the first updated label content to the first updated work node, but the first updated work node does not respond in time, and the master node issues the second updated label content to the next updated work node, and the next updated work node updates the target label content in the file system, if the first updated work node updates the target label content in the file system according to the first updated label content again, it will cause confusion of the label content in the file system and abnormal service. Therefore, after receiving the first updated label content, the updated work node cannot directly update the target label content in the file system according to the received first updated label content, but needs to judge whether the received first updated label content and the target label content meet the update condition, if yes, replace the target label content in the file system with the first updated label content, if not, prohibit updating. The update condition is pre-configured, which is used to determine whether the first updated label content is the latest label content issued by the master node.

[0081] For example, assuming that there are three work nodes A, B, and C, A is the current work node, and the label content of the access label stored locally is epoch A. When work node A is abnormal, the master node lets work node B take over and issues epoch B to work node B. However, the service process of work node B has not responded to the master node for a long time due to some problems. The master node believes that work node B cannot take over, and lets work node C take over and issues epoch C to work node C. Work node C modifies the target label content in the file system to epoch C. At this time, the service process of work node B is restored and prepares to take over the target partition. Work node B compares the epoch B received by itself with the current epoch C in the file system, and determines that the label content received by itself is not the latest label content issued by the master node, and does not meet the update condition. Work node B gives up taking over the target partition and prohibits updating.

[0082] In a specific implementation, if it is confirmed that the first update label content and the target label content meet the update condition, the first update label content is assigned to the target access label in the file system by using a SetXAttr function, so that the target label content of the target access label in the file system is updated to the first update label content. The SetXAttr function is used to set an attribute value, and the parameters of the function are an object, a string, and an arbitrary value. The string can name an existing attribute or a new attribute. If the object is allowed, the function assigns the value to the attribute.

[0083] For example, the value of the / data / P / extension attribute epoch can be updated by using a SetXAttr(" / data / P / ", "epoch", epoch_new, condition) function. The epoch is a label title of the target access label, the value of the epoch is the target label content of the target access label, / data / P / indicates that the partition P is to be updated, and the condition indicates a condition parameter to be updated. The condition includes condition.path, condition.xAttrKey, and condition.xAttrValue. The condition.path indicates that the partition to be updated exists, the condition.xAttrKey indicates the label title of the target access label, and the condition.xAttrValue indicates the label value of the target access label. The condition.path=" / data / P / ", that is, the partition P is to be modified. The condition.xAttrKey="epoch", that is, the access label epoch is to be modified. The condition.xAttrValue=epoch_new, that is, the label content of the access label epoch is epoch_new (the first update label content). The function is used to set the target label content of the target access label epoch in the file system to epoch_new, so that the label content update is implemented.

[0084] In the embodiments of the present disclosure, it can be determined whether the first update label content and the target label content meet the update condition. If yes, the first update label content is used to replace the target label content in the file system. If not, the update is prohibited, so that the target label content in the file system is prevented from being updated by multiple update work nodes, the partition migration is implemented, and the normal service of the distributed system is ensured.

[0085] In an optional implementation of the embodiment, the first updated label content is the partition migration number issued by the master node, and the target label content is the partition migration number indicated by the target access label currently stored in the file system. Before the first updated label content is used to replace the target label content in the file system in the case where the first updated label content and the target label content meet the update condition, the method further includes:

[0086] In the case where the target label content is less than the first updated label content, it is determined that the first updated label content and the target label content meet the update condition.

[0087] In the case where the target label content is not less than the first updated label content, it is determined that the first updated label content and the target label content do not meet the update condition.

[0088] In actual application, the first updated label content is the partition migration number issued by the master node, and the target label content is the partition migration number indicated by the target access label currently stored in the file system. In the case where the target label content is less than the first updated label content, it is indicated that the partition migration number currently stored in the file system is the old partition migration number, no new migration occurs, and the new partition migration number issued by the master node is not used. At this time, it can be determined that the first updated label content and the target label content meet the update condition, and the first updated label content is used to replace the target label content in the file system, so that the target label content in the file system is the updated partition migration number. In the case where the target label content is not less than the first updated label content, it is indicated that the target label content in the file system is greater than or equal to the first updated label content, and it is indicated that the first updated label content received by the update worker node is not the current latest partition migration number. At this time, it is determined that the first updated label content and the target label content do not meet the update condition, and the update is prohibited.

[0089] In the above example, A is the current worker node, the label content epoch of the access label stored locally is 1, the master node lets the worker node B take over when the worker node A abnormally, and the epoch issued to the worker node B is 2. The target label content epoch obtained by the worker node B from the file system is 1. Since the target label content epoch in the file system is 1, which is less than the value of the epoch received by the worker node B, the worker node B can update the target label content epoch in the file system to 2.

[0090] In addition, assuming that the master node issues epoch 2 to the worker node B, the service process of the worker node B does not reply to the master node for a long time due to some problems, the master node considers that the worker node B cannot take over, and lets the worker node C take over, and issues epoch 3 to the worker node, and the worker node C modifies the target label content epoch in the file system to 3. At this time, the service process of the worker node B recovers and is ready to take over the target partition, the worker node B receives epoch 2, but the current epoch in the file system is 3, that is, the value of the target label content epoch in the file system is greater than the value of the epoch received by the worker node B, which indicates that the target partition has been successfully taken over by other worker nodes, the worker node B determines that the received label content is not the latest label content issued by the master node, and the update condition is not met, the worker node B gives up taking over the target partition, and the update ends.

[0091] In the embodiment, the number of partition migration times can be used as the label content, and whether the target label content and the first update label content meet the update condition is determined by comparing the number of partition migration times, so as to determine whether to update the target label content in the file system. By using simple time identification, the target label content in the file system is prevented from being updated by multiple update worker nodes.

[0092] In addition, the label content can also be an identification character or word instead of a label value, which identifies the worker node currently serving the target partition. At this time, if the first worker node identified by the first update label content is different from the second worker node identified by the target label content in the file system, and the global sequence of the first worker node is later than that of the second worker node, it is determined that the update condition is met; otherwise, the update condition is not met.

[0093] For example, A is the current worker node, and the label content epoch of the access label stored locally by the worker node A indicates the worker node A. When the worker node A abnormally operates, the master node lets the worker node B take over, and issues epoch indicating the worker node B to the worker node B. The worker node B obtains the target label content epoch from the file system, which indicates the worker node A. Since the worker node B is different from the worker node A, and the global sequence of the worker node B is later than that of the worker node A, the worker node B can update the target label content epoch in the file system to indicate the worker node B.

[0094] In addition, assuming that the service process of the worker node B does not reply to the master node for a long time due to some problem, the master node considers that the worker node B cannot take over, lets the worker node C take over, and issues an epoch to the worker node, indicating the worker node C. The worker node C modifies the target label content in the file system to indicate the worker node C. At this time, the service process of the worker node B recovers and is ready to take over the target partition. The worker node B receives the epoch indicating the worker node B, but the current epoch in the file system indicates the worker node C. Although the worker node B is different from the worker node C, the global sequence of the worker node B is earlier than that of the worker node C, which indicates that another worker node has successfully taken over the target partition. The worker node B determines that the received label content is not the latest label content issued by the master node and does not meet the update condition. The worker node B gives up taking over the target partition and ends the update.

[0095] In the embodiment, in addition to using the number of partition migration times as the label content, a word or character can also be directly used as the label content to identify the update worker node of the current service partition. Whether the update condition is met is determined by comparing the identified worker nodes, so as to determine whether the target label content in the file system is updated. The setting of the access label is flexible and can adapt to various application scenarios.

[0096] In an optional implementation of the embodiment, after the target label content in the file system is updated according to the first update label content, the method further includes:

[0097] In the case of successful update of the target label, an update success message is returned to the master node. The update success message is used for the master node to determine whether the target worker node successfully takes over the target partition.

[0098] It should be noted that the target worker node initiating the access request is the update worker node. After the target label content in the file system is updated according to the first update label content, if the update is successful, an update success message can also be returned to the master node to notify the master node that the target partition is successfully taken over by the target worker node. Subsequently, the target worker node provides services for the target partition, avoids errors of the target access label caused by multiple worker nodes updating the target label content in the file system, and further avoids service exceptions of the distributed system.

[0099] In actual implementation, if the master node receives the update success message returned by the target worker node, it indicates that the target worker node successfully takes over the target partition, and the master node does not need to assign other update worker nodes. If the master node does not receive the update success message returned by the target worker node within a set time, it can determine that the target worker node cannot take over the target partition, and based on the global order of each worker node, determine the next update worker node, issue the second update label content to the update worker node, notify the update worker node to take over the target partition and provide corresponding services. The update worker node receives the second update label content, updates the target label content in the file system based on the second update label content, to take over the services of the target partition. The specific update process is similar to the specific update process of updating the target label content in the file system based on the first update label content, which will not be described herein again.

[0100] Step 204: generating a conditional access request of the target partition according to the to-be-verified access label, and accessing the target partition based on the conditional access request, wherein the to-be-verified access label is used to match the target access label, the matching result is used to determine whether the target worker node has the access right of the target partition, the target access label is the access label currently stored in the file system, and the conditional access request is used to indicate that access is allowed in the case that the target worker node has the access right of the target partition.

[0101] It should be noted that the conditional access request generated by the target worker node that wants to access the target partition carries the to-be-verified access label. The target worker node sends the conditional access request to the access interface of the file system. The file system verifies the to-be-verified access label in the conditional access request. If the to-be-verified access label carried in the conditional access request matches the target access label currently stored in the file system, it indicates that the to-be-verified access label carried by the target worker node initiating the access request is the updated access label, and the target worker node has the access right of the target partition and is allowed to access. At this time, the file system will open the data in the corresponding target partition based on the conditional access request to realize access. If the to-be-verified access label carried in the conditional access request does not match the target access label currently stored in the file system, it indicates that the to-be-verified access label carried by the target worker node initiating the access request is the old access label, and the target worker node does not have the access right of the target partition. The file system prohibits access and does not perform the operation of opening the data of the target partition.

[0102] In actual implementation, the data access interface of the file system can be configured as a conditional interface, that is, a conditional parameter is added to the data access interface of the file system to make it a conditional interface. When the constraint condition is met, the data of the corresponding partition is allowed to be opened for access.

[0103] The condition parameter condition of the conditional interface can be defined as follows:

[0104] Struct Condition{

[0105] String path;

[0106] String xAttrKey;

[0107] String xAttrValue;

[0108] }

[0109] Condition.path indicates that the specified file system object (for example, a file, a directory, etc.) exists, that is, the root directory of the target partition to be accessed exists; Condition.xAttrKey indicates that the specified access tag exists, for example, the title of the extended attribute attached to the root directory of the target partition, for example, in the scenario of the partition, the title of the access tag can be "epoch", which can be understood as a label with the title of epoch being attached to the root directory of the partition; the value of Condition.xAttrKey is equal to Condition.xAttrValue, and Condition.xAttrValue indicates the specific label content of the label "Condition.xAttrKey".

[0110] Specifically, the conditional access request can be implemented by using an open function, which is a function for opening and creating a file. For example, the generated conditional access request is open(path, mode, condition), which indicates that when the condition parameter condition is met, a file on the path path of the mode node is opened to implement access.

[0111] In an optional embodiment of the present embodiment, the conditional access request of the target partition is generated according to the to-be-verified access tag, and includes:

[0112] The conditional access request of the target partition is generated according to the to-be-verified label content, wherein the to-be-verified label content in the conditional access request is used to match the target label content, the target label content is the label content included in the target access tag, and whether the target worker node has the access right of the target partition is determined based on a matching result.

[0113] In actual implementation, as indicated in the foregoing, when the partition migration occurs, the update worker node can receive the update label content issued by the master node, and the update worker node updates the target label content of the target access label of the target partition in the file system based on the update label content. The update worker node is a worker node that currently has access authority of the target partition.

[0114] The target worker node can be the update worker node or the old worker node, that is, the target worker node in the embodiments of the present specification is any worker node that wants to access the target partition in the distributed system. The target worker node can be a worker node that currently has access authority of the target partition or can not be a worker node that currently has access authority of the target partition. When the target worker node initiates an access request to the file system, the file system determines whether the target worker node has access authority based on the to-be-verified access label carried by the target worker node, so as to determine whether the target worker node is allowed to access the target partition.

[0115] It should be noted that the target label content in the file system is the label content updated by the update worker node. If the to-be-verified label content carried by the target worker node is consistent with the target label content in the file system, it indicates that the target worker node that wants to access is the update worker node, that is, the target worker node is a worker node that currently has access authority of the target partition. Since the old worker node before the migration cannot obtain the update label content issued by the master node, the to-be-verified access label obtained by the old worker node from the local is the old label content. Therefore, if the to-be-verified label content carried by the target worker node is inconsistent with the target label content in the file system, it indicates that the target worker node that wants to access is not the update worker node, that is, the target worker node is not a worker node that currently has access authority of the target partition. That is, the target worker node can be determined to have access authority of the target partition and be allowed to access the target partition when the to-be-verified label content is consistent with the target label content, and the target worker node can be determined not to have access authority of the target partition and be prohibited from accessing the target partition when the to-be-verified label content is inconsistent with the target label content.

[0116] In the embodiments of the present specification, the target work node can carry the obtained to-be-verified label content when generating the conditional access request, so that when the file system receives the conditional access request, it can determine whether the carried to-be-verified label content is the current stored target label content, thereby determining whether the target work node has access permission of the target partition, and allowing access in the case that the target work node has access permission of the target partition. Since the old work node before migration cannot obtain the updated label content issued by the master node, if the target work node is an old work node, the to-be-verified label content carried in the conditional access request initiated by the target work node is the old label content, which does not match the target label content in the file system. The target work node does not have access permission of the target partition and cannot access, and cannot perform the operation of opening the data of the target partition, thereby blocking the access operation of the old work node on the data of the target partition.

[0117] In an optional embodiment of the present embodiment, the conditional access request includes to-be-verified label content; and the conditional access request is used to access the target partition, and the method comprises the following steps of:

[0118] sending the conditional access request to a metadata access interface of the file system, wherein the metadata access interface is used to access the target partition, and the metadata access interface allows updating of file system metadata of the target partition in the case that the to-be-verified label content is consistent with the target label content, and prohibits updating of the file system metadata of the target partition in the case that the to-be-verified label content is inconsistent with the target label content, and the target label content is label content included in the target access label.

[0119] It should be noted that the data of multiple partitions in the distributed system is stored on the file system, and the data of the file system includes two parts of metadata and data. If the old work node is to be blocked from accessing the data of the target partition, a key part is to block it from updating the file system metadata.

[0120] In actual implementation, after the target work node generates the conditional access request, the target work node can send the conditional access request to a metadata access interface of the file system. The metadata access interface is a conditional interface. In the case that the to-be-verified label content is consistent with the target label content, it means that the set condition of the conditional interface is met, and the target partition is allowed to be accessed, that is, the target work node is allowed to update the file system metadata of the target partition. In the case that the to-be-verified label content is inconsistent with the target label content, the target partition is prohibited to be accessed, that is, the target work node is prohibited to update the file system metadata of the target partition, thereby blocking the old work node from updating the file system metadata of the target partition, realizing blocking the old work node from accessing the data of the target partition, and ensuring the data consistency of the distributed system.

[0121] In an optional embodiment of the present embodiment, the method further comprises:

[0122] In a case where a read instruction for the target partition is received, an access directory of the target partition is acquired;

[0123] A read request of the target partition is generated according to the access directory, and data in the target partition is read based on the read request.

[0124] It should be noted that the access instruction of the target node to the target partition is a write instruction, since the write instruction will change the data in the partition, if the working nodes before and after migration can update the data in the partition, the data consistency will be damaged, therefore, for the write instruction, the access tag of the target partition can be acquired, a conditional access request is generated based on the access tag, in a case where it is determined through verification that the target working node has access permission of the target partition, the data is allowed to be updated. In addition, for the case where a read instruction is received, since the read instruction is to read the data in the partition, the data will not be updated, and multiple nodes reading at the same time will not damage the data consistency, therefore, the read instruction can not be conditionally verified, for the read instruction, the access directory of the target partition, i.e., the root directory of the target partition, can be acquired, a read request of the target partition is generated based on the root directory, and the data in the target partition is directly read.

[0125] In an embodiment of the present specification, the access permission of the target partition is indicated by configuring the access tag on the target partition, the target working node initiating the write instruction is verified by using the access tag, and the update operation of the partition data by the old abnormal node is blocked. In the above process, the root directory of the partition does not change, when reading data, the data reading can still be realized based on the root directory of the target partition, after the partition migration, the read node does not need to query the master node or the file system again to determine the root directory of the partition; in addition, since the root directory of the partition is unchanged, the root directory of the partition can be quickly located to realize the operation and maintenance of the distributed system, and the implementation and operation and maintenance of the distributed data system are simplified.

[0126] Figure 3 A process flow diagram of a data processing method in a first distributed system provided by an embodiment of the present specification is shown, as shown in Figure 3 The specific steps include the following steps.

[0127] It should be noted that the distributed system includes three working nodes, i.e., working node A, working node B, and working node C, and the global order of the three working nodes is working node A, working node B, and working node C. The distributed system further includes a master node and a file system, and the file system has multiple partitions, and one partition corresponds to one data subset.

[0128] Initially, the working node A is assigned to provide services for the partition P, the access tag of the partition P in the working node A is epoch=1, and the current access tag of the partition P in the file system is epoch=1.

[0129] Step 302: The master node monitors that the worker node A is abnormal, determines the updated worker node as the worker node B according to the global sequence, and issues the access label epoch=2 of the partition P to the worker node B.

[0130] Wherein, the worker node A is the current worker node of the partition P, the master node detects that the current worker node is abnormal, migrates the partition P to the worker node B, and the worker node B is the updated worker node after migration. The partition P is the target partition to be accessed.

[0131] Step 304: The worker node B queries the target access label as epoch=1 from the file system, and since epoch=1 is less than epoch=2, the worker node B updates the target access label in the file system as epoch=2.

[0132] It should be noted that epoch=1 is less than epoch=2, which indicates the number of partition migration of the partition P currently stored in the file system, which is not the number of new partition migration issued by the master node. At this time, epoch=2 can be used to replace epoch=1 in the file system, so that the epoch in the file system is the updated number of partition migration.

[0133] Step 306: In the case that the worker node B receives an access instruction for the partition P, the worker node B obtains the to-be-verified access label epoch=2 locally, generates a conditional access request based on the to-be-verified access label, and sends the conditional access request to the metadata interface of the file system.

[0134] Wherein, the worker node B is the target worker node to access the partition P, obtains the to-be-verified access label epoch=2 locally, generates a conditional access request, and accesses the file system.

[0135] Step 308: The metadata interface of the file system judges that epoch=2 in the conditional access request and the target access label in the file system are both epoch=2, which are consistent, and determines that the worker node B has the access permission of the partition P, and allows the worker node B to update the file system metadata of the partition P.

[0136] Wherein, epoch=2 carried in the conditional access request initiated by the worker node B matches the currently stored epoch=2 in the file system, which indicates that the to-be-verified access label carried by the worker node B initiating the access request is the updated access label, and the worker node B initiating the access request is the worker node newly taking over the partition P, which has the access permission of the partition P and is allowed to access the partition P.

[0137] It should be noted that steps 302-308 are the case that the updated worker node accesses the partition P after the migration of the partition P, and the specific implementation process is the same as the above Figure 2The embodiments shown are similar, and thus the description of the embodiments of the present specification will not be repeated here.

[0138] In the embodiments of the present specification, if the current working node of the target partition is abnormal, the master node migrates the target partition, and the update working node taking over the target partition can update the target access label in the file system. For the update working node, the to-be-verified access label carried in the conditional access request is consistent with the target access label currently stored in the file system, and the update working node has access permission. Therefore, the update working node can successfully access the target partition and update the data in the target partition.

[0139] Figure 4 A flow chart of a processing process of a data processing method in a second distributed system provided by an embodiment of the present specification is shown in FIG. 4. Figure 4 As shown, the method comprises the following steps.

[0140] It should be noted that the distributed system includes three working nodes, namely working node A, working node B, and working node C, and the global order of the three working nodes is working node A, working node B, and working node C. The distributed system further includes a master node and a file system, and the file system includes multiple partitions, one partition corresponding to one data subset.

[0141] Initially, working node A is assigned to provide services for partition P, the access label of partition P in working node A is epoch=1, and the current access label of partition P in the file system is epoch=1.

[0142] Step 402: The master node monitors that working node A is abnormal, determines that the update working node is working node B according to the global order, and issues the access label epoch=2 of partition P to working node B.

[0143] Among them, working node A is the current working node of partition P, the master node detects that the current working node is abnormal, migrates partition P to working node B, and working node B is the update working node after migration. Partition P is the target partition to be accessed.

[0144] Step 404: Working node B queries the target access label from the file system, which is epoch=1. Since epoch=1 is less than epoch=2, working node B updates the target access label in the file system to epoch=2.

[0145] It should be noted that epoch=1 is less than epoch=2, which indicates the number of partition migrations currently stored in the file system of partition P, and is not the number of new partition migrations issued by the master node. At this time, epoch=2 can be used to replace epoch=1 in the file system, so that the epoch in the file system is the number of updated partition migrations.

[0146] Step 406: In the case where the worker node A receives an access instruction for the partition P, the worker node A locally acquires a to-be-verified access tag epoch = 1, generates a conditional access request based on the to-be-verified access tag, and sends the conditional access request to a metadata interface of the file system.

[0147] In the case where the worker node A receives an access instruction for the partition P, the worker node A locally acquires a to-be-verified access tag epoch = 1, generates a conditional access request based on the to-be-verified access tag, and sends the conditional access request to a metadata interface of the file system.

[0148] Step 408: The metadata interface of the file system determines that the conditional access request includes epoch = 1, while the target access tag in the file system is epoch = 2, and the two are inconsistent, and it is determined that the worker node A does not have the access right of the partition P and is not allowed to update the file system metadata of the partition P.

[0149] In the case where the worker node A receives an access instruction for the partition P, the worker node A locally acquires a to-be-verified access tag epoch = 1, generates a conditional access request based on the to-be-verified access tag, and sends the conditional access request to a metadata interface of the file system.

[0150] It should be noted that steps 402-408 are a case where a worker node (i.e., an abnormal worker node) before the migration of the partition P accesses the partition P, and the implementation process is similar to the above Figure 2 The implementation process of the embodiment of the present specification is similar to the above

[0151] In the embodiment of the present specification, if the current worker node of the target partition is abnormal, the master node will migrate the target partition, and the update worker node that takes over the target partition can update the current access tag in the file system. For the worker node before the migration, the conditional access request initiated by the worker node carries an old access tag, while the tag in the file system has been updated by the update worker node, that is, for the worker node before the migration, the to-be-verified access tag carried in the conditional access request is inconsistent with the target access tag in the file system, and the worker node before the migration does not have the access right. Therefore, the worker node before the migration is prohibited from accessing the target partition, and by configuring the to-be-verified access tag on the target partition, the operation of the old abnormal node on the partition data can be blocked, the consistency of the partition data is ensured, the root directory of the partition does not need to be repeatedly changed, the implementation and operation of the distributed system are simplified, and the distributed system is facilitated to implement a storage-computing separation architecture and serverless.

[0152] Figure 5A process flow diagram of a data processing method in a third distributed system provided by an embodiment of the present specification is shown in FIG. 6, which specifically includes the following steps. Figure 5

[0153] It should be noted that the distributed system includes three worker nodes, namely worker node A, worker node B, and worker node C, and the global order of the three worker nodes is worker node A, worker node B, and worker node C. The distributed system also includes a master node and a file system, and the file system has multiple partitions, one partition corresponding to one data subset.

[0154] Initially, worker node A is assigned to provide services for partition P, the access tag of partition P in worker node A is epoch = 1, and the current access tag of partition P in the file system is epoch = 1.

[0155] Step 502: The master node monitors that worker node A has an exception, determines the update worker node as worker node B according to the global order, and issues the access tag epoch = 2 of partition P to worker node B.

[0156] Among them, worker node A is the current worker node of partition P, the master node detects that the current worker node has an exception, migrates partition P to worker node B, and worker node B is the updated worker node after migration. Partition P is the target partition to be accessed.

[0157] Step 504: The master node does not receive the update success message fed back by worker node B within a preset time length, determines the next update worker node as worker node C according to the global order, and issues the access tag epoch = 3 of partition P to worker node C.

[0158] Among them, the master node does not receive the update success message fed back by worker node B within a preset time length, which means that worker node B fails to take over partition P in time, at which time the master node will reselect the next update worker node, i.e., worker node C, and issue the access tag epoch = 3 again.

[0159] Step 506: Worker node C queries the target access tag as epoch = 1 from the file system, and since epoch = 1 is less than epoch = 3, worker node C updates the target access tag in the file system to epoch = 3.

[0160] ​The working node C queries the current access tag from the file system, which is epoch=1, and is less than the epoch=3 received by itself, indicating that the partition migration times of the partition P currently stored in the file system are not the new partition migration times issued by the master node. At this time, the epoch=3 can be used to replace the epoch=1 in the file system, so that the epoch in the file system is updated to the new partition migration times. The working node C successfully takes over the partition P.

[0161] Step 508: The working node B service process recovers, and queries the target access tag from the file system, which is epoch=3. Since the epoch=2 received by the working node B is less than the epoch=3, the working node B ends the update.

[0162] After the working node B process recovers, it is ready to take over the partition P. The working node B queries the current access tag from the file system, which is epoch=3. The working node B compares the epoch=2 received by itself with the current epoch=3 in the file system, and determines that the tag content received by itself is not the latest tag content issued by the master node, and does not meet the update condition. The working node B gives up taking over the partition P. The working node B fails to take over the partition P.

[0163] Step 510: When the working node B receives an access instruction for the partition P, the working node B obtains the to-be-verified access tag epoch=2 from the local, generates a conditional access request based on the to-be-verified access tag, and sends the conditional access request to the metadata interface of the file system.

[0164] The working node B obtains the to-be-verified access tag epoch=2 from the local, generates a conditional access request, and accesses the file system.

[0165] Step 512: The metadata interface of the file system judges that the epoch=2 in the conditional access request is inconsistent with the target access tag epoch=3 in the file system, and determines that the working node B does not have the access right of the partition P and is not allowed to update the file system metadata of the partition P.

[0166] The epoch=2 carried in the conditional access request initiated by the working node B does not match the epoch=3 currently stored in the file system, indicating that the to-be-verified access tag carried by the working node B initiating the access request is an old access tag. The working node B initiating the access request is not the working node taking over the partition P, and does not have the access right of the partition P and is not allowed to access the partition P.

[0167] Step 514: In the case that the worker node C receives an access instruction for the partition P, the worker node C locally acquires the to-be-verified access tag epoch=3, generates a conditional access request based on the to-be-verified access tag, and sends the conditional access request to a metadata interface of the file system.

[0168] In the case that the worker node C receives an access instruction for the partition P, the worker node C locally acquires the to-be-verified access tag epoch=3, generates a conditional access request based on the to-be-verified access tag, and sends the conditional access request to a metadata interface of the file system.

[0169] Step 516: The metadata interface of the file system determines that the epoch=3 in the conditional access request is consistent with the target access tag epoch=3 in the file system, and determines that the worker node C has the access permission of the partition P, and allows the worker node C to update the file system metadata of the partition P.

[0170] In the case that the worker node C receives an access instruction for the partition P, the worker node C locally acquires the to-be-verified access tag epoch=3, generates a conditional access request based on the to-be-verified access tag, and sends the conditional access request to a metadata interface of the file system.

[0171] It should be noted that steps 502-516 are the process of the worker node successfully taking over the partition P and the worker node unsuccessfully taking over the partition P accessing the partition P in the case that the partition P is migrated, and the specific implementation process is similar to the above Figure 2 embodiment, and details are not described herein again.

[0172] In the embodiment of the present specification, if the update worker node determines that the update label content received by the update worker node is not the latest label content issued by the master node, it is determined that another worker node has successfully taken over the target partition, at this time, the update worker node gives up taking over the target partition and ends the update, avoiding multiple update worker nodes updating the target label content in the file system, realizing the migration of the partition, and ensuring the normal service of the distributed system.

[0173] Corresponding to the method embodiment, the present specification also provides a target worker node in a distributed system, Figure 6 Fig. 6 shows a structure diagram of a target worker node in a distributed system according to an embodiment of the present specification. As shown in Figure 6 The target worker node comprises:

[0174] The acquisition module 602 is configured to acquire a to-be-verified access tag of a target partition in the case that an access instruction for the target partition is received, wherein the target partition is any partition of a file system in a distributed system for storing data.

[0175] The access module 604 is configured to generate a conditional access request of the target partition according to a to-be-verified access tag, access the target partition based on the conditional access request, determine whether the target work node has the access right of the target partition based on a matching result of the to-be-verified access tag and a target access tag, and indicate that the access is allowed in a case where the target work node has the access right of the target partition, wherein the target access tag is an access tag currently stored in the file system.

[0176] Optionally, the to-be-verified access tag includes to-be-verified tag content, and the access module 604 is further configured to:

[0177] read the to-be-verified tag content of the target partition from the local;

[0178] Correspondingly, the access module 604 is further configured to:

[0179] generate the conditional access request of the target partition according to the to-be-verified tag content, wherein the to-be-verified tag content is used for matching with target tag content, and the target tag content is tag content included in the target access tag, and the target work node is determined to have the access right of the target partition based on a matching result.

[0180] Optionally, the target access tag includes target tag content, and the distributed system further includes a master node, and the work node in the distributed system further includes an update module configured to:

[0181] receive first update tag content issued by the master node, wherein the first update tag content is sent by the master node in a case where a current work node of the target partition is abnormal;

[0182] query the target tag content of the target access tag from the file system;

[0183] update the target tag content in the file system according to the first update tag content.

[0184] Optionally, the update module is further configured to:

[0185] replace the target tag content in the file system with the first update tag content in a case where the first update tag content and the target tag content meet an update condition;

[0186] end the update in a case where the first update tag content and the target tag content do not meet the update condition.

[0187] Optionally, the first updated label content is a number of partition migrations issued by the master node, and the target label content is a number of partition migrations indicated by the target access label currently stored in the file system; the updating module is further configured to:

[0188] In a case where the target label content is less than the first updated label content, it is determined that the first updated label content and the target label content meet the updating condition.

[0189] In a case where the target label content is not less than the first updated label content, it is determined that the first updated label content and the target label content do not meet the updating condition.

[0190] Optionally, the updating module is further configured to:

[0191] In a case where the updating of the target label content is successful, an updating success message is returned to the master node, where the updating success message is used for the master node to determine whether the target work node successfully takes over the target partition.

[0192] Optionally, the work node in the distributed system further includes a reading module configured to:

[0193] In a case where a read instruction for the target partition is received, an access directory of the target partition is obtained;

[0194] A read request of the target partition is generated according to the access directory, and data in the target partition is read based on the read request.

[0195] Optionally, the target label content is included in the conditional access request; and the access module 604 is further configured to:

[0196] The conditional access request is sent to a metadata access interface of the file system, where the metadata access interface is used to access the target partition, the metadata access interface allows updating of file system metadata of the target partition in a case where the to-be-verified label content is consistent with the target label content, and the metadata access interface prohibits updating of the file system metadata of the target partition in a case where the to-be-verified label content is inconsistent with the target label content, and the target label content is label content included in the target access label.

[0197] Optionally, the to-be-verified access label is an extended attribute label set on a root directory of the target partition; and / or the to-be-verified access label is a flag file or a flag directory created for the target partition.

[0198] In an embodiment of the present specification, a target worker node in a distributed system comprises an obtaining module and an access module. The obtaining module can obtain a to-be-verified access label of a target partition. The access module can generate a conditional access request carrying the to-be-verified access label, so that the file system can determine whether the target worker node has access permission of the target partition based on the to-be-verified access label in the conditional access request and a target access label currently stored in the file system, and access is allowed in the case that the target worker node has access permission of the target partition. The interaction of the obtaining module and the access module makes the worker node with access permission of the target partition access the target partition in the file system, blocks the operation of the old abnormal node on the partition data, guarantees the consistency of the partition data, does not need to repeatedly change the root directory of the partition, simplifies the implementation and operation and maintenance of the distributed system, and facilitates the distributed system to implement the storage-computing separation architecture and serverless.

[0199] The above is a schematic scheme of a target worker node in a distributed system according to an embodiment of the present specification. It should be noted that the technical scheme of the target worker node in the distributed system and the technical scheme of the above-mentioned data processing method belong to the same concept, and the details of the technical scheme of the target worker node in the distributed system that are not described in detail can be referred to the description of the technical scheme of the data processing method.

[0200] Corresponding to the above method embodiment, the present specification also provides a distributed system embodiment, Figure 7 A structure schematic diagram of a distributed system provided by an embodiment of the present specification is shown. As shown in the figure, Figure 7 The distributed system comprises a file system 702 and at least two worker nodes 704;

[0201] The target worker node 7042 is configured to, in the case of receiving an access instruction for a target partition, obtain a to-be-verified access label of the target partition, wherein the target worker node 7042 is any one of the at least two worker nodes 704, and the target partition is any partition in the distributed system for storing data by the file system 702; generate a conditional access request of the target partition according to the to-be-verified access label, and access the file system 702 based on the conditional access request.

[0202] The file system 702 is configured to receive the conditional access request, match the to-be-verified access label with a target access label, determine whether the target worker node 7042 has access permission of the target partition based on the matching result, and allow access to the target partition in the case that the target worker node 7042 has access permission of the target partition, wherein the target access label is a target access label currently stored in the file system.

[0203] In actual implementation, the distributed system includes at least two worker nodes and a file system, the file system is a distributed file system, and the distributed file system is configured to provide storage services for data corresponding to a plurality of partitions, to realize storage and computing separation and serverless, one worker node serves one corresponding partition, and the worker node provides storage and access to data in the partition. The target worker node is any worker node in the distributed system that needs to access the target partition.

[0204] In an optional embodiment of the present embodiment, the access tag includes tag content. Figure 7 As shown in the figure, the distributed system further includes a master node 706.

[0205] The master node 706 is configured to, in a case where it is detected that the current worker node of the target partition is abnormal, determine a first update worker node from the remaining worker nodes except the current worker node among the at least two worker nodes, and issue a first update tag content of the target partition to the first update worker node.

[0206] The first update worker node is configured to query a target tag content of the target access tag from the file system, and update the target tag content in the file system according to the first update tag content.

[0207] In actual application, if the current worker node of the target partition is abnormal and cannot provide services of the target partition, the master node selects one of the other normal worker nodes of the distributed system as the first update worker node, and then issues the first update tag content to the first update worker node. The first update worker node can update the target tag content in the file system by using the first update tag content, so as to update the target access tag in the file system to the new tag content issued by the master node. Subsequently, the target access tag content in the file system and the tag content carried in the conditional access request can be used to verify the worker node that initiates the conditional access request, to determine whether the worker node has access rights, so as to ensure data consistency of the distributed system.

[0208] In an optional embodiment of the present embodiment, the first update worker node is configured to:

[0209] In a case where the target tag content is updated successfully, an update success message is returned to the master node, wherein the update success message is used by the master node to determine whether the first update worker node successfully takes over the target partition.

[0210] The master node 706 is further configured to:

[0211] In a case where the first update worker node does not return an update success message within a preset time period, a second update worker node is determined according to the global working order of the at least two worker nodes, and the second update worker node is instructed to issue the second update label content of the target partition.

[0212] It should be noted that after the first update worker node updates the target label content in the file system, if the update is successful, an update success message can be returned to the master node to notify the master node that it successfully takes over the target partition, and the first update worker node provides services for the target partition subsequently, so as to avoid errors in access to the label-identified partition migration information caused by multiple worker nodes updating the target label content in the file system, and further avoid service abnormalities of the distributed system.

[0213] If the master node does not receive the update success message returned by the first update worker node within a set time, the master node can determine that the first update worker node cannot take over the target partition, and at this time, a second update worker node can be determined based on the global order of the worker nodes, and the second update worker node is instructed to issue the second update label content, so as to notify the second update worker node to take over the target partition and provide corresponding services. The second update worker node receives the second update label content, and can update the target label content in the file system based on the second update label content to take over the services of the target partition, so as to ensure that the partition migration can be successfully implemented.

[0214] In an optional embodiment of the present embodiment, the conditional access request includes to-be-verified label content;

[0215] The target worker node is configured to send a conditional access request to a metadata access interface of the file system;

[0216] The metadata access interface of the file system is configured to allow updating of file system metadata of the target partition in a case where the to-be-verified label content is consistent with the target label content, and to prohibit updating of the file system metadata of the target partition in a case where the to-be-verified label content is inconsistent with the target label content, the target label content being label content included in the target access label.

[0217] In the embodiments of the present specification, after generating the conditional access request, the target worker node can send the conditional access request to a metadata access interface of the file system. The metadata access interface is a conditional interface. When the to-be-verified label content is consistent with the target label content, it indicates that the set condition of the conditional interface is met, and access to the target partition is allowed, that is, the target worker node is allowed to update the file system metadata of the target partition. When the to-be-verified label content is inconsistent with the target label content, access to the target partition is prohibited, that is, the target worker node is prohibited from updating the file system metadata of the target partition, thereby blocking the old worker node from updating the file system metadata of the target partition, achieving blocking of the old worker node from accessing the data of the target partition, and ensuring the data consistency of the distributed system.

[0218] The above is a schematic scheme of a distributed system according to an embodiment of the present specification. It should be noted that the technical scheme of the distributed system and the technical scheme of the above data processing method belong to the same concept, and the details of the technical scheme of the distributed system that are not described in detail can be referred to the description of the technical scheme of the data processing method.

[0219] Figure 8 A structural block diagram of a computing device according to an embodiment of the present specification is shown. The components of the computing device 800 include but are not limited to a memory 810 and a processor 820. The processor 820 is connected to the memory 810 through a bus 830, and a database 850 is used to save data.

[0220] The computing device 800 also includes an access device 840 that enables the computing device 800 to communicate via one or more networks 860. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or combinations of such networks, such as the Internet. The access device 840 can include one or more of any type of network interface (for example, a network interface card (NIC)), such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC).

[0221] In one embodiment of the present specification, the above-mentioned components of the computing device 800 and other components not shown in the Figure 8 may be connected to each other, such as through a bus. It should be understood that Figure 8 The computing device structure diagram shown is merely for the purpose of example, and is not a limitation on the scope of the present specification. Other components can be added or replaced by those skilled in the art as needed.

[0222] The computing device 800 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (for example, a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, and the like), a mobile phone (for example, a smartphone), a wearable computing device (for example, a smart watch, smart glasses, and the like), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 800 can also be a mobile or stationary server.

[0223] The processor 820 is configured to execute computer program / instructions that implement the steps of the above-mentioned data processing method when executed by the processor.

[0224] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts among the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, the computing device embodiment is described simply because it is basically similar to the data processing method embodiment, and the relevant part can be referred to the description of the data processing method embodiment.

[0225] An embodiment of the specification further provides a computer readable storage medium storing computer programs / instructions, which are executed by a processor to implement the steps of the above data processing method.

[0226] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts among the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, the computer readable storage medium embodiment is described simply because it is basically similar to the data processing method embodiment, and the relevant part can be referred to the description of the data processing method embodiment.

[0227] An embodiment of the specification further provides a computer program product including computer programs / instructions, which are executed by a processor to implement the steps of the above data processing method.

[0228] The above is a schematic scheme of the computer program product of the embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above data processing method belong to the same concept, and the details of the technical scheme of the computer program product which are not described in detail can be referred to the description of the technical scheme of the data processing method.

[0229] The above describes specific embodiments of the specification. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps in a claim can be performed in an order different than the order in which the acts or steps are recited and still accomplish the desired results. Also, the process depicted in the accompanying figures does not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0230] The computer readable medium can include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a Read-Only Memory (ROM), a Random Access Memory (RAM), an electrical carrier signal, a telecommunications signal, and a software distribution package, etc. It should be noted that the computer readable medium can include appropriate contents according to the requirements of patent practice, for example, according to the patent practice in some regions, the computer readable medium does not include an electrical carrier signal and a telecommunications signal.

[0231] It should be noted that, for the foregoing method embodiments, in order to facilitate description, each is described as a combination of a series of acts, but those skilled in the art should know that the present specification embodiments are not limited to the order of the acts described, because according to the present specification embodiments, certain steps can be performed in other orders or at the same time. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the acts and modules involved are not necessarily essential to the present specification embodiments.

[0232] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0233] The preferred embodiments of the present specification disclosed above are only used to help explain the present specification. The alternative embodiments do not describe all the details and limit the present invention to the specific embodiments described. Obviously, according to the content of the present specification embodiments, many modifications and changes can be made. The present specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present specification embodiments, so that those skilled in the art can well understand and use the present specification. The present specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method applied to a target work node in a distributed system, the method comprising: in a case where an access instruction for a target partition is received, obtaining a to-be-verified access tag of the target partition, wherein the target partition is any partition in the distributed system used by a file system to store data; generating a conditional access request of the target partition according to the to-be-verified access tag, and accessing the target partition based on the conditional access request, wherein the to-be-verified access tag is used to match a target access tag, a matching result is used to determine whether the target work node has access permission of the target partition, the target access tag is a currently stored access tag in the file system, and the conditional access request is used to indicate that the target partition is allowed to be accessed in a case where the target work node has the access permission of the target partition.

2. The data processing method of claim 1, wherein the to-be-verified access tag comprises to-be-verified tag content; and the obtaining the to-be-verified access tag of the target partition comprises: locally reading the to-be-verified tag content of the target partition; and the generating the conditional access request of the target partition according to the to-be-verified access tag comprises: generating the conditional access request of the target partition according to the to-be-verified tag content, wherein the to-be-verified tag content is used to match target tag content, a matching result is used to determine whether the target work node has the access permission of the target partition, and the target tag content is tag content included in the target access tag.

3. The data processing method of claim 1, wherein the target access tag comprises target tag content; the distributed system further comprises a master node; and the method further comprises: receiving a first updated tag content issued by the master node, wherein the first updated tag content is sent by the master node in a case where a current work node of the target partition is abnormal; querying the target tag content of the target access tag from the file system; and updating the target tag content in the file system according to the first updated tag content.

4. The data processing method of claim 3, wherein the updating the target tag content in the file system according to the first updated tag content comprises: in a case where the first updated tag content and the target tag content meet an update condition, replacing the target tag content in the file system with the first updated tag content; and in a case where the first updated tag content and the target tag content do not meet the update condition, prohibiting updating.

5. The data processing method of claim 4, wherein the first updated tag content is a partition migration number issued by the master node, the target tag content is a partition migration number indicated by a target access tag currently stored in the file system; and before the replacing the target tag content in the file system with the first updated tag content in the case where the first updated tag content and the target tag content meet the update condition, the method further comprises: determining that the first update label content and the target label content meet the update condition in a case that the target label content is less than the first update label content; determining that the first update label content and the target label content do not meet the update condition in a case that the target label content is not less than the first update label content.

6. The data processing method of claim 3, after the target label content in the file system is updated according to the first update label content, further comprising: in a case that the target label content is successfully updated, returning an update success message to the master node, wherein the update success message is used by the master node to determine whether the target worker node successfully takes over the target partition.

7. The data processing method of any one of claims 1-6, further comprising: in a case that a read instruction for a target partition is received, obtaining an access directory of the target partition; generating a read request of the target partition according to the access directory, and reading data in the target partition based on the read request.

8. The data processing method of any one of claims 1-6, wherein the conditional access request comprises to-be-verified label content; and the accessing the target partition based on the conditional access request comprises: sending the conditional access request to a metadata access interface of a file system, wherein the metadata access interface is used to access the target partition, and the metadata access interface allows updating file system metadata of the target partition in a case that the to-be-verified label content is consistent with target label content, and prohibits updating the file system metadata of the target partition in a case that the to-be-verified label content is not consistent with the target label content, the target label content being label content included in a target access label.

9. The data processing method of any one of claims 1-6, wherein the to-be-verified access label is an extended attribute label set on a root directory of the target partition; and / or the to-be-verified access label is a flag file or a flag directory created for the target partition.

10. A target worker node in a distributed system, comprising: an obtaining module configured to, in a case that an access instruction for a target partition is received, obtain a to-be-verified access label of the target partition, wherein the target partition is any partition of a file system in the distributed system for storing data; an access module configured to generate a conditional access request of the target partition according to the to-be-verified access label, and access the target partition based on the conditional access request, wherein the to-be-verified access label is used to match a target access label, a matching result is used to determine whether the target worker node has access authority of the target partition, the target access label is an access label currently stored in the file system, and the conditional access request is used to indicate that access is allowed in a case that the target worker node has the access authority of the target partition.

11. A distributed system comprising a file system and at least two worker nodes. a target worker node configured to, in a case of receiving an access instruction for a target partition, acquire a to-be-verified access tag of the target partition, wherein the target worker node is any one of the at least two worker nodes, and the target partition is any partition of a file system in the distributed system for storing data; generate a conditional access request of the target partition according to the to-be-verified access tag, and access the file system based on the conditional access request; the file system is configured to receive the conditional access request, match the to-be-verified access tag with a target access tag, determine whether the target worker node has access permission of the target partition based on a matching result, and allow access to the target partition in a case that the target worker node has the access permission of the target partition, wherein the target access tag is a currently stored access tag in the file system.

12. The distributed system of claim 11, wherein the target access tag comprises target tag content; and the distributed system further comprises a master node. the master node is configured to, in a case of detecting that a current worker node of the target partition is abnormal, determine a first update worker node from remaining worker nodes except the current worker node among the at least two worker nodes, and issue a first update tag content of the target partition to the first update worker node; the first update worker node is configured to query the target access tag from the file system to obtain target tag content, and update the target tag content in the file system according to the first update tag content.

13. The distributed system of claim 12, wherein the first update worker node is configured to: In case of success of updating the target label content, a success message of updating is returned to the master node, wherein, the update success message is used by the master node to determine whether the first update worker node successfully takes over the target partition; the master node is configured to: in a case of not receiving the update success message returned by the first update worker node within a preset time period, determine a second update worker node according to a global working order of the at least two worker nodes, and issue a second update tag content of the target partition to the second update worker node.

14. The distributed system of claim 11, wherein the conditional access request comprises to-be-verified tag content. the target worker node is configured to send the conditional access request to a metadata access interface of the file system; the metadata access interface of the file system is configured to, in a case that the to-be-verified tag content is consistent with target tag content, allow updating file system metadata of the target partition, and in a case that the to-be-verified tag content is inconsistent with the target tag content, prohibit updating the file system metadata of the target partition, wherein the target tag content is tag content included in the target access tag.

15. A computing device, comprising: a memory and a processor; The memory is configured to store computer programs / instructions, and the processor is configured to execute the computer programs / instructions, so as to implement the steps of the data processing method according to any one of claims 1 to 9.

16. A computer readable storage medium, storing computer programs / instructions, wherein the computer programs / instructions, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 9.

17. A computer program product, comprising computer programs / instructions, wherein the computer programs / instructions, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 9.

Images