Data processing system and method

Abstract

The present specification provides a data processing system and method, wherein the system comprises: a management node and a service node; the management node is configured to set a state attribute for stored data; the service node is configured to obtain the data stored by the management node, generate a target event according to the state attribute of the data, and update the data in external services according to the target event. In the present specification, the data stored by the management node is set with a state attribute, so that after the service node obtains the data stored by the management node, the target event can be directly generated according to the state attribute of the data, and the data in external services is updated. The service node does not need to store the full amount of data of the management node locally, nor does it need to compare the data, thereby greatly saving the processing time and storage consumption.

Description

TECHNICAL FIELD

[0001] The present specification relates to the technical field of distributed database, and particularly relates to a data processing system. The present specification also relates to a data processing method, a computing device, and a computer readable storage medium. BACKGROUND

[0002] With the development of computer technology, in order to meet the increasing needs of users, distributed database emerges as the times require. In the distributed database scenario, the service node relies on the metadata information generated by the management node, such as permission, addressing information, etc., to update the external service, that is, the service node constantly obtains the metadata information from the management node, and applies the change of the metadata to the external service. However, in the case that the management node actively deletes part of the data, the service node cannot effectively perceive the deleted data.

[0003] In the prior art, after the service node first obtains the full data stored by the management node, the service node stores the full data locally. Subsequently, each time the service node obtains the full data stored by the management node, the service node can compare the obtained full data with the cached data stored locally to determine the deleted data, generate a deletion event of the data, and update the external service. However, in the above process, the service node needs to store the full data locally, which consumes memory. If the full data is too large, it needs to be stored in the disk, which affects the read-write performance and the read-write time is too long. In addition, comparing the full data with the cached data to determine whether the data is deleted consumes time. Therefore, a simpler and more convenient method is needed for data processing operation. SUMMARY

[0004] Therefore, the embodiments of the present specification provide a data processing system. The present specification also relates to a data processing method, a computing device, and a computer readable storage medium to solve the technical defects in the prior art.

[0005] According to a first aspect of the embodiments of the present specification, a data processing system is provided, comprising: a management node and a service node;

[0006] The management node is configured to set a state attribute for the stored data.

[0007] The service node is configured to obtain the data stored by the management node, generate a target event according to the state attribute of the data, and update the data in the external service according to the target event.

[0008] Optionally, the management node is further configured to:

[0009] set a deleted state for the to-be-deleted data;

[0010] setting a new state for other data in the stored data except the data to be deleted.

[0011] Optionally, the service node is further configured to:

[0012] traversing each data obtained, determining a state attribute of the data;

[0013] if the state attribute of the data is a deleted state, generating a deletion event according to the data;

[0014] if the state attribute of the data is a new state, generating a new event according to the data.

[0015] Optionally, the data in the external service is updated in an idempotent operation manner.

[0016] Optionally, the system further comprises a common database for common access of the management and control node and the service node, and the management and control node is further configured to:

[0017] store the data in the common database;

[0018] Correspondingly, the service node is further configured to:

[0019] obtain the data stored by the management and control node from the common database.

[0020] Optionally, the system further comprises a private database for access of the management and control node, and the management and control node is further configured to:

[0021] store the data in the private database;

[0022] Correspondingly, the service node is further configured to:

[0023] obtain the data stored by the management and control node in the private database through a full data query operation provided by the management and control node externally.

[0024] Optionally, the management and control node is further configured to:

[0025] after a preset time, deleting the data with the state attribute of the deleted state.

[0026] According to a second aspect of the embodiments of the present specification, a data processing method is provided, applied to a management and control node in a data processing system, and the method comprises:

[0027] storing data;

[0028] setting a state attribute for the stored data.

[0029] Optionally, the setting the state attribute for the stored data comprises:

[0030] setting a deleted state for the data to be deleted;

[0031] setting an added state for the data other than the data to be deleted in the stored data.

[0032] Optionally, the storing the data comprises:

[0033] storing the data in a public database, the public database being a database for common access by the management node and the service node.

[0034] Optionally, the storing the data comprises:

[0035] storing the data in a private database, the private database being a database for access by the management node.

[0036] Optionally, the method further comprises:

[0037] deleting the data with the deleted state attribute after a preset time.

[0038] According to a third aspect of the embodiments of the present specification, a data processing method is provided, applied to a service node in a data processing system, and the method comprises:

[0039] obtaining data stored by a management node, the data carrying a state attribute;

[0040] generating a target event according to the state attribute of the data;

[0041] updating data in an external service according to the target event.

[0042] Optionally, the state attribute comprises a deleted state and an added state, and the generating a target event according to the state attribute of the data comprises:

[0043] traversing each data obtained to determine the state attribute of the data;

[0044] if the state attribute of the data is a deleted state, generating a deletion event according to the data;

[0045] if the state attribute of the data is an added state, generating an added event according to the data.

[0046] Optionally, the data in the external service is updated in an idempotent operation manner.

[0047] Optionally, the obtaining data stored by a management node comprises:

[0048] If the management node stores the data in a public database, the data stored by the management node is obtained from the public database, the public database being a database that is accessed by the management node and the service node.

[0049] Optionally, the obtaining of the data stored by the management node comprises:

[0050] If the management node stores the data in a private database, the data stored by the management node in the private database is obtained through a full-data query operation provided by the management node, the private database being a database that is accessed by the management node.

[0051] According to a fourth aspect of the embodiments of the present specification, a computing device is provided, comprising:

[0052] a memory and a processor;

[0053] The memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions:

[0054] store data;

[0055] set a state attribute for the stored data.

[0056] According to a fifth aspect of the embodiments of the present specification, a computing device is provided, comprising:

[0057] a memory and a processor;

[0058] The memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions:

[0059] obtain data stored by a management node, the data carrying a state attribute;

[0060] generate a target event according to the state attribute of the data;

[0061] update data in an external service according to the target event.

[0062] According to a sixth aspect of the embodiments of the present specification, a computer-readable storage medium is provided, which stores computer-executable instructions, the instructions being executed by a processor to implement the steps of the data processing method provided by any of the second aspects.

[0063] According to a seventh aspect of the embodiments of the present specification, a computer-readable storage medium is provided, which stores computer-executable instructions, the instructions being executed by a processor to implement the steps of the data processing method provided by any of the third aspects.

[0064] The data processing system provided by the specification comprises a management node and a service node, the management node is configured to set a state attribute for stored data, the service node is configured to obtain the data stored by the management node, generate a target event according to the state attribute of the data, and then update the data in the external service according to the target event. In this case, the data stored by the management node is all set with a state attribute, after the service node obtains the data stored by the management node, the service node can directly generate a target event according to the state attribute of the data and update the data in the external service, the service node does not need to store the full amount of data of the management node locally, nor does it need to compare the data, which greatly saves the processing time and storage consumption. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 is a structural schematic diagram of a data processing system provided by an embodiment of the specification;

[0066] Figure 2 is a flowchart of a first data processing method provided by an embodiment of the specification;

[0067] Figure 3 is a flowchart of a second data processing method provided by an embodiment of the specification;

[0068] Figure 4 is a flowchart of a third data processing method provided by an embodiment of the specification;

[0069] Figure 5 is a flowchart of a fourth data processing method provided by an embodiment of the specification;

[0070] Figure 6 is a structural block diagram of a computing device provided by an embodiment of the specification;

[0071] Figure 7 is a structural block diagram of another computing device provided by an embodiment of the specification. DETAILED DESCRIPTION

[0072] In the following description, many specific details are set forth in order to provide a thorough understanding of the specification. However, the specification can be practiced in many different ways beyond the specific embodiments described herein, and it is understood that persons having ordinary skill in the art can make and use modifications to the present specification without departing from the scope of the present specification, and therefore the present specification is not limited to the specific implementation disclosed below.

[0073] The terminology used in this disclosure of one or more embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0074] It is to be understood that, although the terms first, second, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. Rather, these terms are used solely to distinguish one from another only. For example, a first entity discussed below could later be discussed as a second entity, and similarly, a second entity discussed below could later be discussed as a first entity without departing from the scope of one or more embodiments. As used herein, the term "if' can be interpreted to mean "when" or "upon" or "in response to determining" taking into account the context in which the term is used.

[0075] First, a brief overview of the basic concepts of the present disclosure is provided.

[0076] In the distributed database scenario, the service node relies on the metadata information generated by the management node, such as permissions, addressing information, etc., to update the external service, that is, the service node continuously obtains the metadata information from the management node, and applies the metadata changes to the external service. When the data stored in the management node is updated, the service node can extract the data change event and update the data change to the external permission control and addressing service.

[0077] However, in the case of active deletion of part of the data by the management node, the service node cannot effectively perceive the deleted data. In the prior art, in order to enable the service node to perceive the deletion operation of the management node, the service node can store the full amount of data stored in the management node locally after obtaining the full amount of data for the first time, and then compare the full amount of data obtained each time with the cached data stored locally to determine the deleted data, generate a deletion event of the data, and update it to the external service.

[0078] For example, assuming there is a service node A, a management node B, a database, the existing data deletion event acquisition scheme is as follows: B stores the generated data in the database; A accesses the database at startup, extracts all data, updates the data addition event to the external service, and stores it in the node; A starts a background thread, which accesses the database in a loop, extracts all data, and the background thread compares the data extracted from the database with the stored data, extracts the data that exists in the stored data set and does not exist in the data set extracted from the database, considers that the data is deleted, generates a data deletion event, and updates the external service and modifies the maintained data.

[0079] From the above process, in the prior art, to acquire a data deletion event, the full amount of data needs to be stored locally, which consumes memory. If the full amount of data is too large, it needs to be stored on a disk, which affects the read-write performance and the read-write time is too long. In addition, comparing the full amount of data with the cached data to determine whether the data is deleted consumes time.

[0080] In order to avoid the service node extracting the data deletion event through data comparison, reduce storage consumption, and save time, the present specification designs a new data processing system and method in the distributed file storage system HDFS to generate a data deletion event. The management node can add a state attribute to the data, set the state of the deleted data to "deleted", so that the service node directly determines the data deletion event through the state attribute of the data, rather than querying the full amount of data and comparing the existing data to confirm the data deletion event. Assuming that the data stored in the database is n, the storage consumption of extracting the data deletion event by comparing the new data with the full amount of data cached locally in the prior art is n data. The data deletion event extraction method provided in the present specification does not need to cache the full amount of data locally, nor does it need to actively extract the change event, which greatly saves the processing time and storage consumption.

[0081] In the present specification, a data processing system is provided, and the present specification also relates to a data processing method, a computing device, and a computer readable storage medium, which are described in detail one by one in the following embodiments.

[0082] Figure 1 A structural schematic diagram of a data processing system provided according to an embodiment of the present specification is shown, which includes a management node 102 and a service node 104.

[0083] The management node 102 is configured to set a state attribute for the stored data;

[0084] The service node 104 is configured to obtain the data stored by the management node, generate a target event according to a state attribute of the data, and update the data in the external service according to the target event.

[0085] In an optional embodiment, the management node 102 is further configured to:

[0086] set a deleted state for the data to be deleted;

[0087] set an added state for the data other than the data to be deleted.

[0088] In an optional embodiment, the service node 104 is further configured to:

[0089] determine a state attribute of each data obtained;

[0090] if the state attribute of the data is the deleted state, generate a deletion event according to the data;

[0091] if the state attribute of the data is the added state, generate an added event according to the data.

[0092] In an optional embodiment, the data in the external service is updated in an idempotent operation manner.

[0093] In an optional embodiment, the system further comprises a common database accessed by the management node 102 and the service node 104, and the management node 102 is further configured to:

[0094] store the data in the common database;

[0095] Correspondingly, the service node 104 is further configured to:

[0096] obtain the data stored by the management node 102 from the common database.

[0097] In an optional embodiment, the system further comprises a private database accessed by the management node 102, and the management node 102 is further configured to:

[0098] store the data in the private database;

[0099] Correspondingly, the service node 104 is further configured to:

[0100] obtain the data stored by the management node 102 in the private database through a full data query operation provided by the management node 102.

[0101] In an optional embodiment, the management node 102 is further configured to:

[0102] After the preset time, the data with the state attribute of the deleted state is deleted.

[0103] The data processing system provided in the specification includes a management node and a service node. The management node is configured to set a state attribute for stored data. The service node is configured to obtain the data stored by the management node, generate a target event according to the state attribute of the data, and then update the data in the external service according to the target event. In this case, the data stored by the management node is all set with a state attribute. After the service node obtains the data stored by the management node, it can directly generate a target event according to the state attribute of the data and update the data in the external service. The service node does not need to store the full amount of data of the management node locally, nor does it need to compare data, greatly saving processing time and storage consumption.

[0104] Figure 2 A flowchart of a data processing method according to an embodiment of the specification is shown. The method is applied to Figure 1 The management node in the data processing system includes the following steps:

[0105] Step 202: Store the data.

[0106] Specifically, after the management node is started, it will generate metadata information such as permissions, addressing information, etc., and the management node will store the generated data.

[0107] In one or more embodiments of the present embodiment, in the case where the data processing system further includes a common database for common access by the management node and the service node, the management node can store the data in the common database, so that the service node can directly access the common database to obtain the data stored by the management node, that is, the common database can directly provide data query operations at this time.

[0108] In one or more embodiments of the present embodiment, in the case where the data processing system further includes a private database for access only by the management node, the management node can store the data in the private database. If the service node needs to obtain the data stored by the management node later, it can obtain the data stored by the management node in the private database through the full amount of data query operations provided by the management node to the outside, that is, the management node provides data query operations at this time.

[0109] In actual application, the data of the management and control node can be stored in a public database, a private database only accessible by the management and control node, or a local file. In the case where the management and control node stores data in a local file, the subsequent service node can also obtain the data stored in the local file by the management and control node through the full data query operation provided by the management and control node to the outside, that is, the data query operation is also provided by the management and control node.

[0110] In the present specification, the data of the management and control node can be stored in a public database, a private database only accessible by the management and control node, or a local file, and the flexibility of storing data is high. No matter where the management and control node stores the data, the subsequent service node can obtain the data stored by the management and control node, and realize the update of the service to the outside.

[0111] Step 204: setting a state attribute for the stored data.

[0112] Specifically, on the basis of storing data by the management and control node, a state attribute is further set for the stored data, and the state attribute is used to identify the state of the data, such as addition, deletion, etc.

[0113] In one or more embodiments of the present embodiment, a state attribute is set for the stored data, and the specific implementation manner can be:

[0114] The to-be-deleted data is set as a deleted state;

[0115] The other data in the stored data except the to-be-deleted data is set as an added state.

[0116] In actual application, after the management and control node is started and data is generated, the data is stored, at this time, all the data is newly generated data, and the service node has not obtained it, so the management and control node will not delete some of the data (that is, there is no to-be-deleted data), that is, after the management and control node is started, the initial data generated can all be set as an added state. In the case where the management and control node is updated, the to-be-deleted data is set as a deleted state, and the other data in the stored data except the to-be-deleted data (the data in the initial data except the to-be-deleted data and the newly generated data) is set as an added state.

[0117] For example, after the management and control node is started, five data are generated and stored, which are data a, data b, data c, data d, and data e, and the management and control node sets the state attributes of the above five data as added states. Assuming that the management and control node updates the data, the update operation includes deleting data b, newly generating data f and data h, at this time, the state attribute of data b is set as a deleted state, and data a, data c, data d, data e, data f, and data h are all set as added states.

[0118] In actual application, in the case that the management and control node has an update operation, in addition to resetting the state attribute of all data stored by the management and control node, the state attribute of only the changed data can be modified.

[0119] For example, the management and control node generates and stores five data, i.e., data a, data b, data c, data d and data e, after being started. The management and control node sets the state attribute of the above five data to the newly added state. It is assumed that the management and control node updates the data, and the update operation includes deleting data b, newly generating data f and data h, so that data b, data f and data h are determined as the changed data. At this time, only the state attribute of data b, data f and data h is modified. Since data b is the data to be deleted, the state attribute of data b is modified to the deleted state, and data f and data h are newly added data, so they are both set to the newly added state.

[0120] It is worth noting that the management and control node sets the state attribute of the stored data. After the service node obtains the data stored by the management and control node, the target event can be generated according to the state attribute of the data, and the data in the update external service is updated. The service node does not need to store the full amount of data of the management and control node locally, nor does it need to compare the data, which greatly saves the processing time and storage consumption.

[0121] In one or more embodiments of the present embodiment, after the management and control node sets the state attribute of the stored data, the data can also be deleted. The specific implementation manner can be that the data with the state attribute of the deleted state is deleted after a preset time.

[0122] Specifically, the preset time can be set by the technician in advance. The length of the time can be determined according to the frequency of the service node obtaining the data, as long as the data with the state attribute of the deleted state is deleted after the service node obtains the data.

[0123] It is worth noting that the management and control node cannot delete the data with the state attribute of the deleted state immediately after setting the state attribute of the stored data, but waits for a period of time to determine that the service node has obtained the data, and then deletes the data, so as to ensure that the service node can obtain the data to be deleted, and update the external service according to the state attribute of the data to be deleted.

[0124] In addition, if the data with the state attribute of the deleted state is small, the management and control node can also not delete the data, but for the data that has been stored for a long time, it is meaningless to store and will occupy a certain memory, so it can be deleted. Therefore, in one or more embodiments of the present embodiment, the management and control node can also filter the stored data every preset time interval, and delete the data with the state attribute of the deleted state.

[0125] The data processing method provided in the specification, the management node stores data, sets a deleted state for data to be deleted, and sets a new state for other data, so that when the subsequent service node obtains the data stored by the management node, a deletion event and / or a new event can be generated according to the state attribute of the data, and the data in the external service is updated, that is, the subsequent service node does not need to store the full amount of data of the management node locally, and does not need to compare the data, so as to know that the management node has deleted the data, thereby greatly saving the processing time and storage consumption.

[0126] Figure 3 A flowchart of a data processing method according to an embodiment of the specification is shown, which is applied to a service node in a data processing system as shown Figure 1 The service node specifically includes the following steps:

[0127] Step 302: Obtain the data stored by the management node, the data carrying a state attribute.

[0128] In actual application, the management node sets a state attribute for the data after storing the data, so that the data stored by the management node obtained by the service node carries a state attribute.

[0129] In one or more embodiments of the present embodiment, the location of the data stored by the management node is different, and the location of the data obtained by the service node is also different. If the management node stores data in a public database, the service node obtains the data stored by the management node from the public database, and the public database is a database for the management node and the service node to access. If the management node stores data in a private database, the service node obtains the data stored by the management node in the private database through a full data query operation provided by the management node, and the private database is a database for the management node to access.

[0130] Step 304: Generate a target event according to the state attribute of the data.

[0131] Specifically, on the basis of the service node obtaining the data stored by the management node, the service node will further generate a target event according to the state attribute of the data.

[0132] In one or more embodiments of the present embodiment, after the service node obtains the data stored by the management node, the service node will traverse the obtained data to update the external service, and the specific implementation process can be:

[0133] Traverse each data obtained to determine the state attribute of the data.

[0134] If the state attribute of the data is the deleted state, a deletion event is generated according to the data;

[0135] If the state attribute of the data is the added state, an addition event is generated according to the data.

[0136] Specifically, the deletion event refers to an event conveying the information that the data is deleted when two nodes communicate. The addition event refers to an event conveying the information that the data is added when two nodes communicate.

[0137] In actual application, when the management node changes the stored data, the state attribute of the data to be deleted is set to the deleted state, and the state attribute of the data not to be deleted can be set to the added state. Thus, after the service node obtains the data stored by the management node, the state attribute of each data can be traversed, and a deletion or addition event is generated according to the state attribute.

[0138] Step 306: updating the data in the external service according to the target event.

[0139] Specifically, on the basis that the service node generates the target event according to the state attribute of the data, the data in the external service is further updated according to the target event.

[0140] In one or more embodiments of the present embodiment, after the service node generates the deletion event (and / or the addition event), the data in the external service is updated according to the deletion event (and / or the addition event) in an idempotent operation manner, that is, the operation of updating the change of the data to the external service is idempotent.

[0141] Specifically, idempotent is a mathematical and computer science concept, and an idempotent operation refers to an operation whose effect is the same as that of a single operation when repeated multiple times, that is, the characteristic of the idempotent operation is that the effect of any multiple executions is the same as that of a single execution.

[0142] In actual application, the service node obtains the full data stored by the management node each time, and thus for the same data, the data can be updated to the external service multiple times. Thus, in the present specification, the operation of updating the change of the data to the external service is idempotent, which ensures that the result of updating the same data to the external service multiple times is the same as that of updating the same data to the external service once, and further ensures the accuracy of the update.

[0143] For example, the management node generates and stores five data, data a, data b, data c, data d and data e, after starting up, and sets the state attribute of the above five data to all be the new state. The service node acquires the five data stored by the management node after starting up, and since the state attribute of the five data is all the new state, the service node generates the new event of data a, data b, data c, data d and data e according to data a, data b, data c, data d and data e, respectively, and updates the data in the external service according to the new event of the data a, data b, data c, data d and data e.

[0144] Suppose the management node updates the data, and the update operation includes deleting data b, newly generating data f and data h, so the state attribute of data b is the deleted state, and the state attribute of data a, data c, data d, data e, data f and data h is the new state. The service node acquires the data stored by the management node, and generates the new event of data a, data c, data d, data e, data f and data h according to data a, data c, data d, data e, data f and data h, respectively, generates the delete event of data b according to data b, and updates the data in the external service according to the new event of data a, data c, data d, data e, data f and data h, and the delete event of data b.

[0145] Suppose that after 5 seconds, the management node does not update the data, but the service node acquires the data stored by the management node, at this time, the state attribute of data a, data c, data d, data e, data f and data h is still the new state, and the state attribute of data b is still the deleted state. At this time, the new event of data a, data c, data d, data e, data f and data h is still generated, the delete event of data b is still generated, and the data in the external service is updated again according to the new event of data a, data c, data d, data e, data f and data h, and the delete event of data b.

[0146] In the above process, the update operation of updating the data in the external service according to the new event of data a, data c, data d and data e is executed three times, and the update operation of updating the data in the external service according to the delete event of data b is executed twice, but since the data in the external service is updated according to the new event and the delete event in the form of an idempotent operation, the result of updating twice and updating once is the same, and the data will not be wrong.

[0147] In actual application, if the operation of updating the data to the external service is not idempotent, that is, the result of updating the same data to the external service multiple times is different from that of updating the same data to the external service once, in order to ensure the accuracy of the update, after the service node obtains the data stored by the management and control node, generates the target event according to the state attribute of the data, the service node can perform a local confirmation to determine whether the data in the external service has been updated according to the target event, if yes, the data in the external service is not updated according to the target event, if no, the data in the external service is updated according to the target event.

[0148] The data processing method provided in the specification, the service node can obtain the data stored by the management and control node, the data carries a state attribute, then generates a target event according to the state attribute of the data, and updates the data in the external service according to the target event. In this case, if the state attribute of the data is a deleted state, a deletion event can be generated, and if the state attribute of the data is an added state, an addition event can be generated, that is, the service node can generate a deletion event and / or an addition event directly according to the state attribute of the data, and update the data in the external service, the service node does not need to store the full amount of data of the management and control node locally, and does not need to compare the data, that is, the service node can know that the management and control node has deleted the data, which greatly saves the processing time and storage consumption.

[0149] The following describes the data processing method provided in the specification in combination with the accompanying Figure 4 The data processing method provided in the specification is described in further detail with reference to the accompanying drawings. Figure 4 A flowchart of a data processing method provided in an embodiment of the specification is shown, which specifically includes the following steps:

[0150] Step 402: The management and control node stores data in a public database, and the public database is a database accessed by the management and control node and the service node.

[0151] Step 404: The management and control node sets a deleted state to the data to be deleted, and sets an added state to other data in the stored data except the data to be deleted.

[0152] Step 406: The service node obtains the data stored by the management and control node from the public database.

[0153] Step 408: The service node traverses each data obtained, determines the state attribute of the data, if the state attribute of the data is a deleted state, generates a deletion event according to the data, if the state attribute of the data is an added state, generates an addition event according to the data.

[0154] Step 410: updating the data in the external service in an idempotent operation manner according to the deletion event and / or the addition event.

[0155] Step 412: deleting, by the management node, the data with the state attribute being the deleted state after a preset time.

[0156] It should be noted that the operation step of deleting, by the management node, the data with the state attribute being the deleted state can be performed only after the service node obtains the data, that is, step 412 can be performed only after step 406, and does not necessarily be performed after step 410.

[0157] The data processing method provided in the specification, the management node stores the data and sets the deleted state for the data to be deleted and sets the addition state for other data; when the service node obtains the data stored by the management node, the service node can directly generate a deletion event or an addition event according to the state attribute of the data, and update the data in the external service, that is, the subsequent service node does not need to store the full amount of data of the management node locally, and does not need to compare the data, so as to know that the management node has deleted the data, thereby greatly saving the processing time and storage consumption.

[0158] It should be noted that the technical solutions of the data processing method shown in the above Figure 4 The technical solutions of the data processing method shown in the above Figure 2 and Figure 3 The technical solutions of the data processing method shown in the above Figure 4 The details of the technical solutions of the data processing method shown in the above Figure 2 and Figure 3 The details of the technical solutions of the data processing method shown in the above

[0159] The following will be described in detail with reference to the accompanying drawings. Figure 5 The data processing method provided in the specification is taken as an example to further illustrate the data processing method. Wherein, Figure 5 Fig. 1 shows a processing flowchart of a data processing method provided in an embodiment of the specification, and specifically includes the following steps:

[0160] Step 502: storing, by a management node, data in a private database, the private database being a database accessed by the management node.

[0161] Step 504: setting, by the management node, a deleted state for to-be-deleted data, and setting an addition state for other data in the stored data except the to-be-deleted data.

[0162] Step 506: obtaining, by a service node, the data stored in the private database by the management node through a full amount of data query operation provided by the management node.

[0163] Step 508: The service node iterates through each piece of data it has obtained, determines the status attribute of the data, and if the status attribute of the data is deleted, a deletion event is generated based on the data; if the status attribute of the data is added, an addition event is generated based on the data.

[0164] Step 510: Update the data in the external service in an idempotent manner based on the deletion event and / or addition event.

[0165] Step 512: After a preset time, the control node deletes the data whose status attribute is "deleted".

[0166] It should be noted that the operation step of the control node to delete data with the status attribute of being deleted only needs to be executed after the service node obtains the data. That is, step 512 only needs to be executed after step 506, and does not necessarily have to be executed after step 510.

[0167] The data processing method provided in this manual involves the control node storing data and setting the deleted data to a deleted state, while setting the added state for other data. When the service node obtains the data stored by the control node, it can directly generate a delete event or an add event based on the data's state attributes and update the data in the external service. In other words, subsequent service nodes do not need to store the full data of the control node locally, nor do they need to perform data comparisons to know that the control node has deleted data, which greatly saves processing time and storage consumption.

[0168] It should be noted that the above Figure 5 The technical solution of the data processing method shown is the same as that described above. Figure 2 and Figure 3 The technical solutions of the data processing methods shown belong to the same concept. Figure 5 For details not described in the technical solution of the data processing method shown above, please refer to the above. Figure 2 and Figure 3 The technical solution of the data processing method shown is described.

[0169] Figure 6 A structural block diagram of a computing device 600 according to an embodiment of this specification is shown. The components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and a database 650 is used to store data.

[0170] The computing device 600 also includes an access device 640 that enables the computing device 600 to communicate via one or more networks 660. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or combinations of such networks, such as the Internet. The access device 640 can include one or more of any type of network interface (for example, a network interface card (NIC)), such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a global microwave interconnection access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and so on.

[0171] In one embodiment of the present specification, the above-mentioned components of the computing device 600 and other components not shown in the Figure 6 may be connected to each other, for example, through a bus. It should be understood that Figure 6 The computing device structure diagram shown is only for the purpose of example, and is not a limitation on the scope of the present specification. Other components can be added or replaced as needed by those skilled in the art.

[0172] The computing device 600 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (for example, a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, and so on), a mobile phone (for example, a smartphone), a wearable computing device (for example, a smart watch, smart glasses, and so on), or other types of mobile devices, or a stationary computing device such as a desktop computer or a PC. The computing device 600 can also be a mobile or stationary server.

[0173] wherein the processor 620 is configured to execute computer-executable instructions:

[0174] store data;

[0175] set a state attribute for the stored data.

[0176] The above is a schematic scheme of a computing device according to the present embodiment. It should be noted that the technical scheme of the computing device belongs to the same concept as the technical scheme of the data processing method described above Figure 2 , and the details of the technical scheme of the computing device that are not described in detail can be referred to the description of the technical scheme of the data processing method described above Figure 2 .

[0177] Figure 7A structural block diagram of a computing device 700 is shown, according to an embodiment of the present specification. The components of the computing device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected with the memory 710 through a bus 730, and a database 750 is used to save data.

[0178] The computing device 700 also includes an access device 740, which enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include the public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 can include one or more of any type of network interface (e.g., network interface card (NIC)), wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0179] In an embodiment of the present specification, the above-mentioned components of the computing device 700 and other components not shown in the above-mentioned components can be connected with each other, for example, through a bus. It should be understood that Figure 7 the computing device structure block diagram shown is only for the purpose of example, and is not a limitation on the scope of the present specification. Other components can be added or replaced by those skilled in the art as needed. Figure 7 the computing device structure block diagram shown is only for the purpose of example, and is not a limitation on the scope of the present specification. Other components can be added or replaced by those skilled in the art as needed.

[0180] The computing device 700 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or a PC. The computing device 700 can also be a mobile or stationary server.

[0181] The processor 720 is configured to execute computer-executable instructions, including:

[0182] Obtaining data stored by the management node, the data carrying a state attribute;

[0183] Generating a target event according to the state attribute of the data;

[0184] Updating data in the external service according to the target event.

[0185] The above is a schematic scheme of the computing device of the embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the data processing method shown in the above Figure 3 The technical scheme of the data processing method shown in the above Figure 3 The technical scheme of the data processing method shown in the above

[0186] An embodiment of the present specification further provides a computer readable storage medium, which stores computer instructions, and the instructions are executed by a processor to be used for:

[0187] storing data;

[0188] setting a state attribute for the stored data.

[0189] The above is a schematic scheme of the computer readable storage medium of the embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the data processing method shown in the above Figure 2 The technical scheme of the data processing method shown in the above Figure 2 The technical scheme of the data processing method shown in the above

[0190] An embodiment of the present specification further provides a computer readable storage medium, which stores computer instructions, and the instructions are executed by a processor to be used for:

[0191] obtaining data stored by a control node, the data carrying a state attribute;

[0192] generating a target event according to the state attribute of the data;

[0193] updating data in an external service according to the target event.

[0194] The above is a schematic scheme of the computer readable storage medium of the embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the data processing method shown in the above Figure 3 The technical scheme of the data processing method shown in the above Figure 3 The technical scheme of the data processing method shown in the above

[0195] The above-described embodiments of the application have several aspects, no single one of which is solely responsible for the application's desirable attributes. Without limiting the scope of the application as expressed by the claims which follow, some further embodiments make these aspects even more useful. Other embodiments can result in less desirable attributes.

[0196] The computer readable medium can include any entity or apparatus capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, software distribution medium, etc. It should be noted that the computer readable medium can include appropriate contents according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0197] It should be noted that for the foregoing method embodiments, the acts described can be performed in a different order than that described, and that various interlocking and / or parallel conduits can be used. Furthermore, various additional or different acts can be performed and still conform to the description of the embodiments. Also, it should be noted that the description of the embodiments is not as limiting as the acts described since additional acts can be performed and still conform to the description of the embodiments.

[0198] In the above-described embodiments, the description of each embodiment is focused on different aspects, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0199] The preferred embodiments of the present specification disclosed above are only used to help explain the present specification. The alternative embodiments do not describe all the details and do not limit the application to the specific embodiments described. Obviously, according to the content of the present specification, many modifications and changes can be made. The present specification selects and describes these embodiments in order to better explain the principles and practical applications of the present specification, so that those skilled in the art can well understand and utilize the present specification. The present specification is limited by the claims and their full scope and equivalents.

Claims

1. A data processing system in a distributed file storage system, comprising: Control nodes and service nodes; The control node is configured to set status attributes for the stored data. The control node is further configured to set an "add" status for the generated initial data after the control node is started, set a "deleted" status for the data to be deleted after the control node is updated, and set an "add" status for the data in the initial data other than the data to be deleted and the newly generated data. The service node is configured to acquire data stored by the control node, generate a target event based on the status attributes of the data, wherein the service node is further configured to generate a deletion event if the status attribute of the data is deleted, and generate a addition event if the status attribute of the data is added; and update the data in the external service based on the target event, wherein the operation of updating the external service is idempotent. The control node is further configured to delete data with the status attribute of "deleted" after a preset time, wherein the preset time is determined based on the frequency at which the service node acquires data.

2. The data processing system according to claim 1, wherein the service node is further configured as follows: Iterate through each piece of data obtained and determine the state attributes of the data; If the status attribute of the data is a newly added status, then a new event is generated based on the data.

3. The data processing system according to claim 1, further comprising a public database accessible to both the control node and the service node, wherein the control node is further configured to: The data is stored in the public database; Accordingly, the service node is further configured as follows: The data stored by the control node is obtained from the public database.

4. The data processing system according to claim 1, further comprising a private database accessible to the control node, wherein the control node is further configured to: The data is stored in the private database; Accordingly, the service node is further configured as follows: The data stored in the private database of the control node can be obtained through the full data query operation provided by the control node.

5. A data processing method applied to a control node in a data processing system, the method comprising: Set the new status for the generated initial data and store the initial data; Setting status attributes for stored data includes setting a deleted status for data to be deleted after an update to the management node, and setting an added status for data other than the data to be deleted and newly generated data in the initial data. The stored data and its status attributes are used to cause service nodes to generate target events. These target events include deletion and addition events. The deletion event is generated based on the data when its status attribute is deleted, and the addition event is generated based on the data when its status attribute is added. The target events are used to cause service nodes to update data in external services based on the target events, and the operation of updating external services is idempotent. After a preset time, data with the status attribute set to "deleted" is deleted. The preset time is determined based on the frequency at which the service node acquires data.

6. The data processing method according to claim 5, wherein storing the data includes: The data is stored in a public database, which is a database that can be accessed by both the control node and the service node.

7. The data processing method according to claim 5, wherein storing the data includes: The data is stored in a private database, which is accessible to the control node.

8. A data processing method applied to a service node in a data processing system, the method comprising: The system acquires data stored at the management node. This data carries status attributes, including deleted and newly generated statuses. Data in the deleted status is deleted from the management node after a preset time has elapsed since it was set to the deleted status. The preset time is determined based on the frequency at which the service node acquires data. The management node sets status attributes for the data stored at the management node. This data includes initial data, data to be deleted, and newly generated data. The initial data is set to the newly generated status after the management node is started. The data to be deleted is set to the deleted status after the management node is updated. All data in the initial data except for the data to be deleted, as well as the newly generated data, are set to the newly generated status after the management node is updated. Based on the state attributes of the data, a target event is generated, wherein generating the target event based on the state attributes of the data includes: if the state attribute of the data is a deleted state, then a deletion event is generated based on the data; if the state attribute of the data is a newly added state, then a newly added event is generated based on the data. Based on the target event, update the data in the external service, wherein the operation of updating the external service is idempotent.

9. The data processing method according to claim 8, wherein generating a target event based on the state attributes of the data includes: Iterate through each piece of data obtained and determine the state attributes of the data; If the status attribute of the data is a newly added status, then a new event is generated based on the data.

10. The data processing method according to claim 8, wherein acquiring the data stored by the control node includes: If the control node stores the data in a public database, then the data stored by the control node is retrieved from the public database, which is a database that is accessible to both the control node and the service node.

11. The data processing method according to claim 8, wherein acquiring the data stored by the control node includes: If the control node stores the data in a private database, the data stored in the private database can be obtained through the full data query operation provided by the control node. The private database is a database accessible to the control node.

12. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions: Set the new status for the generated initial data and store the initial data; Setting status attributes for stored data includes setting a deleted status for data to be deleted after an update to the management node, and setting an added status for data other than the data to be deleted and newly generated data in the initial data. The stored data and its status attributes are used to cause service nodes to generate target events. These target events include deletion and addition events. The deletion event is generated based on the data when its status attribute is deleted, and the addition event is generated based on the data when its status attribute is added. The target events are used to cause service nodes to update data in external services based on the target events, and the operation of updating external services is idempotent. After a preset time, data with the status attribute set to "deleted" is deleted. The preset time is determined based on the frequency at which the service node acquires data.

13. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions: The system acquires data stored at the management node. This data carries status attributes, including deleted and added statuses. Data in the deleted status is deleted from the management node after a preset time has elapsed since it was set to the deleted status. The preset time is determined based on the frequency at which the service node acquires data. The management node sets status attributes for the data stored at the management node. This data includes initial data, data to be deleted, and newly generated data. The initial data is set to the added status after the management node is started. The data to be deleted is set to the deleted status after the management node is updated. All data in the initial data except for the data to be deleted, as well as the newly generated data, are set to the added status after the management node is updated. Based on the state attributes of the data, a target event is generated, wherein generating the target event based on the state attributes of the data includes generating a deletion event based on the data if the state attribute of the data is a deleted state, and generating a new event based on the data if the state attribute of the data is a newly added state. Based on the target event, update the data in the external service, wherein the operation of updating the external service is idempotent.

14. A computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 5 to 7.

15. A computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 8 to 11.

Images