Data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a data processing method and device, electronic equipment and a storage medium, and relates to the technical field of data processing, in particular to the field of cloud computing and distributed systems. According to the specific implementation scheme, data needing to be synchronized in the ETCD are obtained; obtaining a native first key Key of the data and at least one version number of the data associated with the first Key; and generating at least one second Key according to the first Key, and associating each version number of the data with the second Key to generate an index structure of multi-version concurrency control MVCC.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, specifically to the field of cloud computing and distributed systems, and in particular to a data processing method, device, electronic device and storage medium. Background Art

[0002] ETCD is a key-value storage database that supports Pub-Sub mode. To ensure the stability of the database, the ETCD proxy module is usually used to access ETCD to reduce the traffic accessing ETCD. The current proxy module cannot completely proxy the list and monitor the watch traffic. The one-to-one design of ETCD's Key and Value also limits the flexibility of the business module, and data reading is subject to the design of the Key. Summary of the invention

[0003] The present disclosure provides a data processing method, device, electronic device and storage medium.

[0004] According to one aspect of the present disclosure, a data processing method is provided, which is executed by a distributed key-value storage system ETCD agent, including:

[0005] Obtain the data that needs to be synchronized in the ETCD;

[0006] Obtain a first key Key native to the data and at least one version number of the data associated with the first key;

[0007] At least one second key is generated according to the first key, and each version number of the data is associated with the second key to generate an index structure of multi-version concurrency control MVCC.

[0008] According to another aspect of the present disclosure, there is provided a data processing device, comprising:

[0009] A first acquisition module is used to acquire data that needs to be synchronized in the ETCD;

[0010] A second acquisition module, used to acquire a first key Key native to the data and at least one version number of the data associated with the first key;

[0011] The third acquisition module is used to generate at least one second key according to the first key, and associate each version number of the data with the second key to generate an index structure of multi-version concurrency control MVCC.

[0012] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect embodiment.

[0013] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the embodiment of the first aspect.

[0014] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising the steps of implementing the method described in the embodiment of the first aspect when the computer program is executed by a processor.

[0015] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0017] Figure 1 is a flow chart of a data processing method provided by an embodiment of the present disclosure;

[0018] Figure 2 is a flow chart of another data processing method provided by an embodiment of the present disclosure;

[0019] Figure 3 is a flow chart of another data processing method provided by an embodiment of the present disclosure;

[0020] Figure 4 is a flow chart of another data processing method provided by an embodiment of the present disclosure;

[0021] Figure 5 is a structural block diagram of a data processing device provided by an embodiment of the present disclosure;

[0022] Figure 6 A schematic block diagram of an electronic device for implementing an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0024] Data processing is the collection, storage, retrieval, processing, transformation and transmission of data. Its basic purpose is to extract and derive valuable and meaningful data for certain specific people from a large amount of data that may be disorganized and difficult to understand.

[0025] Cloud computing is an Internet-based computing model that provides computing, storage, and network resources through remote servers. Users do not need to worry about the maintenance and management of the underlying hardware, but can use these resources on demand through the Internet.

[0026] A distributed system is a software system built on a network. Its characteristic is that hardware or software components are distributed on different network computers and communicate and coordinate through message passing. A distributed system allows multiple servers or multiple computing units to collaborate to complete the overall computing task, and has a high degree of cohesion and transparency.

[0027] Figure 1 is a flow chart of a data processing method provided by an embodiment of the present disclosure; Figure 1 As shown, the method is executed by the distributed key-value storage system ETCD agent, and includes the following steps:

[0028] S101, obtaining data that needs to be synchronized in ETCD.

[0029] Distributed key-value storage system (etcetera distributed, ETCD) can be used for configuration management, service discovery and distributed coordination in distributed systems. It supports data monitoring and updating, and supports multi-version concurrency control (Multi-version Concurrency Control, MVCC). Each key-value pair in ETCD can have multiple versions. The versions are distinguished by the version number Revision. When the data is modified or deleted, the Revision will increase accordingly.

[0030] In some implementations, data can be acquired through an application programming interface (API), which may include conventional data read and write interfaces of ETCD, such as a Range interface, a DeleteRange interface, a Transmit interface (Txn), and a Watch interface; the data to be synchronized is acquired from ETCD by calling the interface.

[0031] Optionally, you can obtain the starting version number that needs to be synchronized, and pull data from etcd starting from the starting version number to obtain the data that needs to be synchronized.

[0032] S102, obtaining a data native first key Key and at least one version number of data associated with the first key.

[0033] It can be understood that the data in ETCD is stored in the form of key-value pairs. After determining the data that needs to be synchronized in ETCD, the native first key of the data can be determined.

[0034] In some implementations, a Key in ETCD can correspond to one or more version numbers. Whenever the Key is modified, ETCD will create a new version and assign it a unique version number. The version number can be used to identify changes in data in ETCD to ensure data consistency. After determining the first Key of the data, at least one version number of the data associated with the first Key can be obtained.

[0035] S103, generating at least one second key according to the first key, and associating each version number of the data with the second key to generate an index structure of multi-version concurrency control MVCC.

[0036] In some implementations, the first key can be input into the corresponding key generation function to obtain the second key. The second key can be one or more. The key generation function can be pre-configured according to different types or formats, thereby achieving efficient custom key generation.

[0037] It can be understood that the second Key is generated by the first Key, that is, each first Key corresponds to one or more second Keys, and the version number corresponding to the first Key is associated with each second Key; for example, assuming that the native first Key has 4 version numbers, and 3 custom second Keys are generated based on the first Key, the 4 version numbers are associated with each custom second Key respectively, thereby completing the association of each version number of the data with the second Key, so that the corresponding data can be quickly located according to the second Key.

[0038] The MVCC index structure contains the data location information, data version information, and second key information, which can support multi-version concurrency control and perform concurrent access and modification of data without affecting data consistency.

[0039] In this embodiment, by obtaining the data that needs to be synchronized in ETCD, determining the first key of the data and the version number of the data associated with the first key, generating one or more customized second keys based on the first key, and associating each version number of the data with the second key, an index structure of multi-version concurrency control MVCC is generated, and the one-to-one relationship between key values ​​in ETCD is expanded to a many-to-one relationship. Supporting multi-key indexing makes data reading more flexible and meets more application needs of users.

[0040] Figure 2 is a flow chart of another data processing method provided by an embodiment of the present disclosure; Figure 2 As shown, the method comprises the following steps:

[0041] S201, obtaining the data that needs to be synchronized in ETCD.

[0042] In some implementations, the version numbers can be compared by binary search to determine the oldest version number of the entire ETCD, and data can be pulled from the ETCD based on the oldest version number of the entire ETCD. It can be understood that binary search is an algorithm for searching for a specific element in an ordered array. In this embodiment, the range of the binary search is the range between 0 and the latest version number. A specific candidate version number is randomly selected within the range to determine whether the candidate version number exists in the ETCD. If so, the binary search upper bound is reduced and the search continues in a smaller range. If not, the binary search lower bound is increased and the search continues in a larger range until the binary search upper bound and the lower bound are adjacent or equal to each other, thereby obtaining the oldest version number of the entire ETCD. The oldest version number of the entire ETCD refers to the version number of the earliest recorded data in the ETCD. Pulling and synchronizing data from the ETCD based on the oldest version number of the entire ETCD can pull more data.

[0043] In some implementations, the CompactRevision of ETCD can be detected based on the Range interface and binary search of ETCD. The Range interface is an API provided by ETCD, which allows users to query data based on a specified key range. Compact Revision is a version number used for data compression in ETCD. When the amount of data in ETCD grows to a certain extent, in order to save storage space, ETCD will delete some historical versions of data and only retain data after the Compact Revision. Further, the oldest version number of the full amount can be obtained based on the Range interface and Compact Revision.

[0044] Furthermore, the maximum version number in the pulled data is determined, and the maximum version number in the pulled data is determined to be the oldest version number of the entire amount; the maximum version number (maximum Revision) in the pulled data is the current latest Revision, and the current latest Revision is used as the new oldest version number of the entire amount, so that data can be synchronized from the new oldest version number of the entire amount when data is pulled next time.

[0045] In some implementations, it is also possible to monitor the incremental data of ETCD starting from the oldest full version number and receive the incremental data of ETCD; this embodiment can use the watch interface of ETCD to monitor data changes, and combine the Compact Revision to determine the oldest full version number to start monitoring, thereby realizing the acquisition of incremental data starting from the oldest full version number and realizing efficient incremental data synchronization.

[0046] Furthermore, the maximum version number in the incremental data can be determined, and the maximum version number in the incremental data can be determined to be the current latest version number; it can be understood that the maximum version number in the incremental data is also the latest Revision in the currently received incremental data, and the latest Revision is also the current latest version number, which provides an accurate basis for subsequent data synchronization and facilitates efficient data synchronization.

[0047] In the embodiments of the present disclosure, the implementation method of step S201 can be implemented by any of the methods in the embodiments of the present disclosure, which is not limited here and will not be described in detail.

[0048] S202, obtaining a data-native first key Key and at least one version number of data associated with the first key.

[0049] In the embodiments of the present disclosure, the implementation method of step S202 can be implemented by any of the methods in the embodiments of the present disclosure, which is not limited here and will not be described in detail.

[0050] S203: Generate at least one second key based on the first key, and associate each version number of the data with the second key to generate an index structure of multi-version concurrency control MVCC.

[0051] In the embodiments of the present disclosure, the implementation method of step S203 may be implemented in any manner in the embodiments of the present disclosure, which is not limited here and will not be described in detail.

[0052] S204, write the version number of the data and all the keys of the data into the B-tree of the memory data storage.

[0053] B-tree is a self-balancing tree data structure consisting of one or more nodes that can keep data in order and support search, insertion, and deletion operations in logarithmic time.

[0054] In some implementations, the version number of the data can be determined as the Key on the B-tree node, and the data and the second Key are combined to obtain the value Value on the B-tree node; the Key and Value on the node are written into the node of the B-tree; that is, the Key on the B-tree node is the version number of the data, and the Value on the B-tree node is the combination of the data and the second Key, then the key-value pair (key, value) on the B-tree node is (version number of the data, data + second Key).

[0055] Optionally, when writing data to the B-tree, the B-tree can be write-locked, and the B-tree pointer can be determined through atomic operations based on the key on the node; the write lock is a synchronization mechanism used to ensure that only one thread or process modifies the B-tree at a certain time to prevent data inconsistency or structural damage caused by concurrent write operations; obtaining the B-tree pointer through atomic operations means that the B-tree pointer operation is atomic, avoiding data structure inconsistency caused by partial updates, and the node pointed to by the B-tree pointer is the node that should be written or updated based on the key on the node.

[0056] Further write the Key and Value on the node into the node pointed to by the B-tree pointer, that is, write the Key and Value on the node into the node that should be written to by the B-tree pointer.

[0057] Optionally, the B-tree pointer can also be copied and marked as a copy-on-write state; copy-on-write is a method that can improve data sharing and modification efficiency. When a node of the B-tree needs to be modified, the original node can be not modified directly, but the B-tree pointer pointing to the node can be copied. At this time, there are two independent B-tree pointers pointing to the same node, and the Key and Value are written based on the copied B-tree pointer, and a new B-tree node is created when writing, which does not affect the reading of the original B-tree pointer; after the Key and Value on the node are written to the new B-tree node, the B-tree pointer is updated through atomic operations, and the B-tree is unlocked, wherein the B-tree pointer updated by the atomic operation is the original B-tree pointer, which does not affect the reading of the original node when the node is updated, thereby achieving efficient data sharing and efficient modification efficiency.

[0058] In this embodiment, the oldest version number of the full amount of ETCD can be obtained, and data can be pulled from ETCD according to the oldest version number of the full amount to achieve data synchronization. The watch interface of ETCD can also be used to synchronize incremental data. The data to be synchronized is determined based on the two stages of full synchronization and incremental synchronization. The data synchronization range is maximized by the oldest version number of the full amount, which is convenient for realizing efficient data synchronization. The first key of more comprehensive data and the version number corresponding to the first key are further determined, at least one custom second key of the first key is generated and the version number of the data is associated with the second key, and the MVCC index structure is generated. Multi-key indexing is supported to make data reading more flexible; further, the version number of the data and all the keys of the data can be written into the B-tree of the memory data storage, which can keep the data in order, write the version number as the key on each node of the B-tree, and the data and the second key are combined as the key-value pair of the value, which can realize efficient data reading and writing, support multiple operations, and prevent data inconsistency or structural damage caused by concurrent write operations through write locking and write-time copy when writing to the B-tree, so as to realize efficient data sharing and efficient data modification.

[0059] Figure 3 is a flow chart of another data processing method provided by an embodiment of the present disclosure; Figure 3 As shown, the method comprises the following steps:

[0060] S301, obtaining the data that needs to be synchronized in ETCD.

[0061] In the embodiments of the present disclosure, the implementation method of step S301 may be implemented in any manner in the embodiments of the present disclosure, which is not limited here and will not be described in detail.

[0062] S302, obtaining a data native first key Key and at least one version number of data associated with the first key.

[0063] In the embodiments of the present disclosure, the implementation method of step S302 may be implemented in any manner in the embodiments of the present disclosure, which is not limited here and will not be described in detail.

[0064] S303: Input the first Key into a pre-built Key generation function, and generate a second Key through the Key generation function.

[0065] Optionally, one or more Key generation functions can be pre-built, each Key generation function corresponds to a different type, and one or more different customized second keys can be obtained by inputting the first Key into the Key generation function; the Key generation function can be constructed according to the rules, algorithms or logic of the features to meet different optimized storage and retrieval performance.

[0066] S304: Determine the format of the second Key or the Key generation function, and determine the type of the second Key.

[0067] In some implementations, the format of the second Key may be its corresponding composition method or expression form, such as a string or number format. The type of the corresponding second Key is determined according to the format of the second Key. The Key type may include a string type and a byte sequence type, etc.

[0068] In some implementations, the Key generation function may include a type constructed in advance, so the type of the generated second Key may also be determined according to the Key generation function.

[0069] S305, associate each version number of the data with the second Key to generate an index structure of multi-version concurrency control MVCC.

[0070] In some implementations, a first MVCC index structure is generated based on the first key; that is, the key in the first MVCC index structure is the first key native to the data, and each first key may correspond to one or more version numbers.

[0071] Furthermore, for each type of Key, a second MVCC index structure matching the type is constructed based on the second Key under the type; that is, the Key in the second MVCC index structure is the second Key under the same type, and the number of types is the same as the number of second MVCC index structures.

[0072] Subsequent data access based on the first MVCC index structure and multiple second MVCC index structures can improve the efficiency of data access and maintain data consistency and isolation during concurrent access control.

[0073] In the embodiments of the present disclosure, the implementation method of step S305 can be implemented by any of the methods in the embodiments of the present disclosure, which is not limited here and will not be described in detail.

[0074] S306, write the version number of the data and all the keys of the data into the B-tree of the memory data storage.

[0075] In the embodiments of the present disclosure, the implementation method of step S306 may be implemented in any manner in the embodiments of the present disclosure, which is not limited here and will not be described in detail.

[0076] In this embodiment, the second key is generated and obtained through the key generation function to meet different optimized storage and retrieval performance, the type of each second key is distinguished, and the version number is associated with the second key to generate the second MVCC corresponding to each type and the first MVCC corresponding to the native first key, thereby obtaining an index structure of multi-version concurrent control MVCC, which can support multiple keys to query the same data, improve the efficiency of data access, quickly and correctly query the required data information according to the key and version number, and maintain data consistency and isolation during concurrent access control.

[0077] Figure 4 is a flow chart of another data processing method provided by an embodiment of the present disclosure; Figure 4 As shown, the method comprises the following steps:

[0078] S401, obtaining the data that needs to be synchronized in ETCD.

[0079] In the embodiments of the present disclosure, the implementation method of step S401 may be implemented in any manner in the embodiments of the present disclosure, which is not limited here and will not be described in detail.

[0080] S402, obtaining a data native first key Key and at least one version number of data associated with the first key.

[0081] In the embodiments of the present disclosure, the implementation method of step S402 may be implemented in any manner in the embodiments of the present disclosure, which is not limited here and will not be described in detail.

[0082] S403: Input the first Key into a pre-built Key generation function, and generate a second Key through the Key generation function.

[0083] In the embodiments of the present disclosure, the implementation method of step S403 may be implemented in any manner in the embodiments of the present disclosure, which is not limited here and will not be described in detail.

[0084] S404, determine the format of the second Key or the Key generation function, and determine the type of the second Key.

[0085] In the embodiments of the present disclosure, the implementation method of step S404 may be implemented in any manner in the embodiments of the present disclosure, which is not limited here and will not be described in detail.

[0086] S405, associate each version number of the data with the second Key to generate an index structure of multi-version concurrency control MVCC.

[0087] In the embodiments of the present disclosure, the implementation method of step S405 can be implemented by any of the methods in the embodiments of the present disclosure, which is not limited here and will not be described in detail.

[0088] S406, write the version number of the data and all the keys of the data into the B-tree of the memory data storage.

[0089] In the embodiments of the present disclosure, the implementation method of step S406 may be implemented in any manner in the embodiments of the present disclosure, which is not limited here and will not be described in detail.

[0090] S407, receiving a list request from the client, and querying target data according to the List request.

[0091] The List request includes a query Key; according to the query interface and the type of the query Key corresponding to the List request, a target MVCC index structure is determined from multiple MVCC index structures.

[0092] Optionally, the query interface may include a range interface Range and a newly added custom range interface CustomRange, different query interfaces correspond to different MVCC index structures, and different query Key types correspond to different second MVCC index structures.

[0093] In some implementations, in response to the receiving interface being the original range query interface Range of the ETCD agent, the first MVCC index structure is determined as the target MVCC index structure; in some implementations, in response to the receiving interface being the newly added range query interface Custom Range of the ETCD agent, the target MVCC index structure is determined from multiple second MVCC index structures according to the Key type; multi-Key concurrent control can be implemented to improve data access efficiency.

[0094] Furthermore, according to the query Key, an index query is performed in the target MVCC index structure to obtain the target version number corresponding to the query Key; that is, the first Key or second Key consistent with the query Key is determined in the target MVCC index structure, and then the target version number corresponding to the query Key is obtained according to the version number corresponding to the first Key or the second Key.

[0095] Further, according to the target version number, the target data is obtained by querying in the B-tree of the memory data storage; optionally, based on the target version number, the B-tree pointer can be determined by atomic operation; the node pointed to by the B-tree pointer is determined, and the target data corresponding to the target version number is determined from the pointed node.

[0096] It can be understood that the key of the node in the B-tree is the version number of the data. Therefore, the target version number can be used as the index key to search in the B-tree, and the B-tree pointer can be determined through atomic operations to ensure data consistency and integrity. The node pointed to by the B-tree pointer is determined to be the node corresponding to the target version number, and the target data corresponding to the target version number is determined from the value Value of the node pointed to by the B-tree pointer.

[0097] S408, receiving a watch request from the client, and pushing an event to the client according to the watch request.

[0098] The watch request initiated by the client is used to request to monitor the occurrence of a certain event. The watch request may include information such as the type of event to be monitored, such as events such as the addition and deletion of data.

[0099] In response to monitoring the occurrence of an event, the first key and the second key corresponding to the event are determined; that is, when monitoring the occurrence of an event, the first key and the second key of the data corresponding to the event are determined to identify the data corresponding to the event.

[0100] Furthermore, the listeners corresponding to the first key and the second key are determined. Optionally, the listeners can be registered in different watch tables, namely the single key watch table and the prefix watch interval tree; when the client monitors events for a specific key, the listener is registered in the single key watch table, and when the client monitors events for more than a single feature key, the listener is registered in the prefix watch interval tree, and all the first keys and second keys are traversed to determine the corresponding listener.

[0101] Furthermore, an event is sent to the listener, and the event is pushed to the client through the listener, that is, the event is sent to the listener corresponding to the first Key and the second Key. The event information may include the corresponding Key, data, and Value, etc. The listener further pushes the event to the client, so that the client can obtain relevant events in time.

[0102] In this embodiment, after the MVCC index structure is determined according to the first key and the second key of the data, and the data is written into the storage B-tree, the client's list request and watch request can be received, the corresponding target MVCC index structure is determined according to the client's list request, and the target version number is queried according to the target MVCC index structure, and the target data is searched in the B-tree with the target version number. It supports multi-key concurrent query, fully supports list semantics and complete traffic proxy, and can meet more user application needs; according to the client's watch request, the first key and the second key are determined when an event occurs, and the event is obtained and pushed according to the listeners corresponding to the first key and the second key respectively, to achieve multi-key support, and can list or watch based on the native key like the existing single key, without making additional design, facilitating data operation and processing, more flexibly querying the required data information, and improving data access efficiency.

[0103] Figure 5 is a structural block diagram of a data processing device provided by an embodiment of the present disclosure; Figure 5 As shown, the data processing device 500 includes:

[0104] The first acquisition module 501 is used to acquire data that needs to be synchronized in ETCD;

[0105] A second acquisition module 502 is used to acquire a data native first key Key and at least one version number of data associated with the first key;

[0106] The third acquisition module 503 is used to generate at least one second key according to the first key, and associate each version number of the data with the second key to generate an index structure of multi-version concurrency control MVCC.

[0107] In some implementations, the third acquisition module 503 further includes:

[0108] Write the data version number and all the keys of the data into the B-tree of the memory data storage.

[0109] In some implementations, the data includes the native value of ETCD and the first Key, and the third acquisition module 503 includes:

[0110] Determine the version number of the data as the Key on the B-tree node, and combine the data with the second Key to obtain the value Value on the B-tree node;

[0111] Write the Key and Value on the node into the node of the B-tree.

[0112] In some implementations, the third acquisition module 503 includes:

[0113] Write-lock the B-tree and determine the B-tree pointer through atomic operations based on the key on the node;

[0114] Write the Key and Value on the node into the node pointed to by the B-tree pointer.

[0115] In some implementations, the third acquisition module 503 further includes:

[0116] Duplicate the B-tree pointer and mark it as copy-on-write state;

[0117] After writing the Key and Value on the node into the node pointed to by the B-tree pointer, it also includes:

[0118] Update the B-tree pointer through atomic operations and unlock the B-tree.

[0119] In some implementations, the first acquisition module 501 includes:

[0120] Perform a binary comparison on the version numbers to determine the oldest version number of the entire ETCD, and pull data from ETCD based on the oldest version number of the entire ETCD;

[0121] Determine the maximum version number in the pulled data, and determine that the maximum version number in the pulled data is the oldest version number in the entire amount.

[0122] In some implementations, the first acquisition module 501 includes:

[0123] Monitor the incremental data of ETCD starting from the oldest version number of the full data, and receive the incremental data of ETCD;

[0124] Determine the maximum version number in the incremental data, and determine that the maximum version number in the incremental data is the current latest version number.

[0125] In some implementations, the third acquisition module 503 includes:

[0126] Input the first key into the pre-built key generation function, and generate the second key through the key generation function;

[0127] Determine the format of the second key or the key generation function, and determine the type of the second key.

[0128] In some implementations, the apparatus 500 further includes:

[0129] Based on the first Key, generate a first MVCC index structure;

[0130] For each type of Key, based on the second Key under the type, a second MVCC index structure matching the type is constructed.

[0131] In some implementations, the apparatus 500 further includes:

[0132] Receive a List request from a client, where the List request includes a query Key;

[0133] According to the query interface corresponding to the List request and the type of the query key, determine the target MVCC index structure from multiple MVCC index structures;

[0134] According to the query key, index the target MVCC index structure to obtain the target version number corresponding to the query key;

[0135] According to the target version number, query in the B-tree of the memory data storage to obtain the target data.

[0136] In some implementations, the apparatus 500 includes:

[0137] Based on the target version number, determine the B-tree pointer through atomic operations;

[0138] Determine the node pointed to by the B-tree pointer, and determine the target data corresponding to the target version number from the pointed node.

[0139] In some implementations, the apparatus 500 includes:

[0140] In response to the receiving interface being the original range query interface of the etcd agent, determining the first MVCC index structure as the target MVCC index structure;

[0141] In response to the receiving interface being a range query interface newly added by the ETCD agent, a target MVCC index structure is determined from multiple second MVCC index structures according to the Key type.

[0142] In some implementations, the apparatus 500 further includes:

[0143] Receive the client's watch request;

[0144] In response to monitoring the occurrence of an event, determining a first key and a second key corresponding to the event;

[0145] Determine the listeners corresponding to the first key and the second key respectively, send events to the listeners, and push the events to the client through the listeners.

[0146] In an embodiment of the present application, by obtaining the data that needs to be synchronized in ETCD, and determining the first key of the data and the version number of the data associated with the first key, one or more customized second keys are generated based on the first key, and each version number of the data is associated with the second key, an index structure of multi-version concurrency control MVCC is generated, and the one-to-one relationship between key values ​​in ETCD is expanded to a many-to-one relationship. Support for multi-key indexing makes data reading more flexible and meets more application needs of users.

[0147] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0148] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0149] Figure 6 A schematic block diagram of an electronic device for implementing an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0150] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0151] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0152] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as a data processing method. For example, in some embodiments, the data processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the data processing method in any other appropriate manner (e.g., by means of firmware).

[0153] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0154] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0155] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0156] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0157] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0158] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0159] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0160] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for processing data, wherein: The method is performed by an agent of the distributed key-value storage system ETCD, and includes: Obtain the data that needs to be synchronized in the ETCD; Obtain a first key Key native to the data and at least one version number of the data associated with the first key; At least one second key is generated according to the first key, and each version number of the data is associated with the second key to generate an index structure of multi-version concurrency control MVCC.

2. The method according to claim 1, wherein: After the index structure of the multi-version concurrent control MVCC is generated, the following step is also included: The version number of the data and all the keys of the data are written into the B-tree of the memory data storage.

3. The method according to claim 2, wherein: The data includes the native value of etcd and the first key, and the step of writing the version number of the data and all the keys of the data into the B-tree of the memory data storage includes: Determine that the version number of the data is a Key on a B-tree node, and combine the data and the second Key to obtain a value Value on the B-tree node; Write the Key and Value on the node into the node of the B-tree.

4. The method according to claim 3, wherein: Writing the Key and Value on the node into the node of the B-tree includes: Write-lock the B-tree, and determine the B-tree pointer through atomic operations based on the key on the node; Write the Key and Value on the node into the node pointed to by the B-tree pointer.

5. The method according to claim 4, wherein: Before writing the Key and Value on the node into the node pointed to by the B-tree pointer, the method further includes: Copy the B-tree pointer and mark it as copy-on-write state; After writing the Key and Value on the node into the node pointed to by the B-tree pointer, the method further includes: The B-tree pointer is updated through an atomic operation, and the B-tree is unlocked.

6. The method according to claim 1, wherein: The acquiring of data to be synchronized in the ETCD includes: Perform a binary comparison on the version numbers to determine the oldest full version number of the ETCD, and pull data from the ETCD based on the oldest full version number; Determine the maximum version number in the pulled data, and determine that the maximum version number in the pulled data is the oldest version number of the full amount.

7. The method according to claim 1 or 6, wherein: The acquiring of data to be synchronized in the ETCD includes: Starting from the oldest version number of the full amount, the incremental data of the ETCD is monitored and the incremental data of the ETCD is received; Determine the maximum version number in the incremental data, and determine that the maximum version number in the incremental data is the current latest version number.

8. The method according to any one of claims 1 to 7, wherein: The generating at least one second Key according to the first Key includes: Input the first Key into a pre-built Key generation function, and generate the second Key through the Key generation function; Determine the format of the second Key or the Key generation function, and determine the type of the second Key.

9. The method according to claim 8, wherein: The method further comprises: Based on the first Key, generate a first MVCC index structure; For each type of Key, based on the second Key under the type, a second MVCC index structure matching the type is constructed.

10. The method according to claim 9, wherein: The method further comprises: Receive a list request from a client, wherein the list request includes a query key; Determine a target MVCC index structure from multiple MVCC index structures according to the query interface corresponding to the List request and the type of the query Key; According to the query key, index the target MVCC index structure to obtain a target version number corresponding to the query key; According to the target version number, the target data is obtained by searching in the B-tree of the memory data storage.

11. The method according to claim 10, wherein: The querying in the B-tree storing the in-memory data to obtain the target data according to the target version number includes: Based on the target version number, determining the B-tree pointer through an atomic operation; The node pointed to by the B-tree pointer is determined, and the target data corresponding to the target version number is determined from the pointed node.

12. The method according to claim 10, wherein: The determining a target MVCC index structure from a plurality of MVCC index structures according to the query interface corresponding to the List request and the type of the query Key includes: In response to the receiving interface being the original range query interface of the ETCD agent, determining the first MVCC index structure as the target MVCC index structure; In response to the receiving interface being a range query interface newly added to the ETCD agent, the target MVCC index structure is determined from a plurality of second MVCC index structures according to the Key type.

13. The method according to claim 9, wherein: The method further comprises: Receive the client's watch request; In response to monitoring the occurrence of an event, determining a first Key and a second Key corresponding to the event; Determine the listeners corresponding to the first Key and the second Key respectively, send the event to the listeners, and push the event to the client through the listeners.

14. A data processing device, comprising: A first acquisition module is used to acquire data that needs to be synchronized in the ETCD; A second acquisition module, used to acquire a first key Key native to the data and at least one version number of the data associated with the first key; The third acquisition module is used to generate at least one second key according to the first key, and associate each version number of the data with the second key to generate an index structure of multi-version concurrency control MVCC.

15. The device according to claim 14, wherein: The third acquisition module further includes: The version number of the data and all the keys of the data are written into the B-tree of the memory data storage.

16. The device according to claim 15, wherein: The data includes the native value of ETCD and the first Key, and the third acquisition module includes: Determine that the version number of the data is a Key on a B-tree node, and combine the data and the second Key to obtain a value Value on the B-tree node; Write the Key and Value on the node into the node of the B-tree.

17. The device according to claim 16, wherein: The third acquisition module includes: Write-lock the B-tree, and determine the B-tree pointer through atomic operations based on the key on the node; Write the Key and Value on the node into the node pointed to by the B-tree pointer.

18. The device according to claim 17, wherein: The third acquisition module further includes: Copy the B-tree pointer and mark it as copy-on-write state; After writing the Key and Value on the node into the node pointed to by the B-tree pointer, the method further includes: The B-tree pointer is updated through an atomic operation, and the B-tree is unlocked.

19. The device according to claim 14, wherein: The first acquisition module includes: Perform a binary comparison on the version numbers to determine the oldest full version number of the ETCD, and pull data from the ETCD based on the oldest full version number; Determine the maximum version number in the pulled data, and determine that the maximum version number in the pulled data is the oldest version number of the full amount.

20. The device according to claim 14 or 19, wherein: The first acquisition module includes: Starting from the oldest version number of the full amount, the incremental data of the ETCD is monitored and the incremental data of the ETCD is received; Determine the maximum version number in the incremental data, and determine that the maximum version number in the incremental data is the current latest version number.

21. The device according to any one of claims 14 to 20, wherein: The third acquisition module includes: Input the first Key into a pre-built Key generation function, and generate the second Key through the Key generation function; Determine the format of the second Key or the Key generation function, and determine the type of the second Key.

22. The device according to claim 21, wherein The device also includes: Based on the first Key, generate a first MVCC index structure; For each type of Key, based on the second Key under the type, a second MVCC index structure matching the type is constructed.

23. The device according to claim 22, wherein: The device also includes: Receive a list request from a client, wherein the list request includes a query key; Determine a target MVCC index structure from multiple MVCC index structures according to the query interface corresponding to the List request and the type of the query Key; According to the query key, index the target MVCC index structure to obtain a target version number corresponding to the query key; According to the target version number, the target data is obtained by searching in the B-tree of the memory data storage.

24. The device according to claim 23, wherein: The device comprises: Based on the target version number, determining the B-tree pointer through an atomic operation; The node pointed to by the B-tree pointer is determined, and the target data corresponding to the target version number is determined from the pointed node.

25. The device according to claim 23, wherein: The device comprises: In response to the receiving interface being the original range query interface of the ETCD agent, determining the first MVCC index structure as the target MVCC index structure; In response to the receiving interface being a range query interface newly added to the ETCD agent, the target MVCC index structure is determined from a plurality of second MVCC index structures according to the Key type.

26. The device according to claim 22, wherein: The device also includes: Receive the client's watch request; In response to monitoring the occurrence of an event, determining a first Key and a second Key corresponding to the event; Determine the listeners corresponding to the first Key and the second Key respectively, send the event to the listeners, and push the event to the client through the listeners.

27. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-13.

29. A computer program product comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 13.

Images