A data reading and writing method and device based on distributed consistency protocol

Abstract

The present invention discloses a data reading and writing method and device based on a distributed consistency protocol. The method obtains data to be written through multiple access modes, parses the written data through a computing engine, assigns an identifier to the parsed data, stores the identified written data in a corresponding service node through a distributed storage engine, returns the data version number recorded by the updated service node to the client, establishes a data index set, and synchronously stores the data index set in all service nodes. When a data read request is made, the data version number carried in the read request is compared with the data version number in the data index set, and responds to the data read request according to the comparison result. The data reading operation can be completed without half of the service nodes reaching a consensus, thereby overcoming the problem of high data reading overhead caused by the need for more than half of the service nodes to reach a consensus when reading data based on a distributed consistency protocol in the prior art.

Description

Technical Field

[0001] The present invention relates to the technical field of data reading and writing, and in particular to a data reading and writing method and device based on a distributed consistency protocol. Background Art

[0002] Graph database is a data management system that uses nodes and edges as storage units and is designed to efficiently store and query graph data. The concept of graph is crucial to understanding graph database. A graph is a set of nodes and edges, where nodes represent entities and edges represent relationships between entities. In a graph database, the relationships between data are as important as the data itself, and they are stored as part of the data. This architecture enables graph databases to quickly respond to complex related queries because the relationships between entities have been stored in the database in advance. Graph databases can visualize relationships intuitively and are the best way to store, query, and analyze highly interconnected data. Since 2017, in order to meet the query return efficiency under large data volumes and provide real-time business intelligence in a rapidly changing business environment, compared with previous generations of products, the third-generation graph database is designed for real-time update and query of graph data. It not only improves scalability in storage, but also increases parallel computing capabilities, enabling real-time graph analysis. Large-scale native graph storage and distributed parallel computing capabilities are gradually becoming the mainstream in the graph database industry.

[0003] A complete distributed system is formed by many service nodes in different locations connected together through the network, and a large amount of data is distributed in different service nodes of the entire network system. All clients connected to the distributed system can access the data in any service node. The existing data reading method based on the distributed consensus protocol requires that each data reading operation must follow the distributed consensus protocol, that is, the service node that receives the data reading request sends the request to other service nodes in the distributed system through the network. After more than half of the service nodes in the distributed system confirm the data corresponding to the data reading request, the data will be returned to the client, and it also needs to be recorded in the service node log, which leads to too much overhead in reading data and affects the reading performance.

[0004] Therefore, it is necessary to provide a data reading and writing method and device based on a distributed consistency protocol to solve the above technical problems. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a data reading and writing method and device based on a distributed consistency protocol.

[0006] The present invention provides a data reading and writing method based on a distributed consistency protocol, comprising:

[0007] Get data: Get the data to be written through a variety of access methods;

[0008] Data identification: parsing the written data through a computing engine, and assigning an identifier to the parsed data;

[0009] Data writing: When any service node receives a data writing request sent by a client, it executes the data writing operation, stores the marked written data in the corresponding service node through the distributed storage engine, updates the data version number recorded by the service node, and returns the updated data version number recorded by the service node to the client;

[0010] Establishing indexes: Mapping the parsed data, identifiers, and data version numbers, establishing a data index set based on the mapping, and synchronously storing the data index set in all service nodes;

[0011] Data reading: When any service node receives a data reading request sent by a client carrying a data version number, it compares the data version number carried in the reading request with the data version number in the data index set and responds to the data reading request based on the comparison result.

[0012] Preferably, the specific process of responding to the data reading request according to the comparison result is:

[0013] If the data version number carried in the read request is equal to the data version number in the data index set, the service node data storing the latest data version number is returned to the client through the index system;

[0014] If the data version number carried in the read request is lower than the data version number in the data index set, determine whether the data read request is a strongly consistent data read request, and respond to the data read request based on the determination result.

[0015] Preferably, the specific process of responding to the data read request according to the judgment result is:

[0016] If not, the data stored in the service node is returned to the client.

[0017] If so, the data with the latest data version number is returned to the client through the index system, and the data version number of the client is updated to the data version number in the data index set.

[0018] Preferably, the specific steps of parsing the written data by the computing engine and assigning the parsed data to an identifier are:

[0019] Parse the written data, divide the data into NDS tiles according to the latitude and longitude of each element, and obtain the tile number to which it belongs;

[0020] The tile number is hashed, and then the acquisition time and acquisition device number are concatenated to form an identifier;

[0021] After the written data is structured, the data is stored in the graph database using the identifier as the row key value of the graph database.

[0022] Preferably, data synchronization: comparing the data version number in the data index set with the data version number of the service node according to set conditions, and responding to the data synchronization request according to the comparison result.

[0023] Preferably, the steps of data synchronization are:

[0024] Send data synchronization requests to all service nodes through the storage engine;

[0025] Compare the data version number in the data index set with the data version number of the service node. If the data version number of the service node is lower than the data version number in the data index set, synchronize the data in the service node with the higher data version number to the service node with the lower data version number, and synchronize the data version number of the service node with the lower data version number to the higher data version number.

[0026] If the data version number of the service node is equal to the data version number in the data index set, the data will not be synchronized and remain as it is.

[0027] A data reading and writing device based on a distributed consistency protocol is applied in a distributed system including multiple service nodes, and the device includes:

[0028] The acquisition module can obtain data through various access methods;

[0029] The identification module can parse the data to be written and assign the parsed data to an identifier;

[0030] A writing module, adapted to execute a data writing operation and update a data version number recorded by the service node when any service node receives a data writing request sent by a client, and return the updated data version number to the client;

[0031] The index module can map the parsed data, identifiers and data version numbers, and establish a data index set based on the mapping.

[0032] A comparison module, adapted to compare the data version number sent by the client with the data version number recorded by the service node when any service node receives a data read request carrying a data version number sent by the client;

[0033] The response module is adapted to respond to the data read request according to the comparison result.

[0034] Preferably, a synchronization module is also included, which can regularly update the data of all service nodes to the latest data.

[0035] Compared with the related art, the data reading and writing method and device based on the distributed consistency protocol provided by the present invention have the following beneficial effects:

[0036] 1. According to the technical solution of the present invention, each time a data write operation is performed, the data version number needs to be updated, and the updated data version number is written into the data index set, and the data index set is synchronously updated, so as to facilitate the determination of whether the data known by the client is the latest data according to the data version number when reading data, and synchronously update the data version number of the data index set. When any service node receives a data read request carrying a data version number sent by the client, the data version number carried in the read request is compared with the data version number in the data index set, and the data read request is responded to according to the comparison result. The data reading operation can be completed without half of the service nodes reaching a consensus, which overcomes the problem of high data reading overhead caused by the need for more than half of the service nodes to reach a consensus when reading data based on a distributed consistency protocol in the prior art;

[0037] 2. The present invention can compare the data version number in the data index set with the data version number of the service node according to the set conditions, respond to the data synchronization request according to the comparison result, and synchronize the data regularly to prevent the failure to save the stored data when the service node has problems. In addition, by synchronizing the data to the service node, the efficiency of data reading can be improved; BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of a data reading and writing method based on a distributed consistency protocol provided by the present invention;

[0039] Figure 2 for Figure 1 The schematic diagram of the specific process of responding to a data read request according to the comparison result is shown;

[0040] Figure 3 for Figure 1 The data synchronization process diagram shown;

[0041] Figure 4 for Figure 1 A schematic diagram of the structure of a data reading and writing device based on a distributed consistency protocol is shown; DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0043] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0045] In the description of the embodiments of the present invention, it should be noted that if the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the drawings, or the orientation or position relationship in which the product of the invention is usually placed when in use. It is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, if the terms "first", "second", "third", etc. appear, they are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0046] In addition, the terms "horizontal", "vertical", "overhanging" and the like do not mean that the components must be absolutely horizontal or overhanging, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0047] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0048] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0049] The present invention provides a data reading and writing method based on a distributed consistency protocol.

[0050] S1. Obtain data: obtain the data to be written through various access methods;

[0051] S2, data identification: parsing the written data through a computing engine, and assigning an identifier to the parsed data;

[0052] S3, data writing: When any service node receives a data writing request sent by a client, it executes a data writing operation, stores the marked written data in the corresponding service node through the distributed storage engine, updates the data version number recorded by the service node, and returns the updated data version number recorded by the service node to the client;

[0053] S4. Create index: map the parsed data, identifier and data version number accordingly, create a data index set according to the mapping, and synchronously store the data index set in all service nodes;

[0054] S5. Data reading: When any service node receives a data reading request carrying a data version number sent by a client, it compares the data version number carried in the reading request with the data version number in the data index set, and responds to the data reading request based on the comparison result.

[0055] It should be noted that: the data access method includes but is not limited to the connection of database, streaming data, and API, and the data is collected incrementally or in full. The data version number is used to indicate the current update status of the data. The higher the data version number, the newer the data. Each time a data write operation is performed, the data version number needs to be updated, and the updated data version number is written to the data index set, and the data index set is synchronously updated to facilitate the determination of whether the data obtained by the client is the latest data based on the data version number when reading the data; moreover, in the embodiment of the present invention, regardless of the amount of data written, as long as the data write operation is performed, the data version number recorded by the service node needs to be updated, that is, even if only one piece of data is written, the data version number needs to be updated after the execution of this data write operation or before the data write operation is performed, and the data version number of the data index set needs to be synchronously updated.

[0056] In an embodiment of the present invention, the specific process of responding to a data read request according to the comparison result is as follows:

[0057] S21. If the data version number carried in the read request is equal to the data version number in the data index set, the service node data storing the latest data version number is returned to the client through the index system;

[0058] S22: if the data version number carried by the read request is lower than the data version number in the data index set, determine whether the data read request is a strongly consistent data read request, and respond to the data read request according to the determination result.

[0059] S221. If not, return the data stored in the service node to the client.

[0060] S222: If yes, return the data of the latest data version number to the client through the index system, and update the data version number of the client to the data version number in the data index set.

[0061] In an embodiment of the present invention, the specific steps of parsing the written data through a computing engine and assigning an identifier to the parsed data are:

[0062] S31, parsing the written data, dividing the data into NDS tiles according to the latitude and longitude of each element, and obtaining the corresponding tile number;

[0063] S32, hashing the tile number, and then concatenating the acquisition time and the acquisition device number to construct an identifier;

[0064] S33. After the written data is structured, the data is stored in the graph database using the identifier as the row key value of the graph database.

[0065] It should be noted that a tile refers to a square grid image that is cut into several rows and columns of a map within a certain range according to a certain size and format, according to a zoom level or scale. The sliced ​​square grid image is figuratively called a tile.

[0066] In an embodiment of the present invention, data synchronization is also included: comparing the data version number in the data index set with the data version number of the service node according to set conditions, and responding to the data synchronization request according to the comparison result.

[0067] The specific steps are:

[0068] S41, sending a data synchronization request to all service nodes through the storage engine;

[0069] S42: Compare the data version number in the data index set with the data version number of the service node. If the data version number of the service node is lower than the data version number in the data index set, synchronize the data in the service node with the higher data version number to the service node with the lower data version number, and synchronize and update the data version number of the service node with the lower data version number to the higher data version number.

[0070] S43. If the data version number of the service node is equal to the data version number in the data index set, the data is not synchronized and remains as it is.

[0071] A data reading and writing device based on a distributed consistency protocol is applied in a distributed system including multiple service nodes, and the device includes:

[0072] The acquisition module can obtain data through various access methods;

[0073] The identification module can parse the data to be written and assign the parsed data to an identifier;

[0074] A writing module, adapted to execute a data writing operation and update a data version number recorded by the service node when any service node receives a data writing request sent by a client, and return the updated data version number to the client;

[0075] The index module can map the parsed data, identifiers and data version numbers, and establish a data index set based on the mapping.

[0076] A comparison module, adapted to compare the data version number sent by the client with the data version number recorded by the service node when any service node receives a data read request carrying a data version number sent by the client;

[0077] The reading module is adapted to respond to a data reading request according to the comparison result.

[0078] Preferably, a synchronization module is also included, which can regularly update the data of all service nodes to the latest data.

[0079] The working principle of a data reading and writing method and device based on a distributed consistency protocol provided by the present invention is as follows:

[0080] When any service node receives a data read request carrying a data version number sent by a client, it compares the data version number carried in the read request with the data version number in the data index set, and responds to the data read request based on the comparison result. The data reading job can be completed without the need for half of the service nodes to reach a consensus, which overcomes the problem of high data reading overhead caused by the need for more than half of the service nodes to reach a consensus when reading data based on a distributed consistency protocol in the prior art.

[0081] The circuits and controls involved in the present invention are all prior art and will not be described in detail here.

[0082] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A data reading and writing method based on a distributed consistency protocol, characterized in that: include: Get data: Get the data to be written through a variety of access methods; Data identification: The data to be written is parsed through the computing engine, and the parsed data is assigned an identifier; Data writing: When any service node receives a data writing request sent by a client, it executes the data writing operation, stores the marked written data in the corresponding service node through the distributed storage engine, updates the data version number recorded by the service node, and returns the updated data version number recorded by the service node to the client; Establishing indexes: Mapping the parsed data, identifiers, and data version numbers, establishing a data index set based on the mapping, and synchronously storing the data index set in all service nodes; Data reading: When any service node receives a data reading request with a data version number from a client, it compares the data version number carried in the reading request with the data version number in the data index set, and responds to the data reading request based on the comparison result. The specific process of responding to data read requests based on the comparison results is as follows: If the data version number carried in the read request is equal to the data version number in the data index set, the service node data storing the latest data version number is returned to the client through the index system; If the data version number carried by the read request is lower than the data version number in the data index set, determine whether the data read request is a strongly consistent data read request, and respond to the data read request according to the determination result. The specific process of responding to the data read request according to the determination result is as follows: If not, the data stored in the service node is returned to the client. If so, the data with the latest data version number is returned to the client through the index system, and the data version number of the client is updated to the data version number in the data index set; It also includes data synchronization: comparing the data version number in the data index set with the data version number of the service node according to the set conditions, and responding to the data synchronization request according to the comparison result; The steps of data synchronization are: Send data synchronization requests to all service nodes through the storage engine; Compare the data version number in the data index set with the data version number of the service node. If the data version number of the service node is lower than the data version number in the data index set, synchronize the data in the service node with the higher data version number to the service node with the lower data version number, and synchronize the data version number of the service node with the lower data version number to the higher data version number. If the data version number of the service node is equal to the data version number in the data index set, the data will not be synchronized and remain as it is.

2. According to the data reading and writing method based on the distributed consistency protocol according to claim 1, it is characterized in that: The specific steps of parsing the written data through the computing engine and assigning the parsed data to the identifier are: Parse the written data, divide the data into NDS tiles according to the latitude and longitude of each element, and obtain the tile number to which it belongs; The tile number is hashed, and then the acquisition time and acquisition device number are concatenated to form an identifier; After the written data is structured, the data is stored in the graph database using the identifier as the row key value of the graph database.

3. A data reading and writing device based on a distributed consistency protocol, characterized in that: Including acquisition module, which can obtain data through various access methods; The identification module can parse the data to be written and assign the parsed data to an identifier; A writing module, adapted to execute a data writing operation and update a data version number recorded by the service node when any service node receives a data writing request sent by a client, and return the updated data version number to the client; The index module can map the parsed data, identifiers and data version numbers, and establish a data index set based on the mapping; The comparison module is adapted to compare the data version number sent by the client with the data version number recorded by the service node when any service node receives a data read request carrying the data version number sent by the client; the response module is adapted to respond to the data read request according to the comparison result; wherein, The specific process of the comparison module responding to the data reading request according to the comparison result is as follows: If the data version number carried in the read request is equal to the data version number in the data index set, the service node data storing the latest data version number is returned to the client through the index system; If the data version number carried by the read request is lower than the data version number in the data index set, determine whether the data read request is a strongly consistent data read request, and respond to the data read request according to the determination result. The specific process of responding to the data read request according to the determination result is as follows: If not, the data stored in the service node is returned to the client. If so, the data with the latest data version number is returned to the client through the index system, and the data version number of the client is updated to the data version number in the data index set; It also includes a data synchronization module: used to compare the data version number in the data index set with the data version number of the service node according to the set conditions, and respond to the data synchronization request according to the comparison result; The steps of performing data synchronization by the data synchronization module are as follows: Send data synchronization requests to all service nodes through the storage engine; Compare the data version number in the data index set with the data version number of the service node. If the data version number of the service node is lower than the data version number in the data index set, synchronize the data in the service node with the higher data version number to the service node with the lower data version number, and synchronize the data version number of the service node with the lower data version number to the higher data version number. If the data version number of the service node is equal to the data version number in the data index set, the data will not be synchronized and remain as it is.

4. According to claim 3, a data reading and writing device based on a distributed consistency protocol is characterized in that: It also includes a synchronization module that can regularly update the data of all service nodes to the latest data.

Images