Global secondary index updating method, device and equipment based on distributed database

Abstract

The invention provides a global secondary index updating method based on a distributed database. The method can be applied to the technical field of big data and the distributed technical field. The method comprises the steps of obtaining n original data in response to received n original data writing requests of a distributed database, writing the n original data into m fragments of the distributed database, and generating n fragment index logs. And obtaining an index rule of a global secondary index table of the distributed database, and converting the n fragment index logs into n global secondary index update logs according to the index rule of the global secondary index table of the distributed database through a distributed processing framework. And carrying out asynchronous updating on the original global secondary index based on the n global secondary index updating logs to generate a target global secondary index. The invention further provides a global secondary index updating device and equipment based on the distributed database, a storage medium and a program product.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, in particular to the application of distributed technology, and specifically to a global secondary index updating method and device based on a distributed database, equipment, medium and program product. BACKGROUND

[0002] With the development of technological progress, enterprises including banks gradually increase the demand for data query of distributed databases. The traditional data query of distributed databases is generally performed through global secondary index data query. However, the global secondary index data query method on the market causes the spread of shard faults when a single shard of the distributed database fails, which affects the global secondary index of the entire distributed database, causes poor fault tolerance and low reliability of the database index, and results in low utilization of database computing resources. SUMMARY

[0003] In view of the above problems, the present application provides a global secondary index updating method and device based on a distributed database, equipment, medium and program product.

[0004] According to a first aspect of the present application, a global secondary index updating method based on a distributed database is provided. The method comprises: in response to receiving n original data write requests of a distributed database, obtaining the n original data, writing the n original data into m shards of the distributed database, and generating n shard index logs, wherein each shard index log comprises a shard index primary key, an operation type, a first timestamp and a first shard index value, n is an integer and n is greater than 1, m is an integer, m is greater than 0 and m is less than n; obtaining the index rule of the global secondary index table of the distributed database, converting the n shard index logs according to the index rule of the global secondary index table of the distributed database through a distributed processing framework to generate n global secondary index update logs, wherein each global secondary index log comprises a global secondary index primary key, the shard index primary key, the operation type and a second timestamp; and asynchronously updating the original global secondary index based on the n global secondary index update logs to generate a target global secondary index.

[0005] According to an embodiment of the present application, the conversion of the n shard index logs into n global secondary index update logs according to the index rule of the global secondary index table of the distributed database through the distributed processing framework comprises: obtaining the n shard index logs through the distributed processing framework; and converting the shard index value in each shard index log in the n shard index logs into a global secondary index primary key through the index rule of the global secondary index table of the distributed database based on a conversion function to generate the n global secondary index update logs.

[0006] According to an embodiment of the present application, the n shard index logs are converted to generate n global secondary index update logs according to the index rule of the global secondary index table of the distributed database through a distributed processing framework, including: acquiring the n shard index logs through a distributed data stream processing framework, and converting the n shard index logs into time series stream data through the first timestamp; and converting the shard index value in each shard index log in the time series stream data into a global secondary index primary key through the index rule of the global secondary index table of the distributed database through a data stream conversion operator, to generate the n global secondary index update logs.

[0007] According to an embodiment of the present application, the original global secondary index is asynchronously updated based on the n global secondary index update logs to generate a target global secondary index, including: acquiring the second timestamp of the n global secondary index update logs; sorting the n global secondary index update logs based on the second timestamp of the n global secondary index update logs to generate a log update sequence; and asynchronously updating the original global secondary index based on the n global secondary index update logs according to the log update sequence to generate the target global secondary index.

[0008] According to an embodiment of the present application, the original global secondary index is asynchronously updated based on the n global secondary index update logs to generate the target global secondary index, including: sorting the n global secondary index update logs according to the log update sequence based on a distributed consensus algorithm to generate a message queue of the n global secondary index update logs; and sequentially asynchronously updating the original global secondary index based on the message queue of the n global secondary index update logs to generate the target global secondary index.

[0009] According to an embodiment of the present application, the original global secondary index is sequentially asynchronously updated based on the message queue of the n global secondary index update logs to generate the target global secondary index, including: sequentially asynchronously updating the original global secondary index based on the message queue of the n global secondary index update logs in the memory of m shards of the distributed database to generate shard global secondary indexes of the m shards; and performing distributed database persistent storage on the shard global secondary indexes of the m shards to generate the target global secondary index.

[0010] According to an embodiment of the present application, the method further comprises: obtaining a third timestamp of an update start of the asynchronous update on the original global secondary index, a fourth timestamp of an update end of the asynchronous update on the original global secondary index, and a fifth timestamp of the query of the n original data according to the target global secondary index; and performing the target global secondary index query on the n original data according to a ratio of the fifth timestamp to the third timestamp and the fourth timestamp.

[0011] According to a second aspect of the present application, a global secondary index updating device based on a distributed database is provided, the device comprising: a first generation module configured to, in response to receiving an n original data write request of a distributed database, obtain the n original data, write the n original data into m shards of the distributed database, and generate n shard index logs, wherein each shard index log comprises a shard index primary key, an operation type, a first timestamp, and a first shard index value, n is an integer and n is greater than 1, m is an integer, m is greater than 0 and m is less than n; a second generation module configured to obtain an index rule of a global secondary index table of the distributed database, and convert the n shard index logs into n global secondary index update logs according to the index rule of the global secondary index table of the distributed database through a distributed processing framework, wherein each global secondary index log comprises a global secondary index primary key, the shard index primary key, the operation type, and a second timestamp; and a third generation module configured to perform asynchronous update on an original global secondary index based on the n global secondary index update logs, and generate a target global secondary index.

[0012] According to a third aspect of the present application, an electronic device is provided, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0013] A fourth aspect of the present application further provides a computer-readable storage medium having a computer program or instructions stored thereon, wherein the computer program or instructions are executed by a processor to implement the steps of the above method.

[0014] A fifth aspect of the present application further provides a computer program product comprising a computer program or instructions, wherein the computer program or instructions are executed by a processor to implement the steps of the above method.

[0015] The application can effectively improve the data writing performance by generating n piece index logs and asynchronously updating the original global secondary index. And through the technical means of log recording, timestamp and data asynchronous updating mechanism, generating target global secondary index, when a single shard in the distributed database fails, based on asynchronous updating and timestamp processing out-of-order events, the index state can be quickly recovered, and the global secondary index of the entire distributed database affected by the single shard failure is avoided. The technical effects of significantly improving the index fault tolerance and reliability of the database, reducing data delay and improving the utilization rate of database computing resources can be achieved. The technical problems of poor database index fault tolerance, low reliability and low utilization rate of database computing resources caused by the single shard failure affecting the global secondary index of the entire distributed database when a single shard fails in the prior art are solved. BRIEF DESCRIPTION OF DRAWINGS

[0016] The above and other objects, features and advantages of the present application will become more apparent from the following description of embodiments of the present application, taken in conjunction with the accompanying drawings, in which:

[0017] Figure 1 An application scenario diagram of a global secondary index updating method, device, equipment, medium and program product based on a distributed database according to an embodiment of the present application is schematically shown;

[0018] Figure 2 A flowchart of a global secondary index updating method based on a distributed database according to an embodiment of the present application is schematically shown;

[0019] Figure 3 A flowchart of an embodiment of generating n global secondary index updating logs in a global secondary index updating method based on a distributed database according to an embodiment of the present application is schematically shown;

[0020] Figure 4 A flowchart of another embodiment of generating n global secondary index updating logs in a global secondary index updating method based on a distributed database according to an embodiment of the present application is schematically shown;

[0021] Figure 5 A flowchart of generating a target global secondary index by asynchronous updating in a global secondary index updating method based on a distributed database according to an embodiment of the present application is schematically shown;

[0022] Figure 6 A flowchart of generating a target global secondary index based on a distributed consensus algorithm in a global secondary index updating method based on a distributed database according to an embodiment of the present application is schematically shown;

[0023] Figure 7A flow chart of generating a target global secondary index by a distributed database persistent storage in a global secondary index updating method based on a distributed database according to an embodiment of the present application is shown schematically;

[0024] Figure 8 A structure block diagram of a global secondary index updating apparatus based on a distributed database according to an embodiment of the present application is shown schematically;

[0025] Figure 9 A block diagram of an electronic device adapted to implement a global secondary index updating method based on a distributed database according to an embodiment of the present application is shown schematically. DETAILED DESCRIPTION

[0026] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. It is to be understood, however, that the description is merely exemplary of the present application, and is not intended to limit the scope of the present application. In the following detailed description of the embodiments of the present application, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without these specific details. In other instances, well-known structures and functions have not been described in detail in order to avoid obscuring aspects of the present application.

[0027] The terms used herein are merely used to describe specific embodiments, and are not intended to limit the present application. The terms "include" and "have" and the like used herein indicate the presence of the described features, steps, operations, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

[0028] All terms used herein, including technical and scientific terms, have the same meanings as those generally understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having meanings consistent with the context of the present description, and should not be interpreted in an idealized or overly formal way.

[0029] In the case of using expressions similar to "at least one of A, B, and C, etc.", it should generally be interpreted to include at least one of each item enumerated, but not necessarily one of each item enumerated only. In other words, it should be interpreted to mean that it is equivalent to "at least one of A or B; A or C; B or C; A and B; A and C; B and C; or A, B, and C" in a non-exclusive manner.

[0030] Some of the diagrams illustrated in the drawings are block diagrams and / or flowcharts. It will be understood that some of the blocks in the block diagrams and / or flowcharts, or combinations of blocks, can be implemented by computer program instructions. Such computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable control device to produce an apparatus that creates the functions / operations specified in the block diagrams and / or flowcharts.

[0031] First, the technical terms appearing in this paper are explained as follows:

[0032] Global secondary index is a kind of auxiliary data structure in database table, which allows using different keys from the table primary key to efficiently query data. Global secondary index generally creates a new "view" or "mapping" in advance, and the data of this "view" or "mapping" is sorted and partitioned according to the new specified key (generally referred to as index key). When querying with index key, the database can directly search in this smaller and more ordered index, improving the search speed.

[0033] Shard fault diffusion: a shard node failure causes dependent cross-shard transactions to block, leading to global service unavailability.

[0034] Map function is an operation that applies a given transformation function to each element in a collection and returns a new result collection.

[0035] Map operator, similar to Map function, is an operation that applies a given transformation operator to each element in a collection of data streams and returns a new result collection.

[0036] Raft algorithm is a distributed consensus algorithm, which is a consensus algorithm for managing replicated logs. Its core function is to make multiple machines in a distributed system work reliably like a single machine when facing failures.

[0037] Timestamp ordering is a system that assigns a commit timestamp to a transaction when it is submitted, and then commits the transaction in the order of this timestamp.

[0038] Worker cluster is a distributed processing framework composed of a collection of multiple computing nodes (called workers). These nodes work together to process tasks from a message queue, thereby achieving distributed processing of large-scale computing tasks or high-concurrency requests.

[0039] Embodiments of the present application provide a global secondary index updating method based on a distributed database, the method comprising: in response to receiving n original data write requests of the distributed database, obtaining the n original data, writing the n original data into m shards of the distributed database, and generating n shard index logs, wherein each shard index log comprises a shard index primary key, an operation type, a first timestamp, and a first shard index value, n is an integer and n is greater than 1, m is an integer, m is greater than 0 and m is less than n. Obtain the index rule of the global secondary index table of the distributed database, convert the n shard index logs to generate n global secondary index update logs according to the index rule of the global secondary index table of the distributed database through a distributed processing framework, wherein each global secondary index log comprises a global secondary index primary key, the shard index primary key, the operation type, and a second timestamp. And asynchronously updating the original global secondary index based on the n global secondary index update logs to generate a target global secondary index.

[0040] According to embodiments of the present application, by generating n shard index logs and asynchronously updating the original global secondary index, the data write performance can be effectively improved. And through the log recording, timestamp and data asynchronous updating mechanism, the technical means of generating the target global secondary index can process out-of-order events based on asynchronous updating and timestamp when a single shard of the distributed database fails, quickly recover the index state, and avoid the impact of single shard failure on the global secondary index of the entire distributed database. The technical effects of significantly improving the index fault tolerance and reliability of the database, reducing data delay, and improving the utilization rate of database computing resources can be achieved. The technical problems of poor database index fault tolerance, low reliability, and low utilization rate of database computing resources caused by single shard failure affecting the global secondary index of the entire distributed database when a single shard fails in the prior art are solved.

[0041] Figure 1 An application scenario diagram of the global secondary index updating method, apparatus, device, medium and program product based on a distributed database according to embodiments of the present application is schematically shown. It should be noted that, Figure 1 The shown is only an example of a scenario in which embodiments of the present application can be applied, to help those skilled in the art understand the technical content of the present application, but does not mean that embodiments of the present application cannot be used in other devices, systems, environments or scenarios.

[0042] As Figure 1As shown, the application scenario 100 according to this embodiment can include an application scenario of global secondary index updating based on a distributed database. The network 104 is a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 can include various connection types, such as wired, wireless communication links, or optical fiber cables, and the like.

[0043] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, and the like. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, and the like (only as examples).

[0044] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with a display screen and supporting web browsing, including but not limited to smartphones, tablet computers, laptop computers, desktop computers, and the like.

[0045] The server 105 can be a server providing various services, such as a background management server supporting a website browsed by the user using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (only as an example). The background management server can analyze and process received user requests and the like, and feed back the processing results (such as a webpage, information, or data, and the like obtained or generated according to the user request) to the terminal device.

[0046] It should be noted that the global secondary index updating method based on a distributed database provided by the embodiments of the present application can generally be executed by the server 105. Correspondingly, the global secondary index updating device based on a distributed database provided by the embodiments of the present application can generally be arranged in the server 105. The global secondary index updating method based on a distributed database provided by the embodiments of the present application can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105. Correspondingly, the global secondary index updating device based on a distributed database provided by the embodiments of the present application can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105.

[0047] It should be understood that, Figure 1The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0048] The following will be based on Figure 1 The described scene, through Figures 2-7 The disclosed embodiments of the global secondary index update method based on a distributed database are described in detail. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of this application, and the implementation methods of this application are not limited in any way. Rather, the implementation methods of this application can be applied to any applicable scenario.

[0049] Figure 2 A flowchart illustrating a global secondary index update method based on a distributed database according to an embodiment of this application is shown.

[0050] like Figure 2 As shown, the method 200 includes steps S201 to S203.

[0051] Step S201: In response to receiving n raw data write requests from the distributed database, obtain the n raw data, write the n raw data into m shards of the distributed database, and generate n shard index logs. Each shard index log includes: shard index primary key, operation type, first timestamp and first shard index value, where n is an integer and n is greater than 1, m is an integer, m is greater than 0 and m is less than n.

[0052] For example, to obtain n pieces of raw data, write them into m shards of a distributed database, and generate n shard index logs, this process is performed only on the local shard. The primary data within each shard and the local index can be updated using each shard index log. Each local shard can be formatted as [PrimaryKey, OpType, NewValue, Commit-Timestamp], where PrimaryKey is the primary key of the first shard index, OpType is the first data operation type, NewValue is the first shard index value, and Commit-Timestamp is the first timestamp for generating the shard index log.

[0053] Step S202: Obtain the index rules of the global secondary index table of the distributed database, and use the distributed processing framework to convert the n shard index logs into n global secondary index update logs according to the index rules of the global secondary index table of the distributed database. Each global secondary index log includes: global secondary index primary key, the shard index primary key, the operation type, and the second timestamp.

[0054] For example, the format of each global secondary index log can be [GSI_Key, OpType, PrimaryKey, Commit-Timestamp], where PrimaryKey is the primary key of the second shard index, OpType is the second data operation type, GSI_Key is the primary key of the global secondary index, and Commit-Timestamp is the second timestamp that generated the global secondary index update log. The primary key of the second shard index in each global secondary index log is the same as the primary key of the first shard index in its corresponding shard index log, and the second operation type in each global secondary index log is the same as the first operation type in its corresponding shard index log. However, the second timestamp in each global secondary index log is different from the first timestamp of its corresponding shard index log.

[0055] Figure 3 The flowchart illustrates an embodiment of the global secondary index update method based on a distributed database according to an embodiment of this application, which generates n global secondary index update logs.

[0056] like Figure 3 As shown, the method 300 includes steps S301 to S302.

[0057] Step S301: Obtain the n sharded index logs through the distributed processing framework.

[0058] For example, a worker cluster can sequentially pull n sharded index logs from the distributed storage layer.

[0059] Step S302: Based on the conversion function, the shard index value in each of the n shard index logs is converted into a global secondary index primary key through the indexing rules of the global secondary index table of the distributed database, thereby generating the n global secondary index update logs.

[0060] For example, a Map function can be executed on each shard index log record on each compute node. The Map function extracts complete write data from the shard index log, and based on the indexing rules of the distributed database's global secondary index table, it extracts specific key fields from the complete write data or uses these specific key fields to construct the global secondary index primary key, generating the n global secondary index update logs.

[0061] By using transformation functions and the indexing rules of the global secondary index table in the distributed database, the shard index values ​​in each shard index log are converted into global secondary index primary keys, generating n global secondary index update logs. This achieves complete decoupling between the index and the data, significantly improving the reliability and accuracy of generating n global secondary index update logs.

[0062] Figure 4 Fig. 6 schematically shows a flowchart of another embodiment of generating n global secondary index update logs in the method for updating global secondary index based on distributed database according to an embodiment of the present application.

[0063] As shown in Fig. 4, the method 400 comprises steps S401-S403. Figure 4

[0064] In step S401, the n shard index logs are acquired through a distributed data stream processing framework.

[0065] For example, the stream data source of the shard index logs is directly connected through the stream processing framework, so that each shard index log immediately enters the stream processing state.

[0066] In step S402, the n shard index logs are converted into time series stream data through the first timestamp.

[0067] In step S403, the shard index value in each shard index log in the time series stream data is converted into a global secondary index primary key through the index rule of the global secondary index table of the distributed database through a data stream conversion operator, and the n global secondary index update logs are generated.

[0068] For example, the write data of the shard index value in each shard index log in the time series stream data is parsed through a Map operator, and the global secondary index primary key is extracted from the write data according to the index rule of the global secondary index table of the distributed database, so as to generate the global secondary index update log.

[0069] Based on the distributed data stream processing framework, the shard index value in each shard index log in the time series stream data is converted into a global secondary index primary key through the index rule of the global secondary index table of the distributed database through a data stream conversion operator, and the n global secondary index update logs are generated, which can guarantee the time processing time series and the state consistency, and significantly improve the reliability and accuracy of generating the n global secondary index update logs through the ordered time series stream data processing.

[0070] Referring back to Fig. 2, Figure 2 In step S203, the original global secondary index is asynchronously updated based on the n global secondary index update logs, and the target global secondary index is generated.

[0071] Figure 5 Fig. 5 schematically shows a flowchart of generating a target global secondary index through asynchronous update in the method for updating global secondary index based on distributed database according to an embodiment of the present application.

[0072] As shown in Fig. 4, the method 400 comprises steps S401-S403. Figure 5 ​As shown, the method 500 includes steps S501-S503.

[0073] Step S501, acquiring the second time stamp of the n global secondary index update logs.

[0074] Step S502, sorting the n global secondary index update logs based on the second time stamp of the n global secondary index update logs to generate a log update sequence.

[0075] For example, the n global secondary index update logs can be sorted based on the second time stamp of the n global secondary index update logs to generate a log update sequence by time stamp sorting.

[0076] Step S503, updating the original global secondary index based on the n global secondary index update logs according to the log update sequence to generate the target global secondary index.

[0077] Figure 6 As shown in the flowchart, the method 700 includes steps S701-S702.

[0078] As shown, the method 600 includes steps S601-S602. Figure 6

[0079] Step S601, sorting the n global secondary index update logs according to the log update sequence based on a distributed consensus algorithm to generate a message queue of the n global secondary index update logs.

[0080] For example, the n global secondary index update logs can be sorted according to the log update sequence based on a Raft algorithm to generate a message queue of the n global secondary index update logs.

[0081] Step S602, sequentially updating the original global secondary index based on the message queue of the n global secondary index update logs to generate the target global secondary index.

[0082] Figure 7 As shown in the flowchart, the method 700 includes steps S701-S702.

[0083] As shown, the method 700 includes steps S701-S702. Figure 7

[0084] ​​Step S701, according to the message queue of the n global secondary index update logs in the memory of the m shards of the distributed database, sequentially updating the original global secondary index asynchronously to generate the shard global secondary indexes of the m shards.

[0085] Step S702, performing distributed database persistent storage on the shard global secondary indexes of the m shards to generate the target global secondary index.

[0086] By updating the index in the shard memory, the efficiency of index update can be greatly improved, and computer resources can be saved. At the same time, combined with the parallel batch persistent storage of the shards, the target global secondary index with high performance, high throughput and low delay can be generated, and the update efficiency of updating the global secondary index and generating the target global secondary index can be effectively improved. By generating the message queue of the n global secondary index update logs through the distributed consensus algorithm, sequentially updating the original global secondary index asynchronously, the correctness and reliability of generating the target global secondary index can be guaranteed, and the log sequence of updating the original global secondary index asynchronously can be guaranteed, which can effectively avoid the influence of single-point shard failure on global shard index. By sorting the logs according to the second timestamp and updating asynchronously according to the log sequence to generate the target global secondary index, the high performance and robustness of data update can be improved through asynchronous update, and the reliability and correctness of generating the target global secondary index can be guaranteed.

[0087] In addition, the method further comprises: obtaining a third timestamp of the start of updating the original global secondary index asynchronously, a fourth timestamp of the end of updating the original global secondary index asynchronously, and a fifth timestamp of querying the n original data according to the target global secondary index. And according to the ratio of the fifth timestamp to the third timestamp and the fourth timestamp, the n original data is queried according to the target global secondary index.

[0088] For example, when the fifth timestamp is less than the third timestamp, it means that the query occurs before the index update starts, and a "past" state is queried. At this time, the target global secondary index may not yet contain the data that the user wants. At this time, the system will fall back to query the original data table or other ways, rather than using the target global secondary index that may be too old, thereby avoiding reading dirty data. When the third timestamp is less than or equal to the fifth timestamp, and the fifth timestamp is less than or equal to the fourth timestamp, it means that the query occurs during the index update process. At this time, the target global secondary index is in an inconsistent or incomplete state. The system will choose to wait for the update to complete or fall back to the original data query again to ensure the correctness of the result. When the fifth timestamp is greater than the fourth timestamp, it means that the query occurs after the index update is completed. At this time, the target global secondary index is the latest and complete, and the query can be safely and efficiently performed through the target global secondary index to improve the query efficiency.

[0089] By querying the time stamp and the time stamp before and after the update, the update state of the global secondary index can be determined, the accuracy and correctness of querying the n original data through the target global secondary index can be improved, the efficiency of querying the target global secondary index can be significantly improved, and the computer resources can be effectively saved.

[0090] Figure 8 An apparatus for updating a global secondary index based on a distributed database according to an embodiment of the present application is shown in a structural block diagram.

[0091] As shown in Figure 8 the apparatus 800 includes a first generation module 801, a second generation module 802, and a third generation module 803.

[0092] The first generation module 801 is configured to, in response to receiving an n original data write request of a distributed database, acquire the n original data, write the n original data into m shards of the distributed database, and generate n shard index logs, wherein each shard index log includes a shard index primary key, an operation type, a first timestamp, and a first shard index value, n is an integer and n is greater than 1, m is an integer, m is greater than 0 and m is less than n. In an embodiment, the first generation module 801 can be configured to perform the step S201 described above, and details are not repeated here.

[0093] The second generation module 802 is configured to acquire an index rule of a global secondary index table of the distributed database, and convert the n shard index logs into n global secondary index update logs according to the index rule of the global secondary index table of the distributed database through a distributed processing framework, where each global secondary index log comprises a global secondary index primary key, the shard index primary key, the operation type, and the second timestamp. In an embodiment, the second generation module 802 can be configured to perform the step S202 described above.

[0094] The second generation module 802 comprises a first acquisition module and a fourth generation module.

[0095] The first acquisition module is configured to acquire the n shard index logs through a distributed processing framework. In an embodiment, the first acquisition module can be configured to perform the step S301 described above, which will not be repeated here.

[0096] The fourth generation module is configured to convert a shard index value in each of the n shard index logs into a global secondary index primary key according to the index rule of the global secondary index table of the distributed database through a conversion function, and generate the n global secondary index update logs. In an embodiment, the fourth generation module can be configured to perform the step S302 described above, which will not be repeated here.

[0097] The second generation module 802 further comprises a second acquisition module, a fifth generation module, and a sixth generation module.

[0098] The second acquisition module is configured to acquire the n shard index logs through a distributed data stream processing framework. In an embodiment, the second acquisition module can be configured to perform the step S401 described above, which will not be repeated here.

[0099] The fifth generation module is configured to convert the n shard index logs into time series stream data through the first timestamp. In an embodiment, the fifth generation module can be configured to perform the step S402 described above, which will not be repeated here.

[0100] The sixth generation module is configured to convert a shard index value in each of the time series stream data into a global secondary index primary key according to the index rule of the global secondary index table of the distributed database through a data stream conversion operator, and generate the n global secondary index update logs. In an embodiment, the sixth generation module can be configured to perform the step S403 described above, which will not be repeated here.

[0101] The third generation module 803 is configured to update the original global secondary index asynchronously based on the n global secondary index update logs to generate a target global secondary index. In an embodiment, the third generation module 803 can be configured to perform the step S203 described above.

[0102] The third generation module 803 includes a third acquisition module, a seventh generation module, and an eighth generation module.

[0103] The third acquisition module is configured to acquire the second time stamp of the n global secondary index update logs. In an embodiment, the third acquisition module can be configured to perform the step S501 described above, which will not be repeated here.

[0104] The seventh generation module is configured to sort the n global secondary index update logs based on the second time stamp of the n global secondary index update logs to generate a log update sequence. In an embodiment, the seventh generation module can be configured to perform the step S502 described above, which will not be repeated here.

[0105] The eighth generation module is configured to update the original global secondary index asynchronously based on the n global secondary index update logs according to the log update sequence to generate the target global secondary index. In an embodiment, the eighth generation module can be configured to perform the step S503 described above.

[0106] The eighth generation module includes a ninth generation module and a tenth generation module.

[0107] The ninth generation module is configured to sort the n global secondary index update logs according to the log update sequence based on a distributed consensus algorithm to generate a message queue of the n global secondary index update logs. In an embodiment, the ninth generation module can be configured to perform the step S601 described above, which will not be repeated here.

[0108] The tenth generation module is configured to update the original global secondary index sequentially and asynchronously based on the message queue of the n global secondary index update logs to generate the target global secondary index. In an embodiment, the tenth generation module can be configured to perform the step S602 described above.

[0109] The tenth generation module includes an eleventh generation module and a twelfth generation module.

[0110] The eleventh generation module is configured to update the original global secondary index sequentially and asynchronously in the memory of the m shards of the distributed database based on the message queue of the n global secondary index update logs to generate a shard global secondary index of the m shards. In an embodiment, the eleventh generation module can be configured to perform the step S701 described above, which will not be repeated here.

[0111] twelfth generating module, configured to perform distributed database persistent storage on the global secondary index of the m shards, to generate the target global secondary index. In an embodiment, the twelfth generating module can be configured to perform the step S702 described above, and details are not repeated here.

[0112] The apparatus further includes a fourth obtaining module and a querying module.

[0113] The fourth obtaining module is configured to obtain a third timestamp of an update start of the asynchronous update on the original global secondary index, a fourth timestamp of an update end of the asynchronous update on the original global secondary index, and a fifth timestamp of querying the n original data according to the target global secondary index.

[0114] The querying module is configured to perform the target global secondary index query on the n original data according to a ratio of the fifth timestamp to the third timestamp and the fourth timestamp.

[0115] According to embodiments of the present application, any multiple modules of the first generating module 801, the second generating module 802 and the third generating module 803 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of the modules can be combined with at least part of the functions of other modules, and implemented in one module. According to embodiments of the present application, at least one of the first generating module 801, the second generating module 802 and the third generating module 803 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable manner of hardware or firmware that can be integrated or packaged, or implemented in any one of software, hardware and firmware or in a proper combination of any of them. Alternatively, at least one of the first generating module 801, the second generating module 802 and the third generating module 803 can be at least partially implemented as a computer program module that can perform corresponding functions when the computer program module is run.

[0116] Figure 9 A block diagram of an electronic device suitable for implementing the global secondary index update method based on distributed database according to embodiments of the present application is schematically shown.

[0117] As Figure 9As shown, the electronic device 900 according to an embodiment of the present application includes a processor 901 which can perform various appropriate actions and processes according to a program stored in a read only memory (ROM) 902 or a program loaded from a storage section 908 into a random access memory (RAM) 903. The processor 901 can include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chipset, and / or a special purpose microprocessor (e.g., an application specific integrated circuit (ASIC)), and so on. The processor 901 can also include an on-board memory for cache use. The processor 901 can include a single processing unit or multiple processing units for executing different actions of the method processes according to embodiments of the present application.

[0118] In the RAM 903, various programs and data required for the operation of the electronic device 900 are stored. The processor 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. The processor 901 performs various operations of the method processes according to embodiments of the present application by executing the programs in the ROM 902 and / or the RAM 903. Note that the programs can also be stored in one or more memories other than the ROM 902 and the RAM 903. The processor 901 can also perform various operations of the method processes according to embodiments of the present application by executing the programs stored in the one or more memories.

[0119] According to embodiments of the present application, the electronic device 900 can also include an input / output (I / O) interface 905 which is also connected to the bus 904. The electronic device 900 can also include one or more of the following components connected to the input / output (I / O) interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as necessary. A removable medium 911 such as a magnetic disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 910 as necessary, so that a computer program read out from the removable medium 911 is installed in the storage section 908 as necessary.

[0120] The application further provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments, or can exist independently without being assembled into the device / apparatus / system. The computer readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the application.

[0121] According to the embodiments of the application, the computer readable storage medium can be a non-volatile computer readable storage medium, which can include, but is not limited to, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In this application, a computer readable storage medium can be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to the embodiments of the application, the computer readable storage medium can include one or more of the above-described ROM 902 and / or RAM 903 and / or one or more memories other than the ROM 902 and the RAM 903.

[0122] The embodiments of the application also include a computer program product, which includes a computer program containing program codes for executing the method shown in the flow chart. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the distributed database-based global secondary index updating method provided by the embodiments of the application.

[0123] The above-described functions defined in the system / apparatus of the embodiments of the application are performed when the computer program is executed by the processor 901. According to the embodiments of the application, the above-described system, apparatus, module, unit, etc. can be implemented by computer program modules.

[0124] In one embodiment, the computer program can rely on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program can also be transmitted, distributed, and downloaded in the form of a signal on a network medium, and be downloaded and installed through the communication part 909, and / or installed from the detachable medium 911. The program codes contained in the computer program can be transmitted by any appropriate network medium, including but not limited to wireless, wired, etc., or any appropriate combination thereof.

[0125] In such embodiments, the computer program can be downloaded and installed from the network via the communication section 909, and / or installed from the removable media 911. When the computer program is executed by the processor 901, the above-described functions defined in the system of the embodiments of the present application are performed. According to the embodiments of the present application, the system, device, apparatus, module, unit, and the like described above can be implemented by the computer program modules.

[0126] According to the embodiments of the present application, the program code for executing the computer program provided by the embodiments of the present application can be written in any combination of one or more programming languages, and specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming language, and / or assembly / machine language. The programming language includes, but is not limited to, such as Java, C++, python, "C" language, or similar programming language. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case involving a remote computing device, the remote computing device can be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, connected to the Internet through an Internet service provider).

[0127] The flowcharts and block diagrams in the drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment, or a portion of code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order than that shown in the figures. For example, two blocks noted in succession can actually be executed substantially concurrently, or they can sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each block in the flowcharts or block diagrams, and combinations of blocks in the flowcharts or block diagrams, can be implemented by dedicated hardware-based systems that perform the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0128] Those skilled in the art can understand that the features described in various embodiments of the present application can be combined and / or integrated in various combinations, even if such combinations are not explicitly described in the present application. In particular, the features described in various embodiments of the present application can be combined and / or integrated in various combinations without departing from the spirit and teachings of the present application. All such combinations and / or integrations are within the scope of the present application.

Claims

1. A method for updating a global secondary index based on a distributed database, characterized in that, The method includes: In response to receiving n raw data write requests from a distributed database, the n raw data are obtained, and the n raw data are written into m shards of the distributed database to generate n shard index logs. Each shard index log includes: shard index primary key, operation type, first timestamp and first shard index value, where n is an integer and n is greater than 1, m is an integer, m is greater than 0 and m is less than n. The indexing rules of the global secondary index table of the distributed database are obtained. Using a distributed processing framework, the n shard index logs are transformed into n global secondary index update logs according to the indexing rules of the global secondary index table of the distributed database. Each global secondary index log includes: the global secondary index primary key, the shard index primary key, the operation type, and the second timestamp; and The original global secondary index is asynchronously updated based on the update logs of the n global secondary indexes to generate the target global secondary index.

2. The method according to claim 1, characterized in that, The distributed processing framework transforms the n sharded index logs into n global secondary index update logs based on the indexing rules of the global secondary index table of the distributed database, including: The n sharded index logs are obtained through a distributed processing framework; and Based on the transformation function, the shard index value in each of the n shard index logs is converted into a global secondary index primary key according to the indexing rules of the global secondary index table of the distributed database, thereby generating the n global secondary index update logs.

3. The method according to claim 1, characterized in that, The distributed processing framework transforms the n sharded index logs into n global secondary index update logs based on the indexing rules of the global secondary index table of the distributed database, including: The n sharded index logs are obtained through a distributed data stream processing framework. The n sharded index logs are converted into time-series stream data using the first timestamp; and The data stream transformation operator converts the shard index value in each shard index log of the time-series stream data into a global secondary index primary key according to the index rules of the global secondary index table of the distributed database, thereby generating the n global secondary index update logs.

4. The method according to claim 1, characterized in that, Based on the update logs of the n global secondary indexes, the original global secondary indexes are asynchronously updated to generate the target global secondary index, including: Obtain the second timestamp of the update logs of the n global secondary indexes; Based on the second timestamp of the n global secondary index update logs, the n global secondary index update logs are sorted to generate a log update sequence; and Based on the log update sequence, the original global secondary index is asynchronously updated using the n global secondary index update logs to generate the target global secondary index.

5. The method according to claim 4, characterized in that, The original global secondary index is asynchronously updated based on the update logs of the n global secondary indexes to generate the target global secondary index, including: Based on a distributed consensus algorithm, the n global secondary index update logs are sorted according to the log update sequence to generate a message queue of n global secondary index update logs; and Based on the message queues of the n global secondary index update logs, the original global secondary index is asynchronously updated sequentially to generate the target global secondary index.

6. The method according to claim 5, characterized in that, Based on the message queues of the n global secondary index update logs, the original global secondary index is asynchronously updated sequentially to generate the target global secondary index, including: In the memory of m shards of the distributed database, based on the message queues of n global secondary index update logs, the original global secondary index is asynchronously updated sequentially to generate sharded global secondary indexes for m shards; and The global secondary indexes of the m shards are persistently stored in a distributed database to generate the target global secondary index.

7. The method according to any one of claims 1 to 6, further comprising: Obtain the third timestamp of the start of the asynchronous update of the original global secondary index, the fourth timestamp of the end of the asynchronous update of the original global secondary index, and the fifth timestamp of querying the n original data based on the target global secondary index; as well as Based on the ratio of the fifth timestamp to the third and fourth timestamps, a target global secondary index query is performed on the n original data.

8. A global secondary index update device based on a distributed database, characterized in that, The device includes: The first generation module is used to respond to receiving n raw data write requests from the distributed database, obtain the n raw data, write the n raw data into m shards of the distributed database, and generate n shard index logs. Each shard index log includes: shard index primary key, operation type, first timestamp and first shard index value, where n is an integer and n is greater than 1, m is an integer, m is greater than 0 and m is less than n. The second generation module is used to obtain the indexing rules of the global secondary index table of the distributed database, and to convert the n shard index logs into n global secondary index update logs according to the indexing rules of the global secondary index table of the distributed database through a distributed processing framework. Each global secondary index log includes: the global secondary index primary key, the shard index primary key, the operation type, and the second timestamp; and The third generation module is used to asynchronously update the original global secondary index based on the update logs of the n global secondary indexes to generate the target global secondary index.

9. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images