A database access method and apparatus

Abstract

The application provides a database access method and device, relates to the technical field of databases, and can also be used in the financial field.The database comprises a distributed cache database and a distributed storage database.The method comprises the following steps: analyzing each data read request received from a client to obtain a corresponding read request type;the read request type comprises a hot table read request and a non-hot table read request;obtaining a corresponding hot table data processing result and a non-hot table data processing result from a storage area corresponding to each read request type; aggregating the hot table data processing result and the non-hot table data processing result according to a data structure mapping relationship between the distributed cache database and the distributed storage database, and sending the aggregation result to the client.The application can fuse the distributed cache database and the distributed storage database, and process the access request of the client in the distributed cache database and the distributed storage database.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of database, and can be used in the financial field, in particular to a database access method and device. BACKGROUND

[0002] When a business system of a financial institution processes a financial service, it often needs to read and write interactively with a database server of the financial institution. The database server of the financial institution includes a distributed cache database and a distributed storage database. With the increasing of the concurrent access volume of financial services, in order to reduce the access pressure on the distributed storage database and improve the response ability of data access, the financial institution often needs to cache hot data with more read and less write and more frequent access to the distributed cache database according to the business data characteristics of itself.

[0003] Because of the great difference between the data read and write modes of the distributed cache database and the distributed storage database, the industry currently sets the distributed cache database and the distributed storage database as two independent data access systems. When the business level needs to access the two independent data access systems, it needs to complete the data conversion between them by itself. For example, the data in the distributed storage database is converted in structure, the data converted in structure is imported into the distributed cache database through online, batch, etc., and the consistency of the data between the distributed cache database and the distributed storage database needs to be ensured by itself. The logical operation is complicated and prone to errors. SUMMARY

[0004] In view of the problems in the prior art, the present application provides a database access method and device, which can integrate the distributed cache database and the distributed storage database and process the access request of the customer in the distributed cache database and the distributed storage database.

[0005] To solve the above technical problems, the present application provides the following technical solutions:

[0006] In a first aspect, the present application provides a database access method, the database includes a distributed cache database and a distributed storage database, and the method includes:

[0007] Each data read request received from the client is parsed to obtain the corresponding read request type; wherein the read request type includes a hot table read request and a non-hot table read request;

[0008] The corresponding hot table data processing result and non-hot table data processing result are obtained from the storage area corresponding to each read request type;

[0009] The hot meter data processing results and non-hot meter data processing results are aggregated according to the data structure mapping relationship between the distributed cache database and the distributed storage database, and the aggregated results are sent to the client.

[0010] Furthermore, the step of parsing each data read request received from the client to obtain the corresponding read request type includes:

[0011] Perform syntax analysis on the database structured request statement in the data read request to determine the data table to be accessed;

[0012] The read request type is determined based on the type of the data table; wherein, the type of the data table includes hot table type and non-hot table type.

[0013] Furthermore, before retrieving the corresponding hot meter data processing results and non-hot meter data processing results from the storage area corresponding to each read request type, the process also includes:

[0014] The corresponding storage area is determined based on the read request type.

[0015] Further, the read request type is the hot meter read request; the step of obtaining the corresponding hot meter data processing result and non-hot meter data processing result from the storage area corresponding to each read request type includes:

[0016] The data read request is routed to the storage area corresponding to the distributed cache database according to the routing table of the distributed cache database;

[0017] The hot table data is retrieved from the storage area as the result of the hot table data processing; the hot table data includes index data and hash data.

[0018] Further, the read request type is the non-hot meter read request; the step of obtaining the corresponding hot meter data processing result and non-hot meter data processing result from the storage area corresponding to each read request type includes:

[0019] The data read request is routed to the storage area corresponding to the distributed storage database according to the routing table of the distributed storage database;

[0020] Non-hot table data is retrieved from the storage area as the result of the non-hot table data processing; wherein, the non-hot table data is a database structured query statement.

[0021] Further, the mapping relationship includes a first mapping relationship between the index data and the database structured statements, and a second mapping relationship between the hash data and the database structured statements; the aggregation of the hot table data processing results and the non-hot table data processing results according to the data structure mapping relationship between the distributed cache database and the distributed storage database includes:

[0022] Based on the first mapping relationship and the second mapping relationship, the heat table data processing result is converted into a database structured conversion statement;

[0023] The database structure transformation statement and the database structure query statement are packaged together to obtain the aggregation result.

[0024] Furthermore, the database access method further includes:

[0025] The data write request received from the client will be routed to the distributed storage database for processing according to the routing table of the distributed storage database;

[0026] The processed data is transformed according to the data structure mapping relationship between the distributed cache database and the distributed storage database.

[0027] Data synchronization is performed on the distributed cache database and the distributed storage database based on the data structure transformation results.

[0028] Further, the step of routing the data write request received from the client to the distributed storage database for processing according to the routing table of the distributed storage database includes:

[0029] Syntax analysis is performed on the database structured request statement in the data write request to obtain the data to be written and the corresponding data table;

[0030] The data to be written is written to the data table according to the routing table of the distributed storage database; wherein the data table is stored in the storage area of ​​the distributed storage database.

[0031] Further, the mapping relationship includes a first mapping relationship between index data and database structured statements, and a second mapping relationship between hash data and database structured statements; the processed data is the data written to the data table; the step of performing data structure transformation on the processed data according to the data structure mapping relationship between the distributed cache database and the distributed storage database includes:

[0032] Based on the first mapping relationship and the second mapping relationship, the data written to the data table is converted into index data and / or index data.

[0033] Furthermore, the step of synchronizing data between the distributed cache database and the distributed storage database based on the data structure transformation result includes:

[0034] The index data is written into the distributed cache database according to the data structure of the index data;

[0035] The hash data is written into the distributed cache database according to the data structure of the hash data, so that the data in the distributed cache database is consistent with the data in the distributed storage database.

[0036] Secondly, this application provides a database access device, wherein the database includes a distributed cache database and a distributed storage database, and the device includes:

[0037] The read request parsing unit is used to parse each data read request received from the client to obtain the corresponding read request type; wherein, the read request type includes hot meter read requests and non-hot meter read requests;

[0038] The data processing unit is used to obtain the corresponding hot meter data processing results and non-hot meter data processing results from the storage area corresponding to each read request type.

[0039] The aggregation and sending unit is used to aggregate the hot table data processing results and the non-hot table data processing results according to the data structure mapping relationship between the distributed cache database and the distributed storage database, and send the aggregated results to the client.

[0040] Furthermore, in the database access method, the read request parsing unit includes:

[0041] The data table determination module is used to perform syntax analysis on the database structured request statement in the data read request to determine the data table to be accessed.

[0042] The read request type determination module is used to determine the read request type based on the type of the data table; wherein, the type of the data table includes hot table type and non-hot table type.

[0043] Furthermore, the database access device further includes:

[0044] The storage area determination unit is used to determine the corresponding storage area based on the read request type.

[0045] Furthermore, in the database access method, the read request type is the hot table read request; the data processing unit includes:

[0046] The first storage area routing module is used to route the data read request to the storage area corresponding to the distributed cache database according to the routing table of the distributed cache database.

[0047] The hot table data acquisition module is used to acquire hot table data from the storage area as the result of the hot table data processing; the hot table data includes index data and hash data.

[0048] Further, the read request type is the non-hot meter read request; the data processing unit includes:

[0049] The second storage area routing module is used to route the data read request to the storage area corresponding to the distributed storage database according to the routing table of the distributed storage database.

[0050] The non-hot table data acquisition module is used to acquire non-hot table data from the storage area as the non-hot table data processing result; wherein, the non-hot table data is a database structured query statement.

[0051] Further, the mapping relationship includes a first mapping relationship between the index data and the database structured statement, and a second mapping relationship between the hash data and the database structured statement; the aggregation sending unit includes:

[0052] The data conversion module is used to convert the heat table data processing results into database structured conversion statements based on the first mapping relationship and the second mapping relationship;

[0053] The packaging and aggregation module is used to package the database structured transformation statement and the database structured query statement to obtain the aggregation result.

[0054] Furthermore, the database access device further includes:

[0055] The write request processing unit is used to route the data write request received from the client to the distributed storage database for processing according to the routing table of the distributed storage database;

[0056] A data structure conversion unit is used to perform data structure conversion on the processed data according to the data structure mapping relationship between the distributed cache database and the distributed storage database.

[0057] The data synchronization unit is used to synchronize the distributed cache database and the distributed storage database based on the data structure transformation result.

[0058] Further, the write request processing unit includes:

[0059] The module for determining the data to be written and the table is used to perform syntax analysis on the database structured request statement in the data write request to obtain the data to be written and the corresponding data table.

[0060] The data writing module is used to write the data to be written into the data table according to the routing table of the distributed storage database; wherein the data table is stored in the storage area of ​​the distributed storage database.

[0061] Further, the mapping relationship includes a first mapping relationship between index data and database structured statements, and a second mapping relationship between hash data and database structured statements; the processed data is the data written to the data table; the data structure conversion unit includes:

[0062] The data structure conversion module is used to convert the data written to the data table into index data and / or index data according to the first mapping relationship and the second mapping relationship.

[0063] Furthermore, the data synchronization unit includes:

[0064] An index data synchronization module is used to write the index data into the distributed cache database according to the data structure of the index data.

[0065] The hash data synchronization module is used to write the hash data into the distributed cache database according to the data structure of the hash data, so that the data in the distributed cache database is consistent with the data in the distributed storage database.

[0066] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the database access method.

[0067] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the database access method.

[0068] Fifthly, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the database access method.

[0069] To address the problems in the prior art, the database access method and apparatus provided in this application can integrate distributed cache databases and distributed storage databases. Whether a client needs to access a distributed cache database or a distributed storage database, it can send an access request to the server through structured database statements. After receiving the access request, the server can parse it and determine whether the access request should be routed to a distributed cache database or a distributed storage database. Then, it processes the access request based on the specific content in the access request, thereby improving the usability of the database. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Figure 1 This is one of the flowcharts for the database access method in the embodiments of this application;

[0072] Figure 2 This is a flowchart illustrating how the corresponding read request types are obtained in the embodiments of this application;

[0073] Figure 3 This is one of the flowcharts for obtaining data processing results in the embodiments of this application;

[0074] Figure 4 This is the second flowchart of the process for obtaining data processing results in the embodiments of this application;

[0075] Figure 5 This is a flowchart illustrating the aggregation of data processing results in this embodiment of the application;

[0076] Figure 6 This is the second flowchart of the database access method in the embodiments of this application;

[0077] Figure 7 This is a flowchart illustrating the process of handling data write requests in an embodiment of this application;

[0078] Figure 8 This is a flowchart illustrating data synchronization in an embodiment of this application;

[0079] Figure 9 This is one of the structural diagrams of the database access device in the embodiments of this application;

[0080] Figure 10 This is a structural diagram of the read request parsing unit in an embodiment of this application;

[0081] Figure 11 This is one of the structural diagrams of the data processing unit in the embodiments of this application;

[0082] Figure 12 This is the second structural diagram of the data processing unit in the embodiments of this application;

[0083] Figure 13 This is a structural diagram of the aggregation and transmission unit in the embodiments of this application;

[0084] Figure 14 This is a second structural diagram of the database access device in the embodiments of this application;

[0085] Figure 15 This is a structural diagram of the write request processing unit in an embodiment of this application;

[0086] Figure 16 This is a structural diagram of the data synchronization unit in an embodiment of this application;

[0087] Figure 17 This is a schematic diagram of the structure of the electronic device in the embodiments of this application;

[0088] Figure 18 This is a schematic diagram of the integrated architecture of the distributed cache database and the distributed storage database in the embodiments of this application;

[0089] Figure 19 This is a flowchart illustrating the process of handling a data read request sent by the client in an embodiment of this application;

[0090] Figure 20 This is a schematic diagram illustrating the data structure mapping relationship between the distributed cache database and the distributed storage database in this embodiment of the application;

[0091] Figure 21 This is a schematic diagram illustrating the data structure conversion process between the distributed cache database and the distributed storage database in this embodiment of the application. Detailed Implementation

[0092] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0093] It should be noted that the database access method and apparatus provided in this application can be used in the financial field, or in any field other than the financial field. The application field of the database access method and apparatus provided in this application is not limited.

[0094] The acquisition, storage, use, and processing of data in this application all comply with the relevant provisions of national laws and regulations.

[0095] Financial institutions' business systems frequently need to interact with their database servers for read and write operations during financial transactions. These database servers typically include distributed caching databases (also known as distributed buffer storage) and distributed storage databases (which can be distributed relational databases). As the concurrent access volume of financial transactions continues to increase, to reduce the access pressure on distributed storage databases and improve data access responsiveness, financial institutions often need to cache frequently accessed, read-heavy data in distributed caching databases, based on the characteristics of their business data.

[0096] Because distributed caching databases and distributed storage databases differ significantly in their data read / write methods, the industry currently treats them as two independent data access systems. When a client needs to access these two independent systems from a business perspective, it must manually perform data conversion between them and ensure data consistency between the distributed caching database and the distributed storage database. This process is complex and prone to errors. To address these technical problems, this application provides a database access method and apparatus.

[0097] See Figure 18 , Figure 18 This is a schematic diagram of the integrated architecture of the distributed cache database and the distributed storage database in the embodiments of this application.

[0098] in, Figure 18 The components are as follows:

[0099] 1. Proxy Component (DBProxy): A proxy component for distributed storage databases, responsible for parsing the syntax of database structured request statements (essentially SQL statements), and then routing data read requests to distributed cache databases or distributed storage databases according to preset rules.

[0100] 2. The storage component (DBEngine) of the distributed storage database (DBS): It contains data storage areas, and each storage area stores the full amount of business data.

[0101] 3. Distributed caching database (Dcache): Responsible for storing predefined hot table data and providing high-performance read services, reducing the access pressure on the distributed storage database.

[0102] 4. Preloading Component (Ploader): Responsible for loading predefined hot table data from the distributed storage database to the distributed cache database during the initialization phase, and performing data structure transformation according to the data structure mapping relationship between the distributed cache database and the distributed storage database (essentially the mapping relationship between relational tables and key-value data structures).

[0103] 5. Real-time synchronization component (RTSyncer): Used to handle write data requests sent by clients. When hot data is being written, data synchronization between the distributed storage database and the distributed cache database is required.

[0104] In one embodiment, see Figure 1 In order to integrate distributed cache databases and distributed storage databases and process client access requests within these databases, this application provides a database access method, wherein the databases include distributed cache databases and distributed storage databases, and the method includes:

[0105] S101: Parse each data read request received from the client to obtain the corresponding read request type; wherein, the read request type includes hot meter read requests and non-hot meter read requests;

[0106] S102: Obtain the corresponding hot meter data processing results and non-hot meter data processing results from the storage area corresponding to each read request type;

[0107] S103: Aggregate the hot table data processing results and non-hot table data processing results according to the data structure mapping relationship between the distributed cache database and the distributed storage database, and send the aggregation results to the client.

[0108] It is understandable that steps S101 to S103 are the steps for the client to read data from the database. See also Figure 19 The processing steps for reading requests can be understood as follows:

[0109] First, the proxy component receives data read requests (essentially SQL statements) sent by the client and performs syntax (including lexical) parsing.

[0110] Second, access requests involving hot table data are sent to the storage area of ​​the distributed cache database according to the routing table of the distributed cache database.

[0111] Third, access requests that do not involve hot table data are sent to the storage area of ​​the distributed storage database according to the routing table of the distributed storage database.

[0112] Fourth, the proxy component will aggregate the processing results completed in the distributed cache database and the distributed storage database and return them to the client.

[0113] Specifically, see Figure 2 The step of parsing each data read request received from the client to obtain the corresponding read request type includes: performing syntax analysis on the database structured request statement in the data read request to determine the data table to be accessed (S201); determining the read request type according to the type of the data table; wherein, the type of the data table includes hot table type and non-hot table type (S202).

[0114] Understandably, since a database structured request statement (SOP) contains at least the name of the table to be accessed, existing SQL syntax analysis tools can be used to parse the table name from the SOP and thus determine the table to be accessed. Because tables can be categorized as "hot" or "cold," if the table is a hot table, the read request is considered a hot table type, meaning the data accessed involves a distributed cache database. Conversely, if the table is a cold table, the read request is considered a cold table type, meaning the data accessed involves a distributed storage database.

[0115] As can be seen from the above description, the database access method provided in this application can parse each data read request received from the client to obtain the corresponding read request type.

[0116] In one embodiment, before obtaining the corresponding hot meter data processing results and non-hot meter data processing results from the storage area corresponding to each read request type (S102), the method further includes: determining the corresponding storage area according to the read request type.

[0117] Regarding step S102, in one embodiment, see [link to example]. Figure 3 The read request type is the hot table read request; the step of obtaining the corresponding hot table data processing result and non-hot table data processing result from the storage area corresponding to each read request type includes: routing the data read request to the storage area corresponding to the distributed cache database according to the routing table of the distributed cache database (S301); obtaining the hot table data from the storage area as the hot table data processing result; the hot table data includes index data and hash data (S302).

[0118] It is understandable that S301 implements routing through a consistent hashing algorithm in its specific implementation.

[0119] Regarding step S102, in one embodiment, see [link to example]. Figure 4The read request type is the non-hot table read request; the step of obtaining the corresponding hot table data processing result and non-hot table data processing result from the storage area corresponding to each read request type includes: routing the data read request to the storage area corresponding to the distributed storage database according to the routing table of the distributed storage database (S401); obtaining non-hot table data from the storage area as the non-hot table data processing result; wherein, the non-hot table data is a database structured query statement (S402).

[0120] Understandably, in its implementation, S401 uses syntax parsing, lexical parsing, and algorithms such as table name, primary key, and B+tree to route data to the data storage area.

[0121] Regarding step S103, in one embodiment, see [link to example]. Figure 5 The mapping relationship includes a first mapping relationship between the index data and the database structured statements, and a second mapping relationship between the hash data and the database structured statements; the aggregation of the hot table data processing results and the non-hot table data processing results according to the data structure mapping relationship between the distributed cache database and the distributed storage database includes:

[0122] S501: Convert the heat meter data processing result into a database structured conversion statement according to the first mapping relationship and the second mapping relationship;

[0123] Understandably, see Figure 20 The general approach to converting relational data structures to key-value data structures (also known as KV data structures) is as follows.

[0124] 1. Store the primary key and index in an ordered set (zset) respectively. When storing the index, the member is composed of "index value: table name.primary key value" (see point 4 below for the purpose of this operation).

[0125] 2. Store each record in the relation table as a hash, where the key is the primary key, and the field and value are each attribute and value.

[0126] 3. When the SQL statement queries using the primary key, the row record can be directly retrieved using the hash key.

[0127] 4. When the SQL statement queries with an index, the zscan interface can be used to find the member string containing the query index for the sorted set zset mentioned in point 1 above. Then, based on the primary key appended to this member, the corresponding row record can be obtained through the hash with the primary key as the key mentioned in point 2.

[0128] In practice, firstly, create an ordered set (zset) in Dcache to store the primary key from the distributed storage database, with the key being "table name.primary key column name":

[0129] The primary key type is numeric:

[0130] If member is "table_name.primary_key_value" and score is the primary key value, then members in the zset are sorted by score value.

[0131] The primary key type is string:

[0132] The member is "primary key value: table name.primary key value", and the score is 0, because the zset sorts members lexicographically when the score is 0.

[0133] Second, each row in the distributed storage database table is equivalent to a hash in the DCache. The hash structure is: key field value, where the key corresponds to "table name.primary key value" in the relational table, the field corresponds to the attribute name in the relational table, and the value corresponds to the attribute value of the record in the relational table.

[0134] For example:

[0135] One record in the relational table corresponds to one hash record in the DCache:

[0136] key: Table.key1

[0137] field: Title1, Title2, Title3

[0138] value: key1, value12, value13

[0139] The index in the relation table corresponds to an ordered set (zset) in DCache, with the key being "table name.index name".

[0140] The index column is of numeric type:

[0141] If member is "table_name.primary_key_value" and score is the value of the index column, then the members in the zset are sorted by the score value.

[0142] The index column type is string:

[0143] The member is "value of index column: table name.primary key value", and the score is 0, because the zset sorts the members in lexicographical order when the score is 0.

[0144] The following is a specific operational example:

[0145] 1. Load and read hot table information from the user-defined database, and generate a conversion script from hot table data structure to cache data structure.

[0146] 2. Read the hot table metadata from the distributed storage database, and then generate the conversion script DBS2Cache.sql according to the mapping relationship between the primary key, index and KV data structure mentioned above.

[0147] Note: The DBS2Cache.sql script consists of two parts:

[0148] ①FROM section

[0149] 1) Define the hash key and the names of each field (order.1, etc.).

[0150] 2) Define the correspondence between each field value of the hash and the table data ({id:1,name:wangwu,orderNo:20394,createTime:2022.4.5}).

[0151] 3) Create a hash-type cache write command (hmset)

[0152] 4) Define the key (order.id) corresponding to the primary index.

[0153] 5) Create the cache write command for the primary index (zadd)

[0154] ②CONCAT section

[0155] For each row in the database table, a corresponding cache protocol write request is generated, and finally all write requests are connected together using concat.

[0156] 3. Use the SQL script executor to execute the data transformation script generated in step 1, read table data from the distributed database, and generate a command set composed of a distributed caching protocol.

[0157] 4. Call the distributed cache client or batch import executor to import the generated cache command set into the cache database cluster.

[0158] Furthermore, the following example illustrates the conversion from a table structure in a distributed storage database to a key-value structure in a distributed caching database:

[0159]

[0160]

[0161] S502: Package the database structure transformation statement and the database structure query statement to obtain the aggregation result.

[0162] In this step, since the hot table data with KV data structure can be converted into SQL statements (i.e., database structured conversion statements) based on the aforementioned data structure mapping relationship, the processing results of the two parts of the data with the same data structure, namely the database structured conversion statement and the database structured query statement (read directly from the distributed storage database), can be packaged together to obtain the aggregated result, which is then sent to the client. Therefore, for the client, the entire process of reading data from the database only requires issuing a data read request in the form of an SQL statement from the beginning. The execution entity in this embodiment (i.e., the server executing the method provided in this application) parses the request itself and finds the data the client needs to read according to the requirements in the data read request. Then, it uses the data structure mapping relationship to convert the hot table data with KV data structure into SQL statements, and finally packages it together with the SQL statement read directly from the distributed storage database and returns it to the client. Thus, the client always encounters generic SQL statements and does not need to perform data conversion operations based on whether the data it needs to read is in a distributed cache database or a distributed storage database.

[0163] As described above, the database access method provided in this application can integrate distributed cache databases and distributed storage databases. Whether a client needs to access a distributed cache database or a distributed storage database, it can send the access request to the server through structured database statements. After receiving the access request, the server can parse it and determine whether the access request should be routed to a distributed cache database or a distributed storage database. Then, it processes the access request based on the specific content in the access request, thereby improving the usability of the database.

[0164] In one embodiment, see Figure 6 The database access method provided in this application also includes handling data write requests sent by the client. The specific steps are as follows:

[0165] S601: The data write request received from the client is routed to the distributed storage database for processing according to the routing table of the distributed storage database;

[0166] S602: Perform data structure transformation on the processed data according to the data structure mapping relationship between the distributed cache database and the distributed storage database;

[0167] S603: Synchronize the data between the distributed cache database and the distributed storage database based on the data structure transformation result.

[0168] It is understandable that steps S601 to S603 are the steps by which the client writes data to the database. The processing steps for writing a request can be understood as follows:

[0169] First, the proxy component sends the data write request to the storage area of ​​the distributed storage database according to the routing table of the distributed storage database.

[0170] Second, the synchronization component will detect updates to the database cluster table data in near real-time.

[0171] Third, when updating data involves hot tables, the relevant data is automatically synchronized to the distributed cache database.

[0172] In one embodiment, see Figure 7 The step of routing the data write request received from the client to the distributed storage database for processing according to the routing table of the distributed storage database includes:

[0173] S701: Perform syntax analysis on the database structured request statement (which is essentially an SQL statement) in the data write request to obtain the data to be written and the corresponding data table;

[0174] S702: Write the data to be written into the data table according to the routing table of the distributed storage database; wherein the data table is stored in the storage area of ​​the distributed storage database.

[0175] Understandably, since the database structured request statement contains at least the name of the data table to be accessed, existing SQL syntax analysis tools can be used to parse the name of the data table in the database structured request statement, thereby determining the data table to be accessed.

[0176] In this embodiment, it is assumed that when a client writes data, it always writes to a distributed storage database, and not directly to a distributed cache database. Therefore, when considering data synchronization, the focus is on how to convert the data structure of the SQL statement into the corresponding key-value data structure of the distributed cache database.

[0177] Since data tables can be divided into hot tables and non-hot tables, if the request corresponds to writing data to a hot table, it is assumed that after completing the data processing in the distributed storage database, the corresponding data should be written to the distributed cache database, i.e., data synchronization should be performed.

[0178] As described above, the database access method provided in this application can route data write requests received from the client to the distributed storage database for processing based on the database's routing table. Processing refers to the act of writing the data.

[0179] In one embodiment, the mapping relationship includes a first mapping relationship between index data and database structured statements, and a second mapping relationship between hash data and database structured statements; the processed data is the data written to the data table; the step of performing data structure transformation on the processed data according to the data structure mapping relationship between the distributed cache database and the distributed storage database includes:

[0180] Based on the first mapping relationship and the second mapping relationship, the data written to the data table is converted into index data and / or index data.

[0181] Understandably, see Figure 21 Based on the aforementioned data structure conversion relationship, it can be seen that since the data written to the data table is essentially an SQL statement, and since there is a mapping relationship (including the first mapping relationship and the second mapping relationship) between the SQL statement and the KV data structure in the distributed cache database, the data written to the data table can be converted into index data and / or index data based on the first mapping relationship and the second mapping relationship.

[0182] In one embodiment, see Figure 8 The step of synchronizing data between the distributed cache database and the distributed storage database based on the data structure transformation result includes:

[0183] S801: Write the index data into the distributed cache database according to the data structure of the index data;

[0184] S802: Write the hash data into the distributed cache database according to the data structure of the hash data, so that the data in the distributed cache database is consistent with the data in the distributed storage database.

[0185] It is understood that in the aforementioned embodiments, since the data written to the data table has been converted into index data and / or indexed data, and the data structure corresponding to the index data and / or indexed data is recognizable by the distributed cache database, both index data and hash data can be written to the distributed cache database. This achieves data synchronization between the distributed cache database and the distributed storage database.

[0186] As described above, the database access method provided in this application can integrate distributed cache databases and distributed storage databases. Whether a client needs to access a distributed cache database or a distributed storage database, it can send the access request to the server through structured database statements. After receiving the access request, the server can parse it and determine whether the access request should be routed to a distributed cache database or a distributed storage database. Then, it processes the access request based on the specific content in the access request, thereby improving the usability of the database.

[0187] In summary, the method and apparatus provided in this application have at least the following advantages:

[0188] By integrating distributed storage databases and distributed cache databases, when a client needs to use the functionality of the distributed cache database, it only needs to perform simple configuration using common SQL statements and provide the data table name. The server can then automatically retrieve the data from the corresponding database table. In other words, the client only needs to use standard SQL statements to access both databases, and the server can automatically determine whether a read request from the client should be routed to the distributed storage database or the distributed cache database, improving database usability. Furthermore, when a client sends a write request, there is no need to worry about the data written to the distributed storage database failing to synchronize with the distributed cache database. The server executing the method provided in this application can automatically complete data synchronization using the aforementioned data structure mapping relationship. The client does not need to programmatically convert SQL statements to a key-value data structure.

[0189] 1) The client does not need to perform key-value structure conversion on hot data in DBS and import it into DCache.

[0190] 2) The client does not need to care whether the data in DBS is consistent with the data in Dcache. The server will synchronize the data in DBS and Dcache in a certain time range through the background asynchronous method according to the aforementioned data read and write requests.

[0191] 3) The client does not need to implement the circuit breaker switchback operation when DCache is unavailable; the server will arbitrate the switchback automatically.

[0192] Furthermore, the method provided in this application can automatically convert the table data structure in the relational database into a key-value storage structure, replacing the client, and automatically complete the data synchronization between the two databases. The method provided in this application can also automatically switch back to the relational database when the cache expires or an anomaly occurs, replacing the client, thus improving the overall high availability of the system and ensuring the continuity of business processing.

[0193] The so-called circuit breaker return capability refers to:

[0194] 1. The proxy component periodically probes the status of the distributed cache database.

[0195] 2. If the distributed cache database is in normal condition, the proxy component will process the requests according to the read and write process described above.

[0196] 3. If the distributed cache database is in an abnormal state, the proxy component will directly forward the data request to the corresponding storage area of ​​the distributed storage database, and the processing flow will no longer go through the distributed cache database.

[0197] Based on the same inventive concept, this application also provides a database access device that can be used to implement the methods described in the above embodiments, as described in the following embodiments. Since the principle by which the database access device solves the problem is similar to that of the database access method, the implementation of the database access device can refer to the implementation of the method based on software performance benchmarks, and repeated details will not be elaborated further. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0198] In one embodiment, see Figure 9 In order to integrate distributed cache databases and distributed storage databases and process client access requests in the distributed cache databases and distributed storage databases, this application provides a database access device, wherein the database includes a distributed cache database and a distributed storage database, and the device includes: a read request parsing unit 901, a data processing unit 902 and an aggregation sending unit 903.

[0199] The read request parsing unit 901 is used to parse each data read request received from the client to obtain the corresponding read request type; wherein, the read request type includes hot meter read request and non-hot meter read request;

[0200] Data processing unit 902 is used to obtain the corresponding hot meter data processing results and non-hot meter data processing results from the storage area corresponding to each read request type;

[0201] The aggregation and sending unit 903 is used to aggregate the hot table data processing results and the non-hot table data processing results according to the data structure mapping relationship between the distributed cache database and the distributed storage database, and send the aggregation results to the client.

[0202] In one embodiment, see Figure 10The read request parsing unit 901 includes: a data table determination module 1001 and a read request type determination module 1002.

[0203] The data table determination module 1001 is used to perform syntax analysis on the database structured request statement in the data read request to determine the data table to be accessed.

[0204] The read request type determination module 1002 is used to determine the read request type according to the type of the data table; wherein, the type of the data table includes hot table type and non-hot table type.

[0205] In one embodiment, the database access device further includes:

[0206] The storage area determination unit is used to determine the corresponding storage area based on the read request type.

[0207] In one embodiment, see Figure 11 The read request type is the hot meter read request; the data processing unit includes: a first storage area routing module 1101 and a hot meter data acquisition module 1102.

[0208] The first storage area routing module 1101 is used to route the data read request to the storage area corresponding to the distributed cache database according to the routing table of the distributed cache database;

[0209] The hot table data acquisition module 1102 is used to acquire hot table data from the storage area as the result of the hot table data processing; the hot table data includes index data and hash data.

[0210] In one embodiment, see Figure 12 The read request type is the non-hot meter read request; the data processing unit includes: a second storage area routing module 1201 and a non-hot meter data acquisition module 1202.

[0211] The second storage area routing module 1201 is used to route the data read request to the storage area corresponding to the distributed storage database according to the routing table of the distributed storage database.

[0212] The non-hot table data acquisition module 1202 is used to acquire non-hot table data from the storage area as the non-hot table data processing result; wherein, the non-hot table data is a database structured query statement.

[0213] In one embodiment, see Figure 13The mapping relationship includes a first mapping relationship between the index data and the database structured statement and a second mapping relationship between the hash data and the database structured statement; the aggregation sending unit 903 includes: a data conversion module 1301 and a packaging aggregation module 1302.

[0214] Data conversion module 1301 is used to convert the heat table data processing result into a database structured conversion statement according to the first mapping relationship and the second mapping relationship;

[0215] The packaging and aggregation module 1302 is used to package the database structured conversion statement and the database structured query statement to obtain the aggregation result.

[0216] In one embodiment, see Figure 14 The database access device further includes: a write request processing unit 1401, a data structure conversion unit 1402, and a data synchronization unit 1403.

[0217] The write request processing unit 1401 is used to route the data write request received from the client to the distributed storage database for processing according to the routing table of the distributed storage database;

[0218] The data structure conversion unit 1402 is used to perform data structure conversion on the processed data according to the data structure mapping relationship between the distributed cache database and the distributed storage database.

[0219] The data synchronization unit 1403 is used to synchronize the distributed cache database and the distributed storage database according to the data structure conversion result.

[0220] In one embodiment, see Figure 15 The write request processing unit 1401 includes:

[0221] The module 1501 for determining the data to be written and the table is used to perform syntax analysis on the database structured request statement in the data write request to obtain the data to be written and the corresponding data table.

[0222] The data writing module 1502 is used to write the data to be written into the data table according to the routing table of the distributed storage database; wherein the data table is stored in the storage area of ​​the distributed storage database.

[0223] In one embodiment, the mapping relationship includes a first mapping relationship between index data and database structured statements, and a second mapping relationship between hash data and database structured statements; the processed data is the data written to the data table; the data structure conversion unit 1402 includes:

[0224] The data structure conversion module is used to convert the data written to the data table into index data and / or index data according to the first mapping relationship and the second mapping relationship.

[0225] In one embodiment, see Figure 16 The data synchronization unit 1403 includes:

[0226] The index data synchronization module 1601 is used to write the index data into the distributed cache database according to the data structure of the index data.

[0227] The hash data synchronization module 1602 is used to write the hash data into the distributed cache database according to the data structure of the hash data, so that the data in the distributed cache database is consistent with the data in the distributed storage database.

[0228] From a hardware perspective, in order to integrate distributed cache databases and distributed storage databases and process client access requests within these databases, this application provides an embodiment of an electronic device for implementing all or part of the database access method. The electronic device specifically includes the following components:

[0229] The system comprises a processor, a memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to realize information transmission between the database access device and core business systems, user terminals, and related databases and other related devices; the logic controller can be a desktop computer, tablet computer, or mobile terminal, etc., and this embodiment is not limited to these. In this embodiment, the logic controller can be implemented with reference to the embodiments of the database access method and the database access device in the embodiments, the content of which is incorporated herein, and repeated details will not be described again.

[0230] It is understood that the user terminal may include smartphones, tablet computers, network set-top boxes, portable computers, desktop computers, personal digital assistants (PDAs), in-vehicle devices, smart wearable devices, etc. Among these, the smart wearable devices may include smart glasses, smartwatches, smart bracelets, etc.

[0231] In practical applications, the database access method can be executed on the electronic device side as described above, or all operations can be completed on the client device. The choice can be made based on the processing power of the client device and the limitations of the user's usage scenario. This application does not impose any limitations on this. If all operations are completed on the client device, the client device may further include a processor.

[0232] The aforementioned client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission. The server may include a server on the task scheduling center side; in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a distributed server structure.

[0233] Figure 17 This is a schematic block diagram illustrating the system configuration of the electronic device 9600 according to an embodiment of this application. Figure 17 As shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It is worth noting that... Figure 17 This is an example; other types of structures can also be used to supplement or replace this structure to achieve telecommunications functions or other functions.

[0234] In one embodiment, the database access method functionality can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control:

[0235] S101: Parse each data read request received from the client to obtain the corresponding read request type; wherein, the read request type includes hot meter read requests and non-hot meter read requests;

[0236] S102: Obtain the corresponding hot meter data processing results and non-hot meter data processing results from the storage area corresponding to each read request type;

[0237] S103: Aggregate the hot table data processing results and non-hot table data processing results according to the data structure mapping relationship between the distributed cache database and the distributed storage database, and send the aggregation results to the client.

[0238] As described above, the database access method provided in this application can integrate distributed cache databases and distributed storage databases. Whether a client needs to access a distributed cache database or a distributed storage database, it can send the access request to the server through structured database statements. After receiving the access request, the server can parse it and determine whether the access request should be routed to a distributed cache database or a distributed storage database. Then, it processes the access request based on the specific content in the access request, thereby improving the usability of the database.

[0239] In another embodiment, the database access device can be configured separately from the central processing unit 9100. For example, the data composite transmission device database access device can be configured as a chip connected to the central processing unit 9100, and the database access method function can be implemented through the control of the central processing unit.

[0240] like Figure 17 As shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily need to include these components. Figure 17 All components shown; in addition, the electronic device 9600 may also include Figure 17 For components not shown, please refer to existing technologies.

[0241] like Figure 17 As shown, the central processing unit 9100, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device, which receives inputs and controls the operation of various components of the electronic device 9600.

[0242] The memory 9140 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information, and also store a program for executing that information. The central processing unit 9100 may execute the program stored in the memory 9140 to perform information storage or processing, etc.

[0243] Input unit 9120 provides input to central processing unit 9100. Input unit 9120 may be, for example, a keypad or touch input device. Power supply 9170 provides power to electronic device 9600. Display 9160 displays images and text. Display may be, for example, an LCD display, but is not limited thereto.

[0244] The memory 9140 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 9140 can also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs or processes for executing the operation of the electronic device 9600 via the central processing unit 9100.

[0245] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for the electronic device's communication functions and / or for performing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0246] The communication module 9110 is a transmitter / receiver 9110 that transmits and receives signals via the antenna 9111. The communication module (transmitter / receiver) 9110 is coupled to the central processing unit 9100 to provide input signals and receive output signals, which can be the same as in a conventional mobile communication terminal.

[0247] Based on different communication technologies, multiple communication modules 9110 can be configured in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module (transmitter / receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby realizing typical telecommunications functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 9130 is also coupled to a central processing unit 9100, enabling on-device recording via the microphone 9132 and on-device playback of stored sound via the speaker 9131.

[0248] Embodiments of this application also provide a computer-readable storage medium capable of implementing all steps of the database access method with a server or client as the execution subject in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the database access method with a server or client as the execution subject in the above embodiments. For example, when the processor executes the computer program, it implements the following steps:

[0249] S101: Parse each data read request received from the client to obtain the corresponding read request type; wherein, the read request type includes hot meter read requests and non-hot meter read requests;

[0250] S102: Obtain the corresponding hot meter data processing results and non-hot meter data processing results from the storage area corresponding to each read request type;

[0251] S103: Aggregate the hot table data processing results and non-hot table data processing results according to the data structure mapping relationship between the distributed cache database and the distributed storage database, and send the aggregation results to the client.

[0252] As described above, the database access method provided in this application can integrate distributed cache databases and distributed storage databases. Whether a client needs to access a distributed cache database or a distributed storage database, it can send the access request to the server through structured database statements. After receiving the access request, the server can parse it and determine whether the access request should be routed to a distributed cache database or a distributed storage database. Then, it processes the access request based on the specific content in the access request, thereby improving the usability of the database.

[0253] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0254] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0255] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0256] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0257] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A database access method, characterized in that, The database includes a distributed cache database and a distributed storage database, and the method includes: Each data read request received from the client is parsed to obtain the corresponding read request type; wherein, the read request type includes hot meter read requests and non-hot meter read requests; The corresponding hot table data processing results and non-hot table data processing results are obtained from the storage area corresponding to each read request type. The hot table data processing results include index data and hash data, and the non-hot table data processing results include database structured statements. The hot table data processing results and non-hot table data processing results are aggregated according to the data structure mapping relationship between the distributed cache database and the distributed storage database, and the aggregated results are sent to the client. The mapping relationship includes a first mapping relationship between the index data and the database structured statement and a second mapping relationship between the hash data and the database structured statement. The hot meter data processing results and non-hot meter data processing results are aggregated according to the data structure mapping relationship between the distributed cache database and the distributed storage database, including: Based on the first mapping relationship and the second mapping relationship, the heat table data processing result is converted into a database structured conversion statement; The database structure transformation statement and the database structure query statement are packaged together to obtain the aggregation result.

2. The database access method according to claim 1, characterized in that, The step of parsing each data read request received from the client to obtain the corresponding read request type includes: Perform syntax analysis on the database structured request statement in the data read request to determine the data table to be accessed; The read request type is determined based on the type of the data table; wherein, the type of the data table includes hot table type and non-hot table type.

3. The database access method according to claim 1, characterized in that, Before retrieving the corresponding hot meter data processing results and non-hot meter data processing results from the storage area corresponding to each read request type, the process also includes: The corresponding storage area is determined based on the read request type.

4. The database access method according to claim 1, characterized in that, The read request type is the hot meter read request; the step of obtaining the corresponding hot meter data processing results and non-hot meter data processing results from the storage area corresponding to each read request type includes: The data read request is routed to the storage area corresponding to the distributed cache database according to the routing table of the distributed cache database; The hot table data is retrieved from the storage area as the result of the hot table data processing; the hot table data includes index data and hash data.

5. The database access method according to claim 1, characterized in that, The read request type is the non-hot meter read request; the step of obtaining the corresponding hot meter data processing results and non-hot meter data processing results from the storage area corresponding to each read request type includes: The data read request is routed to the storage area corresponding to the distributed storage database according to the routing table of the distributed storage database; Non-hot table data is retrieved from the storage area as the result of the non-hot table data processing; wherein, the non-hot table data is a database structured query statement.

6. The database access method according to claim 1, characterized in that, Also includes: The data write request received from the client will be routed to the distributed storage database for processing according to the routing table of the distributed storage database; The processed data is transformed according to the data structure mapping relationship between the distributed cache database and the distributed storage database. Data synchronization is performed on the distributed cache database and the distributed storage database based on the data structure transformation results.

7. The database access method according to claim 6, characterized in that, The step of routing the data write request received from the client to the distributed storage database for processing according to the routing table of the distributed storage database includes: Syntax analysis is performed on the database structured request statement in the data write request to obtain the data to be written and the corresponding data table; The data to be written is written to the data table according to the routing table of the distributed storage database; wherein the data table is stored in the storage area of ​​the distributed storage database.

8. The database access method according to claim 6, characterized in that, The mapping relationship includes a first mapping relationship between index data and database structured statements, and a second mapping relationship between hash data and database structured statements; the processed data is the data written to the data table. The step of performing data structure transformation on the processed data according to the data structure mapping relationship between the distributed cache database and the distributed storage database includes: Based on the first mapping relationship and the second mapping relationship, the data written to the data table is converted into index data and / or index data.

9. The database access method according to claim 8, characterized in that, The step of synchronizing data between the distributed cache database and the distributed storage database based on the data structure transformation result includes: The index data is written into the distributed cache database according to the data structure of the index data; The hash data is written into the distributed cache database according to the data structure of the hash data, so that the data in the distributed cache database is consistent with the data in the distributed storage database.

10. A database access device, characterized in that, The database includes a distributed cache database and a distributed storage database, and the device includes: The read request parsing unit is used to parse each data read request received from the client to obtain the corresponding read request type; wherein, the read request type includes hot meter read requests and non-hot meter read requests; The data processing unit is used to obtain the corresponding hot table data processing results and non-hot table data processing results from the storage area corresponding to each read request type. The hot table data processing results include index data and hash data, and the non-hot table data processing results include database structured statements. An aggregation and sending unit is used to aggregate the hot table data processing results and the non-hot table data processing results according to the data structure mapping relationship between the distributed cache database and the distributed storage database, and send the aggregated results to the client. The mapping relationship includes a first mapping relationship between the index data and the database structured statement and a second mapping relationship between the hash data and the database structured statement. The aggregation and sending unit includes: a data conversion module and a packaging and aggregation module. The data conversion module is used to convert the heat table data processing results into database structured conversion statements based on the first mapping relationship and the second mapping relationship; The packaging and aggregation module is used to package the database structured transformation statement and the database structured query statement to obtain the aggregation result.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the database access method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the database access method according to any one of claims 1 to 9.

13. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the database access method according to any one of claims 1 to 9.

Images