Database transformation processing method and device, electronic equipment and medium

Abstract

The invention relates to a database modification processing method and device, electronic equipment and a medium, and is applied to the field of distributed storage, the method provides a reusable and flow-unified single database upgrading modification mode, and the method comprises the steps that corresponding database middleware and segmentation keys are determined for a single database of a target application; according to the generation mode of the table primary key of the single database, a corresponding table primary key transformation strategy is determined, and corresponding transformation processing is carried out, so that the primary key is globally unique in the distributed storage; according to the business logic and the response logic of the target application, determining a distributed read-write transformation strategy corresponding to the read operation and the write operation of the target application, and performing corresponding transformation processing; and integrating the database middleware, and synchronously performing read-write operation on the single database and the transformed sub-databases and sub-tables. And the damage risk of database upgrading iteration to services can be reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of distributed storage technology, and in particular to a method, device, electronic device, and medium for database transformation and processing. Background Art

[0002] As the data storage scale for various applications continues to grow, single databases have slow response times and storage performance bottlenecks in high-concurrency scenarios, leading to the emergence of distributed storage solutions. Typically, a single database needs to be vertically or horizontally partitioned to create sharded databases and tables. This involves splitting a single database into multiple databases, a single data table into multiple tables, or both.

[0003] In the process of realizing the concept of the present disclosure, the inventors found that there are at least the following technical problems in the relevant technology: at different development stages, in view of the increase in data volume and the need to improve performance, the single database corresponding to each application needs to be transformed and upgraded into separate databases and tables. However, during the upgrade process, the database upgrade plan is mostly determined by R&D personnel, which sometimes may not match or conflict with the business logic; and since the scenarios corresponding to different applications are different, the methods used by different R&D personnel to upgrade and transform the database also vary greatly, making it difficult to form a process-based system, that is, it is difficult to reuse a single database transformation method. Summary of the Invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, embodiments of the present disclosure provide a method, apparatus, electronic device, and medium for database transformation processing.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for database transformation and processing. The method includes: determining the corresponding database middleware and sharding key for a single database of a target application; determining the corresponding table primary key transformation strategy and performing corresponding transformation processing based on the generation method of the table primary key of the single database, so that the primary key is globally unique in distributed storage; determining the distributed read and write transformation strategy corresponding to the read and write operations of the target application based on the business logic and response logic of the target application and performing corresponding transformation processing; integrating the database middleware and synchronously performing read and write operations on the single database and the transformed sharded databases and tables.

[0006] According to an embodiment of the present disclosure, the above-mentioned business logic includes: whether delayed query is supported during data reading; the above-mentioned response logic includes: whether the use of partitioning keys is supported during data reading and writing. Among them, according to the business logic and response logic of the above-mentioned target application, the distributed read and write transformation strategy corresponding to the read and write operations of the above-mentioned target application is determined and corresponding transformation processing is performed, including: when the above-mentioned response logic indicates that the data reading and writing process supports the use of partitioning keys, determining the transformation strategy corresponding to the distributed read and write operations of the above-mentioned target application includes: modifying the execution code of the read operation statement or write operation statement of the above-mentioned target application, and obtaining the target execution code for executing the read operation or write operation for the sharded database and table based on the above-mentioned partitioning key.

[0007] According to an embodiment of the present disclosure, based on the business logic and response logic of the above-mentioned target application, the distributed read and write transformation strategy corresponding to the read operation and write operation of the above-mentioned target application is determined and corresponding transformation processing is performed, which also includes: when the above-mentioned response logic indicates that the data reading process does not support the use of split keys, and the above-mentioned business logic indicates that the data reading process supports delayed queries, determining that the transformation strategy corresponding to the distributed read operation of the above-mentioned target application includes: constructing an architecture for synchronizing data from sharded libraries and tables to the main storage node, and forwarding the query request to the above-mentioned main storage node for processing.

[0008] According to an embodiment of the present disclosure, based on the business logic and response logic of the above-mentioned target application, the distributed read and write transformation strategies corresponding to the read and write operations of the above-mentioned target application are determined and corresponding transformation processing is performed, which also includes: when the above-mentioned response logic indicates that the data reading process does not support the use of split keys, and the above-mentioned business logic indicates that the data reading process does not support delayed queries, determining that the transformation strategy corresponding to the distributed read operations of the above-mentioned target application includes: selecting a cache that supports secondary indexes, and forwarding the query request to the above-mentioned cache for processing; the leaf nodes of the index tree corresponding to the above-mentioned secondary indexes store primary keys rather than data.

[0009] According to an embodiment of the present disclosure, the above method further includes: verifying the transformation effect of the above sub-libraries and sub-tables. Among them, integrating the above-mentioned database middleware, and synchronously performing read and write operations on the above-mentioned single database and the transformed sub-libraries and sub-tables, includes: according to the type of the above-mentioned database middleware, integrating the above-mentioned database middleware in the above-mentioned target application, wherein the distributed read and write mode after the transformation is constructed as the read and write routing logic of the above-mentioned database middleware and connected to the transformed sub-libraries and sub-tables; for the new data of the above-mentioned target application, synchronously writing to the above-mentioned single database and the transformed sub-libraries and sub-tables; for the historical data of the above-mentioned target application, synchronizing from the above-mentioned single database to the above-mentioned sub-libraries and sub-tables. Verifying the transformation effect of the above-mentioned sub-libraries and sub-tables, includes: verifying the consistency between the writing results of the above-mentioned sub-libraries and sub-tables and the writing results of the above-mentioned single database; based on the dynamic data source, verifying the consistency between the query results of the above-mentioned sub-libraries and sub-tables and the query results of the above-mentioned single database; when the read and write results of the sub-libraries and sub-tables are consistent with the read and write results of the above-mentioned single database, it is deemed that the transformation effect of the above-mentioned sub-libraries and sub-tables has been verified; the above-mentioned read and write results include write results and query results.

[0010] According to an embodiment of the present disclosure, the above method also includes: when the transformation effect of the above-mentioned sharded libraries and tables is verified to be successful, deleting the logic code used for synchronous reading and writing during the verification period, and constructing a read and write path pointing to the data source of the sharded libraries and tables.

[0011] According to an embodiment of the present disclosure, the above method also includes: determining the scale information of the sub-libraries and tables based on the data increment information and data retention time information corresponding to the business development of the above-mentioned target application; and dividing the transformed sub-libraries and tables based on the above-mentioned scale information.

[0012] According to an embodiment of the present disclosure, based on the generation method of the table primary key of the above-mentioned single database, the corresponding table primary key transformation strategy is determined and the corresponding transformation processing is performed to make the primary key globally unique in the distributed storage, including: when the generation method of the above-mentioned table primary key is the auto-increment method, determining the corresponding table primary key transformation strategy includes: generating the table primary key based on the global identifier generation algorithm.

[0013] According to an embodiment of the present disclosure, for a single database of a target application, the corresponding database middleware and split key are determined, including: presenting middleware options and field options of the data table in the above-mentioned single database in a visual interface; determining the target middleware selected by the user in the above-mentioned middleware options as the database middleware corresponding to the above-mentioned target application; and determining the target field selected by the user in the above-mentioned field options as the split key corresponding to the above-mentioned target application.

[0014] In the second aspect, an embodiment of the present disclosure provides a device for database transformation and processing. The above-mentioned device includes: a determination module, a first transformation module, a second transformation module and a processing module. The above-mentioned determination module is used to determine the corresponding database middleware and sharding key for the single database of the target application. The above-mentioned first transformation module is used to determine the corresponding table primary key transformation strategy and perform corresponding transformation processing based on the generation method of the table primary key of the above-mentioned single database, so that the primary key is globally unique in the distributed storage. The above-mentioned second transformation module is used to determine the distributed read and write transformation strategy corresponding to the read and write operations of the above-mentioned target application according to the business logic and response logic of the above-mentioned target application and perform corresponding transformation processing. The above-mentioned processing module is used to integrate the above-mentioned database middleware and synchronously perform read and write operations on the above-mentioned single database and the transformed sub-libraries and tables.

[0015] In a third aspect, embodiments of the present disclosure provide an electronic device. The electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; the memory is configured to store a computer program; and the processor is configured to implement the above-described database transformation method when executing the program stored in the memory.

[0016] In a fourth aspect, embodiments of the present disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described database transformation method.

[0017] The above technical solutions provided by the embodiments of the present disclosure have at least some or all of the following advantages:

[0018] It provides a reusable and process-unified single database upgrade and transformation method. By targeting the single database of the target application, the corresponding database middleware and sharding key are determined; according to the generation method of the table primary key of the above single database, the corresponding table primary key transformation strategy is determined and the corresponding transformation processing is performed, so that the primary key is globally unique in the distributed storage; according to the business logic and response logic of the above target application, the distributed read and write transformation strategy corresponding to the read and write operations of the above target application is determined and the corresponding transformation processing is performed; by comprehensively considering the business logic and response logic of the target application, the distributed read and write transformation strategy corresponding to the distributed storage is obtained and the corresponding transformation processing is performed. The read and write methods of a single database are transformed to obtain a read and write method for connecting to shard libraries and tables, which is suitable for various business needs and reduces the risk of damage to the business caused by database upgrades and iterations. After integrating the above-mentioned database middleware, it can be connected to shard libraries and tables based on the transformed read and write method, and the primary keys of shard libraries and tables are globally unique. At the same time, by synchronously performing read and write operations on the above-mentioned single database and the transformed shard libraries and tables, based on the synchronous read and write method, not only can the execution of the business be met during the business process, but data can also be written or read (or described as data query) in the shard libraries and tables synchronously, which is helpful for subsequent verification of the transformation effect of the shard libraries and tables. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0020] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] Figure 1 The system architecture of the database transformation processing method applicable to the embodiment of the present disclosure is schematically shown;

[0022] Figure 2 The following schematically shows a flowchart of a method for database transformation processing according to an embodiment of the present disclosure;

[0023] Figure 3 The following schematically shows the architecture of a single database after being transformed into separate libraries and tables according to an embodiment of the present disclosure;

[0024] Figure 4 The following schematically illustrates an implementation process of a method for database transformation processing according to an embodiment of the present disclosure;

[0025] Figure 5 The following schematically shows a structural block diagram of a device for database transformation processing according to an embodiment of the present disclosure;

[0026] Figure 6 The structural block diagram of the electronic device provided by the embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0028] Figure 1 The system architecture of the database transformation processing method applicable to the embodiment of the present disclosure is schematically shown.

[0029] Reference Figure 1 As shown, the system architecture 100 of the method for database transformation processing applicable to the embodiment of the present disclosure includes: a database transformation processing apparatus 101, a server 120 and a terminal device 110.

[0030] The terminal device 110 has installed thereon a target application for which a database upgrade is to be performed. The target application may be any type of application, such as, but not limited to, online shopping applications, web browser applications, video playback applications, search applications, instant messaging tools, financial applications, email clients, social networking platform software, operation and maintenance applications, and development applications. The user of the terminal device is the user of the target application.

[0031] The server 120 is used to provide data processing and data storage services for the target application installed on the terminal device. Before the transformation, the server 120 is based on a single database, such as Figure 1 The database A in the example provides data storage services for the target application. With the development of business and the increase in storage scale, there is a need to upgrade and transform the database. The developer or business personnel of the target application can use the above-mentioned database transformation processing device 101. The above-mentioned database transformation processing device 101 is installed with a functional module corresponding to the transformation code, or a data transformation application program, which is used to execute the database transformation processing method provided by the embodiment of the present disclosure to realize the transformation and upgrade of database A. Database A includes data table S, and the upgraded sub-library and sub-table are obtained. For example, refer to Figure 1As shown, after the upgrade and transformation, database A has two sub-databases: database A1 and database A2. Database A1 contains two sub-tables: S11 and S12; database A2 contains two sub-tables: S21 and S22.

[0032] In the embodiments of the present disclosure, database and table sharding may mean splitting only the database, or only the data table, or splitting both the database and the data table.

[0033] The first exemplary embodiment of the present disclosure provides a method for database transformation processing, which can be executed by the apparatus 101 in the system architecture 100 of the first exemplary embodiment.

[0034] Figure 2 The flowchart of the method for database transformation processing according to an embodiment of the present disclosure is schematically shown. Figure 4 The following schematically shows an implementation diagram of a method for database transformation processing according to an embodiment of the present disclosure.

[0035] Reference Figure 2 As shown, the method for database transformation processing provided by the embodiment of the present disclosure includes the following steps: S210, S220, S230 and S240.

[0036] In step S210 , for a single database of a target application, corresponding database middleware and sharding key are determined.

[0037] The above-mentioned target application is an application that requires database upgrade and transformation.

[0038] Reference Figure 1 As shown, when the server 120 responds to a data processing or data storage request initiated by the terminal device 110, the server 120 performs data writing or data reading operations by accessing a data table S of a database A (as an example of a single database). While a user (e.g., a developer or business person of a target application, using the aforementioned database transformation processing apparatus 101 to perform database transformation) transforms a single database into separate databases and tables using the aforementioned database transformation processing apparatus 101, the target application of the terminal device 110 can still provide corresponding data processing services based on the aforementioned database A, achieving simultaneous transformation while in operation.

[0039] In some embodiments, in the above step S210, for a single database of the target application, the corresponding database middleware and split key are determined, including: presenting middleware options and field options of the data table in the above single database in a visual interface; determining the target middleware selected by the user in the above middleware options as the database middleware corresponding to the above target application; and determining the target field selected by the user in the above field options as the split key corresponding to the above target application.

[0040] In the embodiments of the present disclosure, by providing middleware options for selection, users (e.g., developers or business personnel of target applications) can easily select corresponding database middleware based on business development needs and the adaptability of the target application. For example, users can select a database middleware from the middleware options that has good performance, good scalability, high stability, an active community, and is consistent with the company's technological development direction.

[0041] The aforementioned middleware options include at least one database middleware product, covering as many currently available database middleware products as possible. Newly emerging database middleware products will also be updated to the aforementioned middleware options. This database middleware may include both proxy mode and smart-client mode middleware.

[0042] The proxy mode is to independently deploy a proxy service, which manages multiple database instances (corresponding to sub-databases, such as Figure 1 In the example databases A1 and A2, all database operation statements (e.g., SQL (Structured Query Language) statements) corresponding to user actions (e.g., clicking "Add to Cart," placing an order, checking logistics, or making a payment in a shopping app) in terminal device 120 are first sent to this proxy service. This proxy service then performs the corresponding read and write logic conversions, then operates the underlying database, obtains the database operation results, and returns them to the target application. Proxy-mode middleware products include, but are not limited to, Cobar middleware, MyCat middleware, MySQL-Proxy middleware, Atlas middleware, and Sharding-Sphere.

[0043] The smart client model involves modifying the business code and introducing an SDK (Software Development Kit) on the client side that supports read-write splitting or database sharding. In this model, there are multiple database shards, each corresponding to a number of proxy connection pools. A smart-client module is encapsulated on top of the proxy connection pool or driver to manage connections between each proxy connection pool and the corresponding database shard. SQL statements generated by the application are processed by the smart-client module, which analyzes the statements and sends them to the corresponding proxy connection pool for forwarding to the corresponding database shard. For example, in the case of read-write splitting, the module selects between the slave and master databases. In the case of database sharding, the module performs SQL parsing and rewriting, then routes the statements to the different database shards. The results are merged and returned to the application layer. Middleware products that support the smart client model include, but are not limited to, Zebra middleware, TDDL middleware, and Sharding-JDBC middleware.

[0044] The data table S in the above-mentioned single database (for example, database A) contains multiple fields. Some or all of the above fields can be extracted and provided as field options for the user to select. The user selects one or more fields as the splitting key. The above-mentioned splitting key is used to split the database and data table, which can be horizontal splitting or vertical splitting; the user can select the splitting method in the field options.

[0045] As an example of vertical slicing, the field options for shopping applications include: user ID, user payment account, product number, product category, and product price. For example, the fields selected by the user for vertical slicing (an example of a slicing key) are: user ID and product number, which are divided into two data sub-tables, one of which contains the following fields: user ID, user payment account; the other data sub-table contains the following fields: product number, product category, and product price. This vertical slicing method is generally used in the process of service-oriented transformation of businesses. The data in different data tables can independently support a service module, such as user query, product query, etc. Vertical slicing will increase the response time of a single user request, but it can increase the throughput of the entire service.

[0046] As an example of horizontal sharding, a shopping app might have fields like seller ID, transaction time, order number, logistics number, and payment transaction number. For example, if the user selects the order number as the field for horizontal sharding, then when the data is subsequently stored in a sharded table, each row of data will be hashed based on the order number, and the data will be divided into different sharded tables based on the hash results.

[0047] In step S220, according to the generation method of the table primary key of the single database, a corresponding table primary key transformation strategy is determined and corresponding transformation processing is performed, so that the primary key is globally unique in the distributed storage.

[0048] The primary key of a table is a key used to uniquely identify each data record in a data table. It is usually described by the table ID.

[0049] In some embodiments, in the above step S220, according to the generation method of the table primary key of the above single database, the corresponding table primary key transformation strategy is determined and the corresponding transformation processing is performed, so that the primary key is globally unique in the distributed storage, including: when the generation method of the above table primary key is an auto-increment method, determining the corresponding table primary key transformation strategy includes: generating the table primary key based on a global identifier generation algorithm; when the generation method of the above table primary key is a non-auto-increment method, there is no need to perform transformation processing on the table primary key.

[0050] For non-auto-increment table primary key generation methods, most are based on a global generation algorithm to obtain the primary key, thus ensuring that the primary key is globally unique in distributed storage without the need for modification. In some embodiments, the global identifier generation algorithm includes a snowflake algorithm.

[0051] In the business scenarios corresponding to different applications, some are set up with a table primary key auto-increment method. In this case, since the primary key auto-increment method in a single database can ensure the uniqueness of a single record, in the scenario of distributed storage, the primary key auto-increment method in the table may cause the primary keys to be identical between tables, which cannot guarantee global uniqueness. Therefore, in the embodiment of the present disclosure, adaptive modification is performed by considering the generation method of the table primary key of a single database, so that the primary key is globally unique in distributed storage. This logic of step S220 can be adapted to various actual modification situations.

[0052] In step S230, based on the business logic and response logic of the target application, a distributed read-write transformation strategy corresponding to the read and write operations of the target application is determined and corresponding transformation processing is performed.

[0053] In some embodiments, the business logic includes whether delayed query is supported during data reading; and the response logic includes whether the use of split keys is supported during data reading and writing.

[0054] Considering that in many actual database transformation scenarios, some response logic supports the use of partitioning keys during the read and write process; some response logic supports the use of partitioning keys during the data writing process, but does not support the use of partitioning keys during the data reading process (for read-write separation scenarios); some business logic supports delayed queries, while others do not. For each of these situations, by comprehensively considering the business logic and response logic of the target application, a distributed read and write transformation strategy corresponding to distributed storage is derived and the corresponding transformation processing is performed. Based on the original read and write method pointing to a single database, a transformation is performed to obtain a read and write method for connecting to sharded databases and tables, which is adapted to various business needs and reduces the risk of database upgrades and iterations disrupting the business.

[0055] In some embodiments, in the above step S230, based on the business logic and response logic of the above target application, the distributed read and write transformation strategy corresponding to the read and write operations of the above target application is determined and corresponding transformation processing is performed, including step S230a: when the above response logic indicates that the data reading and writing process supports the use of split keys, determining the transformation strategy corresponding to the distributed read and write operations of the above target application includes: modifying the execution code of the read operation statement or write operation statement of the above target application, and obtaining the target execution code for executing the read operation or write operation for the sharded database and table based on the above split key.

[0056] Figure 3 The schematic diagram schematically shows the architecture of a single database after being transformed into separate libraries and tables according to an embodiment of the present disclosure.

[0057] When both read and write processes support the use of partition keys, the target application's original execution code for read or write operations directed to a single database (e.g., table S in database A) is modified to obtain the target execution code corresponding to the partition key-based read or write operations for the sharded databases and tables, effectively adapting the routing logic for the sharded databases and tables. For example, the SQL statement "update tbl1 setfiled2 (a business query field) = XXX where id = YYY" is converted to the SQL statement with the partition key: "update tbl1 where id = YYY and filed1 (the partition key) = ZZZ."

[0058] In some embodiments, in the above step S230, based on the business logic and response logic of the above target application, the distributed read and write transformation strategy corresponding to the read operation and write operation of the above target application is determined and corresponding transformation processing is performed, and step S230b is also included: when the above response logic indicates that the data reading process does not support the use of split keys, and the above business logic indicates that the data reading process supports delayed queries, determining the transformation strategy corresponding to the distributed read operation of the above target application includes: constructing an architecture for synchronizing data from sharded libraries and tables to the main storage node, and forwarding the query request to the above main storage node for processing.

[0059] For example, in Figure 3 In this example, the ES database (ES is an open source, highly available, and scalable distributed full-text search engine that can provide PB-level near-real-time data storage and retrieval capabilities. In the embodiments of this disclosure, the ES database is used to implement paging queries and support delayed queries) is used as an example of the primary storage node. When the data reading process does not support the use of split keys and the above business logic indicates that the data reading process supports delayed queries, the query request is forwarded to the primary storage node, and the primary storage node executes the data query operation. The primary storage node synchronizes data from the sharded databases and tables, for example Figure 3 The example method of data synchronization based on operation logs (binlog) can generally meet most query requests and achieve accurate queries. When there is a slight delay between the synchronization of data between the main storage node and the shard databases and tables, because the business logic supports delayed queries, for example, the synchronization between the main storage node and the shard databases and tables may be synchronized every 3 hours at night, and every 15 minutes during the day. For nighttime query requests, nighttime delays are supported. If the query results are inaccurate, the query results can be updated based on the updated data the next day. In this way, this architecture meets business needs and can support query transformation of shard databases and tables.

[0060] In some embodiments, in the above step S230, based on the business logic and response logic of the above target application, the distributed read and write transformation strategy corresponding to the read operation and write operation of the above target application is determined and corresponding transformation processing is performed, and step S230c is also included: when the above response logic indicates that the data reading process does not support the use of the split key, and the above business logic indicates that the data reading process does not support delayed query, determining that the transformation strategy corresponding to the distributed read operation of the above target application includes: selecting a cache that supports secondary indexes, and forwarding the query request to the above cache for processing; the leaf nodes of the index tree corresponding to the above secondary index store primary keys instead of data.

[0061] For example, in Figure 3Redis cache (Redis is an open source storage database in the form of key-value pairs written in ANSI C language, supporting the network, memory-based, distributed, and optionally persistent, and provides APIs (application programming interfaces) in multiple languages; in the embodiments of the present disclosure, Redis cache is provided without split keys, and requires high-speed storage of secondary indexes in real-time scenarios) is used as an example of a cache that supports secondary indexes. When the data reading process does not support the use of split keys and the above-mentioned business logic indicates that the data reading process does not support delayed queries, the query request is forwarded to the Redis cache, and the data query operation is performed by the Redis cache.

[0062] A secondary index is a storage method that is different from a primary index. A primary index is an index that stores the index and data together, both stored in the leaf nodes of the same index tree. Generally, primary key indexes are primary indexes.

[0063] The leaf nodes of the secondary index tree store primary keys instead of data; after finding the index, you can get the corresponding primary key and then go back to the primary index to find the data record corresponding to the primary key.

[0064] Since the Redis cache stores the data written in real time and locates the corresponding data records based on the secondary index when querying, real-time query can be achieved.

[0065] In step S240, the above-mentioned database middleware is integrated to synchronously perform read and write operations on the above-mentioned single database and the transformed sub-libraries and sub-tables.

[0066] In some embodiments, in the above method, after synchronously performing read and write operations on the above single database and the transformed sub-libraries and sub-tables, the method may further include: verifying the transformation effects of the above-mentioned sub-libraries and sub-tables.

[0067] The write operation in the above read and write operations includes at least one of the following: a data adding operation, a data modifying operation, and a data deleting operation; the read operation in the above read and write operations includes a data query operation.

[0068] In some embodiments, reference Figure 4 As shown, in the above step S240, the above database middleware is integrated to synchronously perform read and write operations on the above single database and the transformed sub-databases and sub-tables, including:

[0069] Based on the type of the above database middleware, integrate the above database middleware into the above target application, wherein the modified distributed read and write method is constructed as the read and write routing logic of the above database middleware and connected to the modified sharded databases and tables;

[0070] The new data of the above target applications are synchronously written into the above single database and the transformed sub-databases and sub-tables; for example, Figure 3 The shards obtained after the distributed storage transformation are: DB1 to DBN (MySQL is used as the sharding example), where N represents the number of databases and is an exponential value of 2. Based on the original data writing logic for the single database and the data writing routing logic corresponding to the transformed partitioning key (corresponding to the target execution code corresponding to the read or write operations on the sharded databases and tables based on the partitioning key), data can be written synchronously to the single database and the transformed sharded databases and tables.

[0071] The historical data of the above-mentioned target applications are synchronized from the above-mentioned single database to the above-mentioned sub-databases and sub-tables.

[0072] In some embodiments, verifying the effect of the above-mentioned database and table sharding transformation includes:

[0073] Verify the consistency between the writing results of the above-mentioned sharded databases and tables and the writing results of the above-mentioned single database;

[0074] Based on the dynamic data source, verify the consistency of the query results of the above-mentioned sharded databases and tables with the query results of the above-mentioned single database;

[0075] If the read and write results of the sharded databases and tables are consistent with the read and write results of the single database, the transformation of the sharded databases and tables is considered to have passed. The read and write results of the sharded databases and tables include the write results and query results of the sharded databases and tables; the read and write results of the single database include the write results and query results of the single database.

[0076] In an embodiment including the above steps S210 to S240, a reusable and process-unified single database upgrade and transformation method is provided, which determines the corresponding database middleware and sharding key for a single database of a target application; determines the corresponding table primary key transformation strategy based on the generation method of the table primary key of the above single database and performs corresponding transformation processing, so that the primary key is globally unique in the distributed storage; determines the distributed read and write transformation strategy corresponding to the read and write operations of the above target application based on the business logic and response logic of the above target application and performs corresponding transformation processing; obtains the distributed read and write transformation strategy corresponding to the distributed storage by comprehensively considering the business logic and response logic of the target application and performs It should be transformed and processed based on the original read and write method pointing to a single database, and a read and write method for connecting to shard libraries and tables is obtained, which is suitable for various business needs and reduces the risk of damage to the business caused by database upgrades and iterations. After integrating the above-mentioned database middleware, it can be connected to shard libraries and tables based on the transformed read and write method, and the primary keys of shard libraries and tables are globally unique. At the same time, by synchronously performing read and write operations on the above-mentioned single database and the transformed shard libraries and tables, based on the synchronous read and write method, not only can the business execution be satisfied during the business process, but data can also be written or read (or described as data query) in the shard libraries and tables synchronously, which will help to verify the subsequent transformation effect of the shard libraries and tables.

[0077] According to an embodiment of the present disclosure, the above method further includes: when the transformation effect of the above-mentioned sharding database and sharding table is verified to be passed, deleting the synchronous reading and writing logic code during the verification period, and constructing a reading and writing path pointing to the data source of the sharding database and sharding table. Figure 4 As shown, the above method also includes: removing the relevant information of the single library.

[0078] According to the embodiment of the present disclosure, referring to Figure 4 As shown, the method further includes: determining the scale of the sharded databases and tables based on the data increment information and data retention duration corresponding to the business development of the target application; and dividing the sharded databases and tables based on the scale information to obtain the transformed sharded databases and tables. This step may be performed before the step of integrating the database middleware into the target application.

[0079] For example, considering the amount of table data (less than 5 million per table), the number of connections to a single database instance (less than 1,500 per database), business development within a certain period (e.g., within the past three years), and system scalability, the database middleware is selected as Sharding JDBC middleware. Assuming that the daily increment of a single table is 1 million and the data is retained for half a year, 180 (half a year converted into days) × 1 million ÷ 5 million = 36 databases are required, which is greater than 32, so 64 databases are required. The number of connections to a single database instance = the maximum number of connections to a single JVM (Java Virtual Machine) (10) × the number of nodes in a single instance (several sub-databases per instance) × the number of application instances, ensuring that the number of connections to a single database instance is less than 1,500. The scale information of the sub-databases and tables can be obtained as follows: the number of sub-databases is 64 and the number of connections to a single database instance is less than 1,500.

[0080] A second exemplary embodiment of the present disclosure provides an apparatus for database transformation processing.

[0081] Figure 5 The structural block diagram of the device for database transformation processing according to an embodiment of the present disclosure is schematically shown.

[0082] Reference Figure 5 As shown, the database transformation processing apparatus 500 provided by the embodiment of the present disclosure includes: a determination module 501 , a first transformation module 502 , a second transformation module 503 and a processing module 504 .

[0083] The determination module 501 is used to determine the corresponding database middleware and sharding key for a single database of a target application.

[0084] The first transformation module 502 is used to determine a corresponding table primary key transformation strategy and perform corresponding transformation processing according to the generation method of the table primary key of the single database, so that the primary key is globally unique in the distributed storage.

[0085] The second transformation module 503 is used to determine the distributed read-write transformation strategy corresponding to the read and write operations of the target application according to the business logic and response logic of the target application and perform corresponding transformation processing.

[0086] The processing module 504 is used to integrate the database middleware and perform read and write operations on the single database and the transformed sub-databases and sub-tables simultaneously.

[0087] In some embodiments, the processing module 504 is further used to verify the effect of the transformation of the sharded databases and tables.

[0088] According to an embodiment of the present disclosure, the apparatus 500 further includes: a routing adjustment module.

[0089] The above-mentioned routing adjustment module is used to delete the synchronous reading and writing logic code during the verification period when the transformation effect of the above-mentioned sharding database and table is verified to be passed, and to build a reading and writing path pointing to the data source of the sharding database and table.

[0090] According to an embodiment of the present disclosure, the above-mentioned device 500 further includes: a library and table partitioning module.

[0091] The above-mentioned sharding module is used to: determine the scale information of the sharding database and table according to the data increment information and data retention time information corresponding to the business development of the above-mentioned target application; and divide the transformed sharding database and table according to the above-mentioned scale information.

[0092] According to an embodiment of the present disclosure, based on the generation method of the table primary key of the above-mentioned single database, the corresponding table primary key transformation strategy is determined and the corresponding transformation processing is performed, so that the primary key is globally unique in the distributed storage, including: when the generation method of the above-mentioned table primary key is an auto-increment method, determining the corresponding table primary key transformation strategy includes: generating the table primary key based on a global identifier generation algorithm; when the generation method of the above-mentioned table primary key is a non-auto-increment method, there is no need to perform transformation processing on the table primary key.

[0093] According to an embodiment of the present disclosure, for a single database of a target application, the corresponding database middleware and split key are determined, including: presenting middleware options and field options of the data table in the above-mentioned single database in a visual interface; determining the target middleware selected by the user in the above-mentioned middleware options as the database middleware corresponding to the above-mentioned target application; and determining the target field selected by the user in the above-mentioned field options as the split key corresponding to the above-mentioned target application.

[0094] According to an embodiment of the present disclosure, the above-mentioned business logic includes: whether delayed query is supported during data reading; the above-mentioned response logic includes: whether the use of partitioning keys is supported during data reading and writing. Among them, according to the business logic and response logic of the above-mentioned target application, the distributed read and write transformation strategy corresponding to the read and write operations of the above-mentioned target application is determined and corresponding transformation processing is performed, including: when the above-mentioned response logic indicates that the data reading and writing process supports the use of partitioning keys, determining the transformation strategy corresponding to the distributed read and write operations of the above-mentioned target application includes: modifying the execution code of the read operation statement or write operation statement of the above-mentioned target application, and obtaining the target execution code for executing the read operation or write operation for the sharded database and table based on the above-mentioned partitioning key.

[0095] According to an embodiment of the present disclosure, based on the business logic and response logic of the above-mentioned target application, the distributed read and write transformation strategy corresponding to the read operation and write operation of the above-mentioned target application is determined and corresponding transformation processing is performed, which also includes: when the above-mentioned response logic indicates that the data reading process does not support the use of split keys, and the above-mentioned business logic indicates that the data reading process supports delayed queries, determining that the transformation strategy corresponding to the distributed read operation of the above-mentioned target application includes: constructing an architecture for synchronizing data from sharded libraries and tables to the main storage node, and forwarding the query request to the above-mentioned main storage node for processing.

[0096] According to an embodiment of the present disclosure, based on the business logic and response logic of the above-mentioned target application, the distributed read and write transformation strategies corresponding to the read and write operations of the above-mentioned target application are determined and corresponding transformation processing is performed, which also includes: when the above-mentioned response logic indicates that the data reading process does not support the use of split keys, and the above-mentioned business logic indicates that the data reading process does not support delayed queries, determining that the transformation strategy corresponding to the distributed read operations of the above-mentioned target application includes: selecting a cache that supports secondary indexes, and forwarding the query request to the above-mentioned cache for processing; the leaf nodes of the index tree corresponding to the above-mentioned secondary indexes store primary keys rather than data.

[0097] According to the embodiments of the present disclosure, the above-mentioned database middleware is integrated, and the synchronous reading and writing operations on the above-mentioned single database and the transformed sub-libraries and sub-tables include: according to the type of the above-mentioned database middleware, the above-mentioned database middleware is integrated in the above-mentioned target application, wherein the distributed reading and writing mode after the transformation is constructed as the reading and writing routing logic of the above-mentioned database middleware and connected to the transformed sub-libraries and sub-tables; the new data of the above-mentioned target application is synchronously written to the above-mentioned single database and the transformed sub-libraries and sub-tables; the historical data of the above-mentioned target application is synchronized from the above-mentioned single database to the above-mentioned sub-libraries and sub-tables. The transformation effect of the above-mentioned sub-libraries and sub-tables is verified, including: verifying the consistency between the writing results of the above-mentioned sub-libraries and sub-tables and the writing results of the above-mentioned single database; based on the dynamic data source, verifying the consistency between the query results of the above-mentioned sub-libraries and sub-tables and the query results of the above-mentioned single database; when the reading and writing results of the sub-libraries and sub-tables are consistent with the reading and writing results of the above-mentioned single database, it is deemed that the transformation effect of the above-mentioned sub-libraries and sub-tables has been verified.

[0098] In this embodiment, based on the above-mentioned device 500, a reusable, process-unified single database upgrade and transformation method is provided, and users (such as developers or business personnel of the target application) can implement database upgrades and transformations by interacting with the device 500. Because when determining the distributed read and write transformation logic in the above-mentioned device 500, the distributed read and write transformation strategy corresponding to the distributed storage is obtained by comprehensively considering the business logic and response logic of the target application and performing corresponding transformation processing, the transformation is performed on the basis of the original read and write method pointing to a single database, and a read and write method for connecting to sharded libraries and tables is obtained, which is adapted to various business needs, thereby reducing the risk of database upgrade iterations disrupting the business. Moreover, after integrating the above-mentioned database middleware, it can be connected to the sub-libraries and tables based on the transformed reading and writing methods, and the primary keys of the sub-libraries and tables are globally unique; at the same time, by synchronously performing reading and writing operations on the above-mentioned single database and the transformed sub-libraries and tables, and verifying the transformation effects of the above-mentioned sub-libraries and tables, based on the synchronous reading and writing method, not only can the execution of the business be met during the business process, but data can also be written or read (or described as data query) in the sub-libraries and tables synchronously, which is helpful for subsequent verification of the transformation effects of the sub-libraries and tables.

[0099] For more details of the above-mentioned device 500, please refer to the relevant description of the first embodiment and the content about the device 101 in the system architecture 100, which will not be repeated here.

[0100] Any number of the functional modules included in the above-mentioned device 500 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. At least one of the functional modules included in the device 500 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware, or in an appropriate combination of any of them. Alternatively, at least one of the functional modules included in the device 500 can be at least partially implemented as a computer program module, which can perform the corresponding function when the computer program module is executed.

[0101] A third exemplary embodiment of the present disclosure provides an electronic device.

[0102] Figure 6 The structural block diagram of the electronic device provided by the embodiment of the present disclosure is schematically shown.

[0103] Reference Figure 6 As shown, the electronic device 600 provided by the embodiment of the present disclosure includes a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602 and the memory 603 communicate with each other through the communication bus 604; the memory 603 is used to store computer programs; the processor 601 is used to implement the above-mentioned method of database transformation processing when executing the program stored in the memory.

[0104] A fourth exemplary embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned database transformation method.

[0105] The computer-readable storage medium may be included in the device or apparatus described in the above embodiments, or may exist independently without being incorporated into the device or apparatus. The computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the method according to the embodiments of the present disclosure is implemented.

[0106] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as, but not limited to, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0107] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, and storage of user personal information involved in the technical solutions provided by the embodiments of this disclosure all comply with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken with respect to user personal information to prevent unauthorized access to user personal information data and to maintain the security of user personal information, network security, and national security.

[0108] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0109] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not to be limited to the embodiments shown herein, but is to be construed in the broadest manner consistent with the principles and novel features claimed herein.

Claims

1. A method for database transformation processing, characterized in that: include: For the single database of the target application, determine the corresponding database middleware and sharding key; According to the generation method of the table primary key of the single database, determine the corresponding table primary key transformation strategy and perform corresponding transformation processing to make the primary key globally unique in the distributed storage; Determine the distributed read / write transformation strategy corresponding to the read and write operations of the target application and perform corresponding transformation processing based on the business logic and response logic of the target application; The database middleware is integrated to synchronously perform read and write operations on the single database and the transformed sub-databases and sub-tables.

2. The method according to claim 1, characterized in that The business logic includes: whether delayed query is supported during data reading; the response logic includes: whether the use of split keys is supported during data reading and writing; Wherein, according to the business logic and response logic of the target application, the distributed read-write transformation strategy corresponding to the read operation and write operation of the target application is determined and the corresponding transformation processing is performed, including: When the response logic indicates that the data reading and writing process supports the use of partitioning keys, determining the transformation strategy corresponding to the distributed reading and writing operations of the target application includes: modifying the execution code of the read operation statement or write operation statement of the target application to obtain the target execution code for executing the read operation or write operation for the sharded library and table based on the partitioning key.

3. The method according to claim 2, characterized in that Determining, based on the business logic and response logic of the target application, distributed read / write transformation strategies corresponding to the read and write operations of the target application and performing corresponding transformation processing, further comprising: When the response logic indicates that the data reading process does not support the use of split keys and the business logic indicates that the data reading process supports delayed queries, determining the transformation strategy corresponding to the distributed read operation of the target application includes: building an architecture for synchronizing data from sharded libraries and tables to the main storage node, and forwarding the query request to the main storage node for processing.

4. The method according to claim 2, characterized in that Determining, based on the business logic and response logic of the target application, distributed read / write transformation strategies corresponding to the read and write operations of the target application and performing corresponding transformation processing, further comprising: When the response logic indicates that the data reading process does not support the use of split keys and the business logic indicates that the data reading process does not support delayed queries, determining the transformation strategy corresponding to the distributed read operation of the target application includes: selecting a cache that supports secondary indexes and forwarding the query request to the cache for processing; the leaf nodes of the index tree corresponding to the secondary index store primary keys instead of data.

5. The method according to claim 1, wherein The method further includes: verifying the transformation effect of the sub-library and sub-table; The integration of the database middleware to synchronously read and write the single database and the transformed sub-databases and sub-tables includes: Integrate the database middleware into the target application based on the type of the database middleware, wherein the modified distributed read and write method is constructed as the read and write routing logic of the database middleware and connected to the modified sharded libraries and tables; New data for the target application is synchronously written to the single database and the transformed sub-databases and sub-tables; Synchronize the historical data of the target application from the single database to the sub-databases and sub-tables; The verification of the transformation effect of the sharding of databases and tables includes: Verifying the consistency between the writing results of the sub-libraries and sub-tables and the writing results of the single database; Based on the dynamic data source, verify the consistency of the query results of the sub-libraries and sub-tables with the query results of the single database; If the read and write results of the sub-library and sub-table are consistent with the read and write results of the single database, it is deemed that the transformation effect of the sub-library and sub-table has been verified; the read and write results include write results and query results.

6. The method according to claim 5, characterized in that Also includes: If the transformation effect of the sharded database and table is verified to be successful, the logic code used for synchronous reading and writing during the verification period is deleted, and a reading and writing path pointing to the data source of the sharded database and table is constructed.

7. The method according to claim 5, characterized in that Also includes: Determine the scale of sharding databases and tables based on the data increment information and data retention duration information corresponding to the business development of the target application; According to the scale information, the transformed sub-libraries and sub-tables are obtained.

8. The method according to claim 1, characterized in that Based on the generation method of the table primary key of the single database, determine the corresponding table primary key transformation strategy and perform corresponding transformation processing to make the primary key globally unique in the distributed storage, including: In a case where the table primary key is generated in an auto-increment manner, determining a corresponding table primary key transformation strategy includes: generating the table primary key based on a global identifier generation algorithm.

9. The method according to any one of claims 1 to 8, characterized in that For the single database of the target application, determine the corresponding database middleware and sharding key, including: Presenting middleware options and field options of the data table in the single database in a visual interface; Determine the target middleware selected by the user in the middleware options as the database middleware corresponding to the target application; The target field selected by the user in the field option is determined as the split key corresponding to the target application.

10. A device for database transformation processing, characterized in that: include: A determination module is used to determine the corresponding database middleware and sharding key for a single database of the target application; A first transformation module is configured to determine a corresponding table primary key transformation strategy and perform corresponding transformation processing based on a generation method of a table primary key of the single database, so that the primary key is globally unique in the distributed storage; A second transformation module is used to determine the distributed read and write transformation strategy corresponding to the read and write operations of the target application according to the business logic and response logic of the target application and perform corresponding transformation processing; The processing module is used to integrate the database middleware and synchronously perform read and write operations on the single database and the transformed sub-databases and sub-tables.

11. An electronic device, characterized in that: The processor, the communication interface, the memory and the communication bus are connected to each other via the communication bus. Memory for storing computer programs; A processor, configured to implement the method according to any one of claims 1 to 9 when executing a program stored in a memory.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images