Database double-writing track-merging operation method and device in domestic database switching environment
Through the database dual-write and unblocking operation method, the business continuity problem during database migration in the electronic banking system is solved, and the verification and switching of domestic databases are realized, ensuring the stability and syntax compatibility of the application.
Patent Information
- Application Number
- CN202411418589.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-12
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-10-12
AI Technical Summary
During the database migration process, the traditional network routing layer data mirroring convergence solution cannot meet the convergence requirements of the electronic banking system, and domestic databases have syntactic dialect characteristics, resulting in unpredictable impacts during the migration process, affecting business continuity.
A database dual write-in convergence operation method is adopted. By intercepting database components, multi-threaded asynchronously initiates domestic database switching operations, and uses link one and link two to perform database dual writes, and different SQLs identified by the same SQL are identified. The statement is executed in conjunction with the running indicators of the target database in the production environment to verify whether the target database meets the standard of switching source databases.
It realizes verification of domestic databases in production environment, reduces the unpredictable impact of domestic databases on applications, ensures verification and syntax dialect characteristics during the switching process of target databases and source databases, and provides a reliable switching for domestic databases. plan.
Smart Images

Figure CN118939637A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method and device for dual-writing and parallel operation of a database in a domestic database switching environment, specifically a method and device for dual-writing and parallel operation of a database in a domestic trustworthy database switching environment for electronic banking, belonging to the field of database technology. Background Art
[0002] With the advent of the information age, data has become an important asset for enterprises. As a data storage and management platform, the security, stability and efficiency of the database are crucial to the business continuity and development of the enterprise.
[0003] Database migration can achieve the goal of data security and independent control. Business continuity is a crucial aspect of domestic database migration, directly impacting the normal operation of the business and customer experience. Business continuity refers to the ability of an enterprise to maintain or quickly resume operations in the face of various unexpected events. Improper handling during database migration can lead to business interruptions, impacting the normal operations of the enterprise and customer trust.
[0004] Currently, due to the inherent characteristics of electronic banking systems, such as session state, security authentication, and random factors, traditional data mirroring solutions at the network routing layer cannot meet the integration requirements of electronic banking systems. Furthermore, domestic databases often have their own grammatical dialects, and applications in the integration and adaptation of domestic databases must also be able to support multiple database dialects. To reduce the unpredictable impact of domestic databases on applications in production environments, the present invention proposes a method and apparatus for database dual-write integration in an electronic banking-oriented domestic database switching environment. Summary of the Invention
[0005] In order to solve the above problems, the present invention proposes a database dual-write and parallel operation method and device in a domestic database switching environment for electronic banking, which can verify domestic databases in a production environment and reduce the unpredictable impact of domestic databases on applications.
[0006] The technical solution adopted by the present invention to solve its technical problems is: In a first aspect, an embodiment of the present invention provides a method for database dual-write and parallel operation in a domestic database switching environment, comprising the following steps: Step S1: Based on the database operation request, the domestic database switching operation is initiated asynchronously in a multi-threaded manner according to the established configuration by intercepting the database component, and database dual writing is performed using both link 1 and link 2. Link 1 is when the program is running in the production environment. When the database operation is called, the main thread normally performs the database operation in the source library and records the database operation log normally. Link 2 is when the program is running in the production environment. When the database operation is called, the AOP interceptor intercepts the database operation and performs asynchronous processing in the parallel thread. Step S2, parallel execution of different SQL statements with the same SQL identifier during the process of switching the electronic banking system to the domestic database; Step S3: The target database is persistently run in the production environment, and the operating indicators of the target database in the production environment are continuously collected. The indicators of the target database running in the production environment are measured with persistent data as the measurement standard for whether the target database meets the switching source database. The target database is verified. The operating indicators include coverage, abnormal error indicators, performance indicators, data consistency indicators and discard rate indicators.
[0007] As a possible implementation method of this embodiment, in step S1, the dual write operation is not performed when the domestic database operation does not hit the configuration item; during the dual write operation of the database parallel operation, all asynchronously initiated target database operations are completed using independent transactions, query transactions are executed using a shared thread pool, and non-query transactions use a dedicated thread pool, so that the database operations executed in the parallel thread are consistent with the execution order of the main thread database operations, and the execution results are written to a separate log file.
[0008] As a possible implementation of this embodiment, the link 2 is specifically: The interceptor determines whether the database operation requires asynchronous dual-write operation. If it is necessary, the interceptor initiates the asynchronous database operation instruction. After initiating the asynchronous operation, multiple threads in the thread pool begin to execute database operations on the target database and separately record the log of this asynchronous operation. The separate log of this asynchronous operation is used to collect the operation status of the target database in the production environment. Long-term execution and recording serve as a basis for measuring whether the target database can be switched to the source database.
[0009] As a possible implementation of this embodiment, step S1 includes the following steps: Step S11, receiving a database operation request; Step S12: Add database dual-write configuration. The database dual-write configuration includes data source information configuration and a dual-write test switch configuration. The data source information configuration is the connection configuration information of the dual-write test target library. The dual-write test switch configuration determines whether to enable the dual-write test. Two configuration modes, configuration file and configuration center, are supported. When using the configuration center, configuration item modifications can take effect in real time. Step S13: Create a database parallel dual-write interceptor in the project to intercept the unified entry for database execution operations. It intercepts all database operations in the business library through aspects, identifies the database operation type, clones the database operation parameters, and saves snapshot information of the database execution site. This allows Link 1 and Link 2 to perform the same database operations, including query operations and add, delete, and modify operations. Step S14: Start an asynchronous thread for the data source with database merge and dual write enabled, and asynchronously perform the database merge operation. The asynchronous thread tasks include timeout monitoring of the current database operation and database merge execution operation. The asynchronous thread outputs the data source used by the current database operation, execution time, connection acquisition time, execution result status display, exception information, thread pool name, and SQL type; Step S15: After the asynchronous operation is initiated, multiple threads in the thread pool begin to execute the database operation of the target database and separately record the log of this asynchronous operation.
[0010] As a possible implementation method of this embodiment, in step S14, a public thread pool is used to perform parallel operations on the query operations of the main business library, and an exclusive thread pool is used to perform execution operations on the addition, deletion, and modification operations of the main business library. There is only one working thread in the exclusive thread pool. When the parallel task is executed, other tasks are first put into the queue and executed in the order of entry into the queue.
[0011] As a possible implementation of this embodiment, step S2 includes the following steps: Step S21: During the process of switching the electronic banking system to the domestic database, standard SQL identifiers are used for normal business development. The format of standard SQL identifiers is: main table name, operation name, SQL statement, and function description. Step S22: Configure the database name in the data source and declare the data source list supported by the current system; Step S23: Configure the routing strategy for the merge test in the data source to support simultaneous merge verification of multiple libraries according to different conditions; Step S24, set the SQL identifier of the database configured in step S22, the format of which is: main table name, operation name, SQL statement, function description and data source identifier. When performing business development, use the original SQL statement to perform database operations. The state framework automatically identifies the database and selects the corresponding SQL statement for execution.
[0012] As a possible implementation of this embodiment, step S3 includes the following steps: Step S31: Dynamically compare the operating indicators in the dynamic library of the production environment, using the source database and the target database as the comparison objects, and compare the log contents of the dynamic execution of the two databases to draw preliminary conclusions; Step S32: Based on the preliminary conclusion, a static comparison is performed in the static library. SQL statements with inconsistent execution results or that do not meet the index requirements in the dynamic comparison results are placed in the static library and compared again to verify whether they are caused by other operations in the dynamic library. Step S33 : perform a dynamic and static persistence comparison between the target database and the source database, compare the database structure and the number of SQL affected rows, form a phased comparative analysis result, and determine whether the target database and the source database can be switched based on the comparative analysis result.
[0013] As a possible implementation of this embodiment, in step S3, the coverage index is that the total coverage of the source database and the target database in the production environment running in parallel does not exceed 70%; The abnormal error indicator is that the number of abnormal errors reported by the target database in the production environment is 0; The performance indicators are that the overall SQL execution time in the target database in the production environment is within 50% higher than that in the source database, the average time consumption of the interface increases within 15%, and the application can support 8500 TPS; The data consistency indicator is that the SQL execution results of the source database and the target database in the production environment are consistent; The discard rate indicator is that the SQL execution discard rate of the target database in the production environment is 0, and there is no backlog in the update thread pool.
[0014] In a second aspect, an embodiment of the present invention provides a database dual-write parallel operation device in a domestic database switching environment, comprising: The database dual-write module is used to initiate a domestic database switching operation in accordance with the established configuration and multi-thread asynchronously by intercepting database components according to database operation requests, and adopts two methods, link one and link two, to perform database dual-write. In link one, when the program is running in the production environment, when the database operation is called, the main thread normally performs the database operation in the source library and records the database operation log normally. In link two, when the program is running in the production environment, when the database operation is called, the AOP interceptor intercepts the database operation and performs asynchronous processing in the parallel thread. The SQL parallel execution module is used to parallel execute different SQL statements with the same SQL identifier during the process of switching the electronic banking system to the domestic database; The target database verification module is used to continuously collect the target database's operating indicators in the production environment by persistently running the target database in the production environment. The indicators of the target database running in the production environment are measured with persistent data as the criterion for whether the target database meets the criteria for switching to the source database, and the target database is verified. The operating indicators include coverage, abnormal error indicators, performance indicators, data consistency indicators and discard rate indicators.
[0015] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate through the bus, and the processor executes the machine-readable instructions to perform the steps of the database dual-write and parallel operation method in any domestic database switching environment as described above.
[0016] In a fourth aspect, an embodiment of the present invention provides a storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the database dual-write and parallel operation method in any of the above-mentioned domestic database switching environments are executed.
[0017] The beneficial effects of the technical solutions of the embodiments of the present invention are as follows: A method for parallel operation of database dual-write in a domestic database switching environment, according to a technical solution of an embodiment of the present invention, comprises the following steps: step S1, based on a database operation request, asynchronously initiating a domestic database switching operation in accordance with a predetermined configuration and multi-threaded by intercepting database components, and performing database dual-write using both link one and link two; step S2, paralleling the execution of different SQL statements with the same SQL identifier during the process of switching an electronic banking system to a domestic database; and step S3, continuously collecting operating indicators of the target database in the production environment by persistently running the target database in the production environment, using the persistent data to measure the indicators of the target database running in the production environment as a criterion for determining whether the target database meets the criteria for switching to a source database, and thereby verifying the target database. In the process of switching the domestic database in the electronic banking system, the present invention adopts the method of dual-write parallel operation of the database to realize the verification of the domestic database in the production environment, and performs the operation of single-track switching of the domestic database through configuration, thereby reducing the unpredictable impact of the domestic database on the application; the domestic database is used as the target database, and the foreign database to be replaced is the source database; for each database operation in the production environment, in addition to the normal source database operation, the target database being paralleled simultaneously initiates a target database operation asynchronously, and the parallel operation of the asynchronous database does not affect the final result of the business in the production environment. The persistent data is only used for comparison and verification and does not affect the source database data. The present invention adopts the method of dual-write parallel operation of the source database and the target database to ensure that the target database is fully verified during the switching process of the target database and the source database, and at the same time ensures that the application development during the switching process can take into account the grammatical dialect characteristics of the target database and the source database, providing a reliable switching solution for the domestic database.
[0018] The database dual-write and parallel operation device in a domestic database switching environment according to the technical solution of an embodiment of the present invention has the same beneficial effects as the database dual-write and parallel operation method in a domestic database switching environment according to the technical solution of an embodiment of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 This is a flow chart of a method for running a database dual-write parallel track in a domestic database switching environment according to an exemplary embodiment; Figure 2 This is a schematic diagram showing the structure of a database dual-write and parallel operation device in a domestic database switching environment according to an exemplary embodiment; Figure 3 This is a flowchart showing a specific implementation of database dual writing according to an exemplary embodiment; Figure 4 is a flowchart showing a database merging operation according to an exemplary embodiment; Figure 5This is a flowchart showing a parallel execution of SQL statements according to an exemplary embodiment; Figure 6 The present invention is a flowchart showing a comparison and verification between a target and a source database according to an exemplary embodiment. DETAILED DESCRIPTION
[0020] In order to more clearly illustrate the technical features of the present invention, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings.
[0021] like Figure 1 As shown, an embodiment of the present invention provides a method for running a database dual-write parallel track in a domestic database switching environment, comprising the following steps: Step S1: Based on the database operation request, the domestic database switching operation is initiated asynchronously in a multi-threaded manner according to the established configuration by intercepting the database component, and database dual writing is performed using both link 1 and link 2. Link 1 is when the program is running in the production environment. When the database operation is called, the main thread normally performs the database operation in the source library and records the database operation log normally. Link 2 is when the program is running in the production environment. When the database operation is called, the AOP interceptor intercepts the database operation and performs asynchronous processing in the parallel thread. Step S2, parallel execution of different SQL statements with the same SQL identifier during the process of switching the electronic banking system to the domestic database; Step S3: The target database is persistently run in the production environment, and the operating indicators of the target database in the production environment are continuously collected. The indicators of the target database running in the production environment are measured with persistent data as the measurement standard for whether the target database meets the switching source database. The target database is verified. The operating indicators include coverage, abnormal error indicators, performance indicators, data consistency indicators and discard rate indicators.
[0022] As a possible implementation method of this embodiment, in step S1, the dual write operation is not performed when the domestic database operation does not hit the configuration item; during the dual write operation of the database parallel operation, all asynchronously initiated target database operations are completed using independent transactions, query transactions are executed using a shared thread pool, and non-query transactions use a dedicated thread pool, so that the database operations executed in the parallel thread are consistent with the execution order of the main thread database operations, and the execution results are written to a separate log file.
[0023] As a possible implementation of this embodiment, the link 2 is specifically: The interceptor determines whether the database operation requires asynchronous dual-write operation. If it is necessary, the interceptor initiates the asynchronous database operation instruction. After initiating the asynchronous operation, multiple threads in the thread pool begin to execute database operations on the target database and separately record the log of this asynchronous operation. The separate log of this asynchronous operation is used to collect the operation status of the target database in the production environment. Long-term execution and recording serve as a basis for measuring whether the target database can be switched to the source database.
[0024] As a possible implementation of this embodiment, step S1 includes the following steps: Step S11, receiving a database operation request; Step S12: Add database dual-write configuration. The database dual-write configuration includes data source information configuration and a dual-write test switch configuration. The data source information configuration is the connection configuration information of the dual-write test target library. The dual-write test switch configuration determines whether to enable the dual-write test. Two configuration modes, configuration file and configuration center, are supported. When using the configuration center, configuration item modifications can take effect in real time. Step S13: Create a database parallel dual-write interceptor in the project to intercept the unified entry for database execution operations. It intercepts all database operations in the business library through aspects, identifies the database operation type, clones the database operation parameters, and saves snapshot information of the database execution site. This allows Link 1 and Link 2 to perform the same database operations, including query operations and add, delete, and modify operations. Step S14: Start an asynchronous thread for the data source with database merge and dual write enabled, and asynchronously perform the database merge operation. The asynchronous thread tasks include timeout monitoring of the current database operation and database merge execution operation. The asynchronous thread outputs the data source used by the current database operation, execution time, connection acquisition time, execution result status display, exception information, thread pool name, and SQL type; Step S15: After the asynchronous operation is initiated, multiple threads in the thread pool begin to execute the database operation of the target database and separately record the log of this asynchronous operation.
[0025] As a possible implementation method of this embodiment, in step S14, a public thread pool is used to perform parallel operations on the query operations of the main business library, and an exclusive thread pool is used to perform execution operations on the addition, deletion, and modification operations of the main business library. There is only one working thread in the exclusive thread pool. When the parallel task is executed, other tasks are first put into the queue and executed in the order of entry into the queue.
[0026] As a possible implementation of this embodiment, step S2 includes the following steps: Step S21: During the process of switching the electronic banking system to the domestic database, standard SQL identifiers are used for normal business development. The format of standard SQL identifiers is: main table name, operation name, SQL statement, and function description. Step S22: Configure the database name in the data source and declare the data source list supported by the current system; Step S23: Configure the routing strategy for the merge test in the data source to support simultaneous merge verification of multiple libraries according to different conditions; Step S24, set the SQL identifier of the database configured in step S22, the format of which is: main table name, operation name, SQL statement, function description and data source identifier. When performing business development, use the original SQL statement to perform database operations. The state framework automatically identifies the database and selects the corresponding SQL statement for execution.
[0027] As a possible implementation of this embodiment, step S3 includes the following steps: Step S31: Dynamically compare the operating indicators in the dynamic library of the production environment, using the source database and the target database as the comparison objects, and compare the log contents of the dynamic execution of the two databases to draw preliminary conclusions; Step S32: Based on the preliminary conclusion, a static comparison is performed in the static library. SQL statements with inconsistent execution results or that do not meet the index requirements in the dynamic comparison results are placed in the static library and compared again to verify whether they are caused by other operations in the dynamic library. Step S33 : perform a dynamic and static persistence comparison between the target database and the source database, compare the database structure and the number of SQL affected rows, form a phased comparative analysis result, and determine whether the target database and the source database can be switched based on the comparative analysis result.
[0028] As a possible implementation of this embodiment, in step S3, the coverage index is that the total coverage of the source database and the target database in the production environment running in parallel does not exceed 70%; The abnormal error indicator is that the number of abnormal errors reported by the target database in the production environment is 0; The performance indicators are that the overall SQL execution time in the target database in the production environment is within 50% higher than that in the source database, the average time consumption of the interface increases within 15%, and the application can support 8500 TPS (Transactions Per Second). The data consistency indicator is that the SQL execution results of the source database and the target database in the production environment are consistent; The discard rate indicator is that the SQL execution discard rate of the target database in the production environment is 0, and there is no backlog in the update thread pool.
[0029] like Figure 2 As shown, an embodiment of the present invention provides a database dual-write and parallel operation device in a domestic database switching environment, comprising: The database dual-write module is used to initiate a domestic database switching operation in accordance with the established configuration and multi-thread asynchronously by intercepting database components according to database operation requests, and adopts two methods, link one and link two, to perform database dual-write. In link one, when the program is running in the production environment, when the database operation is called, the main thread normally performs the database operation in the source library and records the database operation log normally. In link two, when the program is running in the production environment, when the database operation is called, the AOP interceptor intercepts the database operation and performs asynchronous processing in the parallel thread. The SQL parallel execution module is used to parallel execute different SQL statements with the same SQL identifier during the process of switching the electronic banking system to the domestic database; The target database verification module is used to continuously collect the target database's operating indicators in the production environment by persistently running the target database in the production environment. The indicators of the target database running in the production environment are measured with persistent data as the criterion for whether the target database meets the criteria for switching to the source database, and the target database is verified. The operating indicators include coverage, abnormal error indicators, performance indicators, data consistency indicators and discard rate indicators.
[0030] like Figure 3 As shown, the specific process of performing database dual-write and parallel operation using the database dual-write and parallel operation device of the present invention is as follows.
[0031] 1. Database double writing.
[0032] Database dual write can be divided into link 1 and link 2, where: Link 1 is when a program is running in a production environment. When a database operation is called, the main thread performs the database operation normally in the source database and records the database operation log normally.
[0033] Link 2 is when the program is running in the production environment. When a database operation is called, the AOP interceptor intercepts the database operation and performs asynchronous processing in other threads.
[0034] like Figure 4 As shown, the database double writing process of the present invention includes the following steps.
[0035] S11, added database parallel dual-write configuration; Configuration is divided into data source information configuration and track test switch configuration. Data source information configuration is the connection configuration information of the track test target library, and track test switch configuration is whether to enable the track test. Two configuration modes are supported: configuration file and configuration center. When using the configuration center, configuration item modifications can take effect in real time.
[0036] S12: Create a database parallel dual-write interceptor in the project to intercept the unified entry for database execution operations; This step is the prerequisite for database dual-write. It intercepts all database operations of the business library through the aspect, identifies the database operation type, clones the parameters of the database operation, saves the snapshot information of the database execution site, and realizes that link 1 and link 2 perform the same database operation.
[0037] S13: Start an asynchronous thread for the data source that has enabled database merge and dual write, and asynchronously perform the database merge operation; After S12 identifies the operations in the main business database, they are divided into two categories based on the operation type. The first category is query operations, which are merged using a common thread pool. The second category is add, delete, and modify operations. To ensure that the execution order of the merged database is consistent with that of the main database, an exclusive thread pool is used for execution. The exclusive thread pool has only one worker thread. When the merge task is executed, other tasks are queued first and executed in the order they were queued, ensuring the consistency of the execution order of change operations.
[0038] S14: After initiating the asynchronous operation, multiple threads in the thread pool begin to execute the database operation of the target database and separately record the log of this asynchronous operation; Asynchronous thread tasks include timeout monitoring for the current database operation and database parallel execution. They output the data source used by the current database operation, execution time, connection acquisition time, execution result status, exception information, thread pool name, and SQL type. After all of these operations are completed, the thread context information is cleared.
[0039] The log of this asynchronous operation is used to collect the operation status of the target database in the production environment. It is executed and recorded over a long period of time to serve as a basis for measuring whether the target database can achieve the switching of the source database.
[0040] 2. SQL statements are executed in parallel.
[0041] During the process of switching to a domestic database in an electronic banking system, SQL statement parallel execution is used to resolve SQL statement incompatibility issues between different databases. Since ORM database operations do not have SQL compatibility issues, this invention only targets named SQL. SQL statement parallel execution precisely refers to the parallel execution of different SQL statements with the same SQL identifier. When a named SQL statement exists that is incompatible with two databases at the same time, SQL needs to be developed separately for that specific database to adapt to this scenario. For different database grammatical dialects, it is necessary to dynamically select compatible SQL statements for execution based on the different databases.
[0042] The term "same SQL identifier" refers to a single logical SQL identifier. Physically, the corresponding SQL identifier and SQL statement must be defined in the Mapper.xml file based on the database type. From a developer's perspective, only one SQL identifier is passed in. At runtime, the corresponding physical SQL identifier is dynamically routed based on the database type, enabling multi-database compatibility during target database integration without impacting the normal operation of existing business logic.
[0043] like Figure 5 As shown, the process of parallel execution of SQL statements in the present invention includes the following steps.
[0044] S21. During the process of switching the electronic banking system to a domestic database, standard SQL identifiers shall be used in normal business development in compliance with development specifications. The format is: main table name, operation name, SQL statement, and function description.
[0045] S22, configure a specific database name in the data source and declare a list of data sources supported by the current system. The data source can include a normally running business library. It also supports configuring multiple libraries to be verified and tested. The configuration item is spring.datasource.multi.ds.dsNames=data source name list.
[0046] S23 configures the routing strategy for the parallel test in the data source, supporting simultaneous parallel verification of multiple databases according to different conditions. For example, institution A uses A TEST DS (A TEST Decision Support Benchmark) for parallel verification, and institution B uses B TEST DS (B TEST Decision Support Benchmark). The number of parallel databases and strategies can be customized.
[0047] S24, SQL identifier for a specific database, the format is: main table name, operation name, SQL statement, function description, data source identifier. Compared with the standard SQL defined in S21, the data source identification field is added to support routing different SQL statements according to different data sources, which solves the problem of inconsistent SQL statements due to different target databases for the same function under parallel testing. This type of routing specification is transparent to developers. When performing business development, the original SQL statement is used for database operations. The state framework automatically identifies the database and selects the corresponding SQL statement for execution.
[0048] After completing the above configuration, if a SQL instruction FUNCTION_A is supported in database A but not in database B, to execute FUNCTION_B in database B, define the SQL identifier as the original standard SQL identifier _B. In this case, SQL will only execute the SQL statement of FUNCTION_B on the data source of database B, and the SQL identifier passed in the program will remain the original standard SQL identifier. This ensures multi-database compatibility during target database integration without affecting the normal operation of existing business logic.
[0049] 3. Target database verification.
[0050] The target database is persistently run in the production environment, and the target database's operating indicators in the production environment are continuously collected. The indicators of the target database running in the production environment are measured with persistent data as the criterion for whether the target database meets the requirements for switching to the source database.
[0051] The target database and source database's merge execution logs are recorded simultaneously to facilitate extraction and comparative analysis. The comparative analysis indicators mainly include coverage, abnormal error indicators, performance indicators, data consistency indicators, and discard rate indicators.
[0052] like Figure 6 As shown, the target database verification process of the present invention includes the following steps.
[0053] S31. First, dynamically compare the above indicators in the dynamic library of the production environment (the dynamic library here refers to the database that may change at any time). The source database and the target database are used as the comparison objects. The comparison content is the log content of the dynamic execution of the two, including coverage, abnormal error indicators, performance indicators, data consistency indicators, and discard rate indicators, and draw preliminary conclusions; the coverage indicator is that the total coverage of the source database and the target database in the production environment does not exceed 70%; the abnormal error indicator is that the number of abnormal errors of the target database in the production environment is 0; the performance indicator is that the overall SQL execution time of the target database in the production environment is higher than that of the source database by less than 50%, the average time consumption of the interface increases by less than 15%, and the application can support 8500TPS (Transactions Per Second); the data consistency indicator is that the SQL execution results of the source database and the target database in the production environment are consistent; the discard rate indicator is that the SQL execution discard rate of the target database in the production environment is 0, and there is no backlog in the update thread pool.
[0054] S32. Secondly, based on the preliminary conclusion of the dynamic comparison, a static comparison is performed in the static library (the static library here refers to the developer's private library, which will not cause database changes due to operations of other people). The SQL statements with inconsistent execution results or that do not meet the index requirements in the dynamic comparison results are placed in the static library and compared again to verify whether they are caused by other operations of the dynamic library.
[0055] S33, finally, a dynamic and static persistence comparison is performed between the target database and the source database, comparing the database structure, the number of rows affected by SQL, etc., to form a phased comparative analysis result, and the result is used to determine whether the target database and the source database can be switched.
[0056] An embodiment of the present invention provides an electronic device, including a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate through the bus, and the processor executes the machine-readable instructions to perform the steps of the database dual-write and parallel operation method in any domestic database switching environment as described above.
[0057] Specifically, the above-mentioned memory and processor can be general-purpose memory and processor, which are not specifically limited here. When the processor runs the computer program stored in the memory, it can execute the above-mentioned database dual-write and parallel operation method in the domestic database switching environment.
[0058] Those skilled in the art will understand that the structure of the computer device does not constitute a limitation of the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently.
[0059] In some embodiments, the computer device may also include a touch screen for displaying a graphical user interface (e.g., an application launch interface) and receiving user operations on the graphical user interface (e.g., application launch operations). Specifically, the touch screen may include a display panel and a touch panel. The display panel may be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), or other devices. The touch panel can detect contact or non-contact operations performed by the user on or near it and generate pre-set operation instructions. For example, the user can use a finger, a stylus, or any other suitable object, or an attachment, on or near the touch panel to perform operations. Furthermore, the touch panel may include a touch detection device and a touch controller. The touch detection device detects the user's touch position and posture, detects signals generated by the touch operation, and transmits the signals to the touch controller. The touch controller receives the touch information from the touch detection device, converts it into information that can be processed by the processor, and then transmits it to the processor. The touch controller can also receive and execute commands from the processor. In addition, the touch panel can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave, and any technology developed in the future can also be used to implement the touch panel. Furthermore, the touch panel can cover the display panel, and the user can operate on or near the touch panel covered on the display panel according to the graphical user interface displayed on the display panel. After the touch panel detects the operation on or near it, it transmits it to the processor to determine the user input, and then the processor provides a corresponding visual output on the display panel in response to the user input. In addition, the touch panel and the display panel can be implemented as two independent components or integrated.
[0060] Corresponding to the above-mentioned application startup method, an embodiment of the present invention also provides a storage medium on which a computer program is stored. When the computer program is run by a processor, the steps of the database dual-write and parallel operation method in any domestic database switching environment are executed.
[0061] The startup device of the application provided in the embodiment of the present application can be specific hardware on the device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in the embodiment of the present application are the same as those of the aforementioned method embodiment. For the sake of brief description, for any part not mentioned in the device embodiment, reference can be made to the corresponding content in the aforementioned method embodiment. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.
[0062] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0063] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interface, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.
[0064] Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of these modules may be selected to achieve the purpose of this embodiment based on actual needs.
[0065] In addition, each functional module in the embodiments provided in the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.
[0066] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0067] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.
[0068] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.
Claims
1. A method for running a database dual-write parallel track in a domestic database switching environment, characterized in that: The steps include: Step S1, according to the database operation request, by intercepting the database component, according to the established configuration, multi-threaded asynchronous initiation of the domestic database switching operation, and using link 1 and link 2 to perform database dual writing; the link 1 is the program running in the production environment, when the database operation is called, the main thread normally performs the database operation in the source library, and the database operation log is normally recorded; the link 2 is the program running in the production environment, when the database operation is called, the AOP interceptor intercepts the database operation, and performs asynchronous processing in the parallel thread; Step S2, executing different SQL statements with the same SQL identifier in parallel during the process of switching the electronic banking system to the domestic database; Step S3, adopt the method of persistently running the target database in the production environment, continuously collect the operating indicators of the target database in the production environment, use the persistent data to measure the indicators of the target database running in the production environment as the measurement standard for whether the target database meets the switching source database, and verify the target database. The operating indicators include coverage, abnormal error indicators, performance indicators, data consistency indicators and discard rate indicators.
2. The method for running a database dual-write and parallel track under a domestic database switching environment according to claim 1 is characterized in that: In step S1, the dual write operation is not performed when the domestic database operation does not hit the configuration item; during the dual write operation of the database, all asynchronously initiated target database operations are completed using independent transactions, query transactions are executed using a shared thread pool, and non-query transactions are executed using a dedicated thread pool, so that the database operations executed in the merged thread are consistent with the execution order of the main thread database operations, and the execution results are written into a separate log file.
3. The method for running a database dual-write and parallel track under a domestic database switching environment according to claim 1, characterized in that: The link 2 is specifically: The interceptor determines whether the database operation needs to perform an asynchronous dual-write operation. If it is determined to be necessary, the interceptor initiates an asynchronous database operation instruction. After initiating the asynchronous operation, multiple threads in the thread pool begin to execute database operations on the target database and separately record the log of this asynchronous operation; the separate log of this asynchronous operation is used to collect the operation status of the target database in the production environment, and long-term execution and recording are used as a basis for measuring whether the target database can achieve the switching of the source database.
4. The method for running a database dual-write and parallel track under a domestic database switching environment according to claim 1, characterized in that: The step S1 comprises the following steps: Step S11, receiving a database operation request; Step S12, adding database track dual-write configuration, the database track dual-write configuration includes data source information configuration and track test switch configuration, the data source information configuration is the connection configuration information of the track test target library, the track test switch configuration is whether to turn on the track test, supports two configuration modes of configuration file and configuration center, and supports real-time effectiveness of configuration item modification when using the configuration center; Step S13, create a database parallel dual-write interceptor in the project, intercept the unified entrance of the database execution operation, intercept all database operations of the business library through the aspect, identify the database operation type, clone the parameters of the database operation, save the snapshot information of the database execution site, and realize that link 1 and link 2 perform the same database operation. The database operation type includes query operation and addition, deletion and modification operation; Step S14, start an asynchronous thread for the data source that enables database merge and dual write, asynchronously perform the database merge operation, the asynchronous thread tasks include timeout monitoring of the current database operation and database merge execution operation, the asynchronous thread outputs the data source used by the current database operation, execution time, connection acquisition time, execution result status display, exception information, thread pool name and SQL type; Step S15, after initiating the asynchronous operation, the multiple threads in the thread pool start to execute the database operation of the target database and separately record the log of this asynchronous operation.
5. The method for running a database dual-write and parallel track under a domestic database switching environment according to claim 1, characterized in that: In step S14, the query operation of the main business library is merged using a public thread pool, and the addition, deletion and modification operations of the main business library are executed using an exclusive thread pool. There is only one working thread in the exclusive thread pool. When the merge task is executed, other tasks are first put into the queue and executed in the order of entering the queue.
6. The method for running a database dual-write and parallel track under a domestic database switching environment according to claim 1, characterized in that: The step S2 comprises the following steps: Step S21, during the process of switching the electronic banking system to the domestic database, standard SQL identifiers are used for normal business development. The format of standard SQL identifiers is: main table name, operation name, SQL statement and function description; Step S22, configuring the database name in the data source, and declaring the data source list supported by the current system; Step S23, configuring a routing strategy for the merge test in the data source, supporting simultaneous merge verification of multiple libraries according to different conditions; Step S24, set the SQL identifier of the database configured in step S22, the format of which is: main table name, operation name, SQL statement, function description and data source identifier. When performing business development, the original SQL statement is used for database operation. The state framework automatically identifies the database and selects the corresponding SQL statement for execution.
7. The method for running a database dual-write and parallel track in a domestic database switching environment according to any one of claims 1 to 6, characterized in that: The step S3 comprises the following steps: Step S31, dynamically compare the operation indicators in the dynamic library of the production environment, with the source database and the target database as the comparison objects, and the comparison content is the log content of the dynamic execution of the two, and draw a preliminary conclusion; Step S32: Based on the preliminary conclusion, static comparison is performed in the static library, and SQL statements with inconsistent execution results or not meeting the index requirements in the dynamic comparison results are put into the static library for comparison again to verify whether it is caused by other operations in the dynamic library; Step S33, compare the target database with the source database in terms of dynamic and static persistence, compare the database structure and the number of rows affected by SQL, form a phased comparison analysis result, and determine whether the target database and the source database can be switched based on the comparison analysis result.
8. A database dual-write parallel operation device in a domestic database switching environment, characterized in that: include: The database dual-write module is used to initiate the domestic database switching operation asynchronously in accordance with the established configuration multi-threaded by intercepting the database component according to the database operation request, and adopts two methods, link one and link two, to perform database dual-write; the link one is the program running in the production environment. When the database operation is called, the main thread performs the database operation in the source library normally, and records the database operation log normally; the link two is the program running in the production environment. When the database operation is called, the AOP interceptor intercepts the database operation and performs asynchronous processing in the parallel thread; The SQL parallel execution module is used to parallel execute different SQL statements with the same SQL identifier during the process of switching the electronic banking system to the domestic database; The target database verification module is used to continuously collect the operating indicators of the target database in the production environment by running the target database persistently in the production environment, and use the persistent data to measure the indicators of the target database running in the production environment as the measurement standard for whether the target database meets the switching source database, and verify the target database. The operating indicators include coverage, abnormal error indicators, performance indicators, data consistency indicators and discard rate indicators.
9. An electronic device, characterized in that: It includes a processor, a memory and a bus, the memory stores machine-readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, and the processor executes the machine-readable instructions to perform the steps of the database dual-write and parallel operation method in the domestic database switching environment as described in any one of claims 1-7.
10. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, executes the steps of the method for running a database dual-write and parallel track in a domestic database switching environment as described in any one of claims 1 to 7.
Citation Information
Patent Citations
Database synchronization method, system and device, electronic equipment and medium
CN113094434A
Database writing method, device and equipment based on multi-writing component and storage medium
CN115203262A
Heterogeneous database switching access system and method
CN115934755A
Incremental data synchronization method, system and equipment based on SQL (Structured Query Language) merging
CN118606408A
Collision avoidance in database replication systems
US20020133507A1
Cited By
Multi-shot comparison test method and device, electronic equipment and program product
CN121387752A
