Database migration evaluation method and apparatus, electronic device, and storage medium

Abstract

The application discloses a database migration evaluation method and device, electronic equipment and storage medium, and relates to the technical field of databases. The method comprises the following steps: determining a source database to be migrated, constructing a first test database of the same version as the source database and a second test database of a target type; obtaining business traffic data of the source database in a production environment; performing analysis and structured conversion on the business traffic data to generate a business request sequence with an execution sequence and a logical relationship; performing read-write separation processing on the business request sequence to obtain a read-only business request file and a write operation business request file; simulating real business pressure in the first test database and the second test database, concurrently playing back the read-only business request file in parallel, comparing the results, and after verification, synchronously playing back the write operation business request file in the first test database and the second test database and comparing the results. The application can comprehensively and accurately evaluate the business requests of the source database.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a database migration assessment method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of information technology, enterprise business systems are placing higher demands on database performance, scalability, and cost control. Database migration (such as migrating from traditional relational databases to distributed databases or cloud-native databases) has become an important means of improving system architecture flexibility. However, heterogeneous databases have significant differences in syntax rules, execution engines, function implementations, and optimizer logic, leading to numerous risks during business system migration. The most critical risk stems from the compatibility of the source database's SQL statements with the target database. Therefore, how to assess the compatibility of source database SQL statements with the target database has become a pressing technical problem to be solved in the field of heterogeneous database migration. Summary of the Invention

[0003] This invention provides a database migration assessment method, apparatus, electronic device, storage medium, and computer program product.

[0004] According to one aspect of the present invention, a database migration evaluation method is provided, comprising:

[0005] Identify the source database to be migrated, and construct a first test database of the same version as the source database and a second test database of the target type; wherein, the first test database and the second test database maintain data consistency and are in a static state;

[0006] Obtain the business traffic data of the source database in the production environment;

[0007] The business traffic data is parsed and structurally transformed to generate a sequence of business requests with execution order and logical relationship;

[0008] The business request sequence is subjected to read-write separation processing to obtain a read-only business request file and a write operation business request file;

[0009] Simulate real business pressure in the first test database and the second test database, concurrently replay the read-only business request file and compare the results. After verification, synchronously replay the write operation business request file in the first test database and the second test database and compare the results to determine the database migration evaluation result.

[0010] According to another aspect of the present invention, a database migration assessment apparatus is provided, comprising:

[0011] Identify the source database to be migrated, and construct a first test database of the same version as the source database and a second test database of the target type; wherein, the first test database and the second test database maintain data consistency and are in a static state;

[0012] Obtain the business traffic data of the source database in the production environment;

[0013] The business traffic data is parsed and structurally transformed to generate a sequence of business requests with execution order and logical relationship;

[0014] The business request sequence is subjected to read-write separation processing to obtain a read-only business request file and a write operation business request file;

[0015] Simulate real business pressure in the first test database and the second test database, concurrently replay the read-only business request file and compare the results. After verification, synchronously replay the write operation business request file in the first test database and the second test database and compare the results to determine the database migration evaluation result.

[0016] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0017] At least one processor; and

[0018] A memory that is communicatively connected to at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform the database migration evaluation method of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute and implement the database migration evaluation method of the embodiments of the present invention.

[0021] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps in the above-described method.

[0022] The technical solution of this invention eliminates the interference of initial data differences on evaluation results by constructing a first test database of the same version as the source database and a second test database of the target type, ensuring that the initial data of both is completely consistent and in a static state. This homogeneous benchmark environment ensures that any result differences that occur in subsequent replay tests can be uniquely attributed to differences in database type, rather than initial data or environment configuration issues, providing an objective and reliable comparison benchmark for migration evaluation. By collecting full business traffic data from the production environment, it overcomes the limitations of traditional methods that rely on source database system tables to collect SQL request samples. The business request sequence generated after structured conversion completely retains the original execution order and logical relationships, ensuring that all SQL requests in real business scenarios are included in the evaluation scope, effectively avoiding post-migration risks caused by missing SQL request samples. By splitting business requests into read-only files and write operation files through read-write separation processing, and adopting a read-first approach... The post-write, step-by-step replay strategy allows read-only requests to be executed repeatedly, facilitating multiple verifications of data consistency and performance stability without frequent environment recovery. Furthermore, by simulating real business pressure (rather than low-load or artificially constructed simple requests), it simulates the actual load characteristics during operation, avoiding inaccurate testing due to significant differences between the test and production environments. Write operations are executed synchronously after read-only verification, reducing interference from data pollution caused by write operations on the evaluation. Moreover, write operations can be repeatedly executed through database backup and recovery. This design enhances the flexibility and repeatability of the evaluation process, reduces environment recovery costs, and improves evaluation efficiency through parallel replay of read-only requests.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating a database migration assessment method provided in an embodiment of the present invention;

[0026] Figure 2 This is a flowchart illustrating another database migration assessment method provided in an embodiment of the present invention;

[0027] Figure 3This is a schematic diagram of the structure of a database migration evaluation device provided in an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the database migration evaluation method of this invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] Example 1

[0031] Figure 1 This is a flowchart of a database migration assessment method provided in an embodiment of the present invention. This embodiment is applicable to database migration scenarios. The method can be executed by a database migration assessment device, which can be implemented in hardware and / or software and can be configured in an electronic device.

[0032] like Figure 1 As shown, the database migration evaluation method includes:

[0033] S101. Determine the source database to be migrated, and construct a first test database of the same version as the source database and a second test database of the target type; wherein, the first test database and the second test database maintain data consistency and are in a static state.

[0034] In this embodiment of the invention, the source database refers to the production environment database to be migrated (such as a MySQL production database), which carries real business data and daily operations. Building the first test database includes: using the same version as the source database, restoring the data through a complete backup of the source database to ensure complete consistency with the source database data. Building the second test database involves: using a database of the target migration type (such as migrating from MySQL to PostgreSQL), configuring the same hardware (server specifications, CPU / memory) and network environment as the first test database; and migrating the data from the first test database to the second test database through logical export-import. The data in the first and second test databases are completely consistent and both are in a static state (no external business writes, used only for testing), eliminating the interference of initial data differences on the evaluation results.

[0035] S102. Obtain the business traffic data of the source database in the production environment.

[0036] In this embodiment of the invention, business traffic data refers to all network packets that the business system interacts with the source database in the production environment, including business requests sent by the client (such as SQL requests), responses returned by the database, etc., covering all operations in real business scenarios (such as querying, updating, and transaction processing).

[0037] In some embodiments, acquiring the business traffic data of the source database in the production environment includes: acquiring the business traffic data of the source database in the production environment through a collection tool (e.g., a packet capture tool) pre-deployed in the source database, and persistently storing it. Specifically, the collection tool can acquire business traffic data by directly capturing network packets of the source database port. In addition, if a network traffic probe (such as a switch mirror port, server traffic monitoring, etc.) has been deployed in the environment where the source database is located, existing network packet recording data can be directly reused as business traffic data, without the need for repeated packet capture.

[0038] S103. The business traffic data is parsed and structurally transformed to generate a business request sequence with execution order and logical relationship.

[0039] In this embodiment of the invention, the acquired business traffic data can first be parsed and key elements extracted based on the official protocol of the source database. The parsed key elements are then sorted by their original timestamps, preserving the execution order of business requests (e.g., query-then-update sequence) and logical relationships (e.g., request associations within a transaction), and converted into a structured format to ultimately obtain a business request sequence comprising multiple business requests. It is understood that the business requests involved in this embodiment of the invention refer to SQL requests, that is, database operation requests composed of SQL statements.

[0040] It should be noted that the key elements extracted include the following fields:

[0041] SrcIP: The IP address of the business that issues SQL statements.

[0042] SrcPort: The port number used by the business system to establish a connection with the database when issuing SQL requests. The IP address and port number used by the business system to establish a connection with the database together constitute a unique connection identifier, and all SQL requests are issued through this database connection. In the subsequent simulation evaluation phase, the system will use this connection identifier (IP:Port combination) to maintain the concurrency characteristics and connection mode of the original business requests on the target database (second test database) for replay testing.

[0043] DestIP: The IP address for external database services.

[0044] DestPort: The port number for the database's external services, which, together with the IP address, forms the identifier of the peer of the connection.

[0045] TimeStamp: The timestamp of this service request in the recorded message, which can also be used to help determine the order of requests.

[0046] ReqType: SQL request type, including connection establishment / disconnection (establishing / breaking the connection), transaction start / close, transaction commit / rollback, and data operation (adding / deleting / modifying / querying) and other business request types. This classification system provides a precise basis for interface calls for the replay component in the simulation evaluation phase, ensuring that various SQL operations can be executed accurately according to business scenario requirements.

[0047] STMT_ID: The ID of the SQL statement. When the business request uses the prepare protocol, this ID is used to accurately bind the SQL parameters on the connection to the prepared SQL statement.

[0048] COM_Query: The specific SQL statement parsed from the message.

[0049] ReqRes: Execution result: success / failure.

[0050] SQLTimerWait: The execution time of the SQL statement, which serves as a reference for subsequent evaluation of the SQL's execution efficiency in the simulation environment.

[0051] S104. Perform read-write separation processing on the service request sequence to obtain a read-only service request file and a write operation service request file.

[0052] In this embodiment of the invention, the business request sequence can be split into a read-only business request file and a write operation business request file based on the read and write characteristics of the business requests. The read-only business request file summarizes all read-only business requests, preserving the original timing and concurrency characteristics; the write operation business request file summarizes all write operation business requests, also preserving the timing.

[0053] S105. Simulate real business pressure in the first test database and the second test database, concurrently replay the read-only business request file and compare the results. After verification, synchronously replay the write operation business request file in the first test database and the second test database and compare the results to determine the database migration evaluation result.

[0054] In this embodiment of the invention, simulating real business pressure means replicating the concurrent request volume, request frequency, and timing relationships (such as transaction order and connection mode) of the production environment to make the load characteristics of the test environment consistent with those of production. Concurrency refers to the alternating execution of multiple requests by multiple threads within a single database (such as the first test database or the second test database), simulating the scenario of multiple services accessing the system simultaneously in production. Parallelism means that the first test database and the second test database simultaneously process the same read-only request file, ensuring that both are subjected to the same pressure at the same time dimension, thus guaranteeing the fairness of the comparison.

[0055] In some embodiments, simulating real business pressure in the first test database and the second test database, concurrently replaying the read-only business request file and comparing the results includes the following steps:

[0056] A. Distribute read-only service requests with the same connection identifier in the read-only service request file to the request queue maintained by the same thread.

[0057] In some embodiments, a read-only business request file is parsed to determine the connection identifier of each read-only business request. A consistent hashing algorithm is then used to allocate read-only business requests with the same connection identifier to a request queue maintained by the same thread. Here, a thread refers to an execution unit that processes business requests. In this embodiment, multiple threads process business requests, and dynamic scaling at runtime is supported. For example, the number of threads can be automatically adjusted based on the original business load characteristics. It is understood that allocating requests with the same connection identifier to the same thread's request queue ensures that the execution order of requests within a single connection is consistent with the actual business logic, conforming to the request logic of a single connection in a real scenario. Furthermore, multiple threads simultaneously maintaining request queues for different connections simulates the core characteristics of multiple concurrent connections in a real business scenario (such as multiple users initiating requests simultaneously). Simultaneously, the number of threads can be dynamically adjusted according to the original business load, matching fluctuations in concurrent pressure in a real business scenario (such as an increase in the number of connections during peak periods), ensuring that the pressure intensity is consistent with the real scenario.

[0058] In other embodiments, distributing read-only service requests with the same connection identifier in the read-only service request file to a request queue maintained by the same thread includes: determining the target read-only service requests to be sent within the current time window from the read-only service request file according to the size of a preset time window; wherein the size of the time window can be 1 second, and the current time window is [T, T+1], where T is the current time. Distributing the target read-only service requests with the same connection identifier to the request queue maintained by the same thread; after all threads have finished executing the target read-only service requests within the current time window or are all executing the last target read-only service request, returning to the operation of determining the target read-only service requests to be sent within the current time window from the read-only service request file. Understandably, by dividing the read-only business requests to be executed into a preset time window (such as 1 second), the read-only business requests within the same window are distributed to different threads for parallel processing, which reproduces the time distribution characteristics of multiple concurrent requests in the same time period in real business (such as 100 requests arriving at the same time within a certain 1 second), avoids time disorder during request replay (such as piling up requests from different times), and ensures the authenticity of concurrency in the time dimension.

[0059] In this embodiment, each read-only business request is executed under a timeout control mechanism. The timeout control mechanism configures a timeout parameter for each read-only business request to prevent the read-only business request from hanging and causing the thread and evaluation process to fail to end. The default input parameter of the timeout parameter for each read-only business request is twice the SQLTimerWait value.

[0060] B. By maintaining the request queue, the thread sequentially sends the read-only business requests in the request queue to the first test database and the second test database for execution according to the order of business request execution.

[0061] The execution order of business requests is determined based on the timestamps of read-only business requests in the request queue.

[0062] Understandably, in real-world business scenarios, requests from different connections have a clear temporal order. In step B, the thread executes requests in the queue according to the timestamp order of read-only business requests, ensuring that the relative execution sequence of requests between different connections is consistent with real-world business scenarios. This reproduces the request timeline in a real-world scenario and avoids pressure distortion caused by temporal discrepancies. The thread simultaneously sends requests to the first and second test databases, demonstrating the characteristics of parallel playback. This ensures that both databases accept requests under the same concurrent pressure, providing a fair benchmark for subsequent result comparisons.

[0063] C. Compare the execution results of the first test database with the execution results of the second test database.

[0064] In this embodiment, the execution results of read-only service requests in the first test database and the second test database can be compared one by one to see if they are consistent. For example, the execution status and result set hash values ​​of read-only service requests in the first and second test databases can be compared one by one. If they are consistent, it means that the execution result of the read-only service request in the second test database is the same as the execution result in the source database. This ensures that after the business system is migrated from the source database to a target database of the same type as the second test database, the read-only service requests are more compatible with the target database, thus completing the compatibility test of the read-only service requests. In addition, the execution results of read-only service requests in the first and second test databases include not only the execution status and result set hash values, but also the execution time of the read-only service requests in the first and second test databases. This allows for comparison of the execution time of the same read-only service request in the first and second test databases, thereby determining the database performance.

[0065] It's important to note that if the execution results of read-only business requests are inconsistent between the first and second test databases, this is essentially due to differences in underlying mechanisms such as syntax support, function logic, type handling, and optimizer rules between the heterogeneous test databases. These differences directly reflect the second test database's incompatibility with the business SQL requests from the first test database. The specific cause needs to be identified through log analysis (e.g., syntax errors, function differences, different sorting logic) and targeted optimizations (e.g., rewriting SQL, adjusting the second test database configuration). Otherwise, it will lead to incorrect query results or execution failures after migration. The logs involved here include hierarchically stored summary logs and detailed logs. The detailed logs contain the incrementing sequence number (SQLID), schema identifier, complete SQL text, complete result sets (field values, row count, order) of each read-only business request from both the first and second test databases, environment parameters during execution (e.g., character set, time zone), and the error stack of the second test database (if failure occurred). For example, detailed logs can directly show that the result set returned by the second test database is missing two rows. Combined with the SQL text, it can be found that the problem is caused by differences in function processing in the WHERE condition. Alternatively, the error stack can be used to pinpoint the specific location of the syntax error (such as a unique function name). Summary logs can quickly identify the pattern identifier of abnormal requests, count the inconsistency ratio of this type of read-only business requests, and determine whether it is an isolated problem or a batch compatibility issue (such as a certain type of function being processed abnormally in the second test database).

[0066] In this embodiment of the invention, if each read-only business request passes the comparison test, then consistency and performance analysis are performed on the write operation business requests. It should be noted that unexpected events may occur during the testing process (such as server crashes, network interruptions, program crashes, etc.), which could cause the current write operation business request replay to be interrupted or the results to be unreliable. Therefore, by performing a full backup of the first and second test databases before replaying the write operation business requests, in the event of an unexpected event, the backups can be used to restore both databases to their initial data state before the test, thereby re-executing the write operation business request replay test. This ensures that the entire verification process can be reliably repeated, avoiding test interruption or invalid results due to unexpected events. In other words, the replay of write operation business requests can also be repeated.

[0067] In some embodiments, the write operation service request file is synchronously replayed in the first test database and the second test database, and the results are compared, including steps one to four:

[0068] Step 1: Split the write operation request file according to the database tables involved in the write operation request to obtain multiple sub-files; each sub-file includes a write operation request for a single database table.

[0069] Step 2: For each sub-file, maintain the original timing of write operation requests within that sub-file. That is, the order of all write operation requests for the same database table within the sub-file must be completely consistent with the execution order in the original business traffic. This ensures the consistency of transaction logic and data, and the sequential consistency can recreate the real business pressure scenario.

[0070] Step 3: Control the concurrent execution of the multiple sub-files in the first and second test databases, and record the execution process of each write operation request. This mainly involves recording the incrementing index, execution status, execution time, number of affected rows, and exception stack information for each write operation request. Write operation requests within a single sub-file are executed serially. This improves the efficiency of evaluating write operation requests.

[0071] Step 4: Based on the execution results of each sub-file in the first test database and the execution results in the second test database, perform data consistency and performance comparison analysis; wherein, the execution results include at least the incrementing index, execution status, execution time, number of affected rows, and exception stack information of each write operation business request.

[0072] In some embodiments, based on the execution results of each sub-file in the first test database and the execution results in the second test database, a data consistency and performance comparison analysis is performed, including: using a data comparison tool to perform table-level data consistency verification on the first test database and the second test database; marking database tables with discrepancies as risk tables and analyzing the execution logs corresponding to the risk tables; comparing the execution time of similar write operation business requests in the first test database and the second test database according to the pattern identifier of the write operation business requests, and generating a time consumption comparison trend and quantile difference report.

[0073] In this embodiment of the invention, by constructing a first test database of the same version as the source database and a second test database of the target type, and ensuring that the initial data of both are completely consistent and in a static state, the interference of the initial data difference on the evaluation results is eliminated. This homogeneous benchmark environment ensures that any differences in results during subsequent replay tests can be uniquely attributed to differences in database type, rather than initial data or environment configuration issues, providing an objective and reliable comparison benchmark for migration assessment. By collecting full business traffic data from the production environment, it overcomes the limitations of traditional methods that rely on source database system tables to collect SQL request samples. The business request sequence generated after structured transformation completely preserves the original execution order and logical relationships, ensuring that all SQL requests (i.e., business requests) in real business scenarios are included in the assessment scope, effectively avoiding post-migration risks caused by missing SQL requests. By splitting business requests into read-only files and write operation files through read-write separation, a step-by-step replay strategy of reading first and then writing is adopted: read-only requests can be executed repeatedly, facilitating multiple verifications of data consistency and performance stability without frequent environment recovery; write operations are executed synchronously after read-only verification passes, reducing the interference of data pollution caused by write operations on the assessment; through database backup and recovery, repeated execution tests of write operation business requests can be achieved. This design improves the flexibility and repeatability of the assessment process, reduces environment recovery costs, and improves assessment efficiency by replaying read-only requests in parallel.

[0074] Example 2

[0075] Figure 2 A flowchart of a database migration evaluation method is provided for an embodiment of the present invention. See also... Figure 2 The method includes the following steps:

[0076] S201. Determine the source database to be migrated, and construct a first test database of the same version as the source database and a second test database of the target type; wherein, the first test database and the second test database maintain data consistency and are in a static state.

[0077] S202. Obtain the business traffic data of the source database in the production environment.

[0078] S203. The business traffic data is parsed and structurally transformed to generate a business request sequence with execution order and logical relationship.

[0079] The specific implementation process of steps S201-S203 can be found in the description of the above embodiments, and will not be repeated here.

[0080] S204. Assign an incrementing sequence number to each business request in the business request sequence; perform pattern processing on each business request in the business request sequence to obtain a corresponding patterned statement, and assign the same pattern identifier to business requests with the same patterned statement.

[0081] In this embodiment of the invention, each business request (e.g., an SQL request) is assigned an incrementing sequence number (e.g., SQLID) within the business request sequence. For each business request, a SQL template tool is used to perform pattern processing on the business request to obtain a corresponding patterned statement (digesttext). Business requests with the same patterned statement are assigned the same pattern identifier (e.g., digestid). It is understood that the incrementing sequence number and pattern identifier of the business request can be used for subsequent comparison of execution results between the two databases.

[0082] For example, after processing the business requests "update table1 set a=2 where b=1" and "update table1 set a=3 where b=2" into a pattern, the same patterned statement "update table1 set a=? where b=?" is generated, and the two business requests are assigned the same pattern identifier. This ensures that business requests (SQL statements) with the same pattern have a consistent pattern identifier, while business requests with different patterns correspond to a unique pattern identifier.

[0083] S205. Perform read-write separation processing on the service request sequence to obtain a read-only service request file and a write operation service request file.

[0084] S206. Simulate real business pressure in the first test database and the second test database, concurrently replay the read-only business request file and compare the results. After verification, synchronously replay the write operation business request file in the first test database and the second test database and compare the results to determine the database migration evaluation result.

[0085] In some embodiments, simulating real business pressure in the first test database and the second test database, concurrently replaying the read-only business request file and comparing the results includes the following steps:

[0086] A. Distribute read-only service requests with the same connection identifier in the read-only service request file to the request queue maintained by the same thread.

[0087] In some embodiments, a read-only business request file is parsed to determine the connection identifier of each read-only business request. Read-only business requests with the same connection identifier are allocated to the same request queue maintained by the same thread using a consistent hashing algorithm. Here, a thread refers to an execution unit that processes business requests. In this embodiment of the invention, there are multiple threads that process business requests, and dynamic scaling at runtime is supported. For example, the number of threads can be automatically adjusted according to the original business load characteristics.

[0088] In other embodiments, distributing read-only service requests with the same connection identifier in the read-only service request file to a request queue maintained by the same thread includes: determining the target read-only service requests to be sent within the current time window from the read-only service request file according to the size of a preset time window; wherein the size of the time window can be 1 second, and the current time window is [T, T+1], where T is the current time. Distributing the target read-only service requests with the same connection identifier to the request queue maintained by the same thread; after all threads have finished executing the target read-only service requests within the current time window or are all executing the last target read-only service request, returning to the operation of determining the target read-only service requests to be sent within the current time window from the read-only service request file. It can be understood that requests within each preset time window (e.g., 1 second) are considered concurrent service requests of the same time segment and must be processed simultaneously by each thread within that window period; only when all threads have completed the requests of the current window (or are all processing the last one) will they enter the next window. This mechanism forces all threads to maintain a similar execution pace, preventing any one thread from running too fast or too slow, and ensuring that requests within the same time period must be processed concurrently within the same time window, thus maintaining the synchronization of concurrency. The window time length (e.g., 1 second, 5 seconds) is essentially the time granularity for simulating concurrency. The shorter the length, the higher the time precision of the simulation, and the more accurate the simulation of concurrency pressure, the better it can reproduce the instantaneous characteristics of concurrent requests in real business scenarios.

[0089] B. Using a thread that maintains the request queue, read-only business requests in the request queue are sequentially sent to the first test database and the second test database for execution, according to the order in which the business requests are executed. The execution order of the business requests is determined based on the timestamps of the read-only business requests in the request queue.

[0090] C. Compare the execution results of the first test database with the execution results of the second test database.

[0091] Specifically, the execution status and result set hash value of each read-only service request are compared in the first test database and the second test database according to the incrementing sequence number of the read-only service requests; or, the execution time of similar read-only service requests in the first test database and the second test database is compared according to the pattern identifier of the read-only service requests, and a time consumption comparison trend chart and quantile difference report are generated.

[0092] In this embodiment of the invention, the core of generating a quantile difference report is to quantify the performance differences between the two at different pressure quantiles by statistically analyzing the execution time distribution of similar read-only business requests in the first and second test databases. The specific process is as follows: First, data preparation involves aggregating the execution time data of similar read-only requests. Specifically, the data is grouped by pattern identifier: the pattern identifier (digestID) is a unique identifier for similar SQL business requests. All read-only business requests in the read-only business request file are aggregated by pattern identifier to obtain the execution time list of the first and second test databases corresponding to each pattern identifier. For example, the execution time list corresponding to pattern identifier 1: after aggregation, 1000 execution time records (e.g., [10ms, 15ms, ..., 30ms]) of this type of request in the first test database and 1000 execution time records (e.g., [12ms, 18ms, ..., 40ms]) in the second test database are obtained. Furthermore, execution failures and timeouts (e.g., exceeding a preset threshold) are removed to ensure that all requests participating in the statistics are validly executed, avoiding outliers from interfering with quantile calculations. Further, quantile calculations are performed, primarily generating quantile values ​​for the first and second test databases respectively. Specifically, for each pattern identifier (digestID) corresponding to the time consumption lists of the first and second test databases, key quantiles (typically including P50, P90, and P99, which can be expanded as needed) are calculated, including the following: The time consumption lists are sorted in ascending order (e.g., after sorting the time consumption of the first test database: [8ms, 10ms, 12ms, ..., 50ms]). The index of the corresponding position is calculated according to the quantile definition (index = quantile × total number of samples, rounded or using the nearest neighbor value). For example, P50 (50th percentile) corresponds to the value at the 50th position after sorting. If the sample size is 1000, then the index = 1000 × 50% = 500, and the 500th value is taken; P90 corresponds to an index = 1000 × 90% = 900, and the 900th value is taken; P99 corresponds to an index = 1000 × 99% = 990, and the 990th value is taken. Record the quantiles of the first and second test databases. Further, quantify the performance deviation of the quantiles between the two test databases. Specifically, for each pattern identifier, calculate the difference between the second and first test databases at each quantile, such as calculating absolute or relative differences. Finally, a quantile difference report can be generated based on the calculated quantiles and difference values.

[0093] In this embodiment, when the execution results of a read-only business request differ between the first test database and the second test database, the problem can be located, root cause analysis performed, and high-risk SQL queries identified, providing a concrete basis for migration risk management. Specifically, the difference is first located when the hash values ​​of the execution results from the two databases do not match (i.e., data consistency verification fails). The specific differences are then identified by correlating detailed logs (recording complete result sets, SQL text, etc.) to clarify the specific manifestations of the result set inconsistency. Further, root cause analysis focuses on the differences or anomalies. The complete business request (SQL statement) is retrieved using the incrementing sequence number (SQLID) of the business request, and executed twice in both databases to reproduce the problem and verify whether the difference is stable. If the business request fails to execute, the direct cause and cascading effects of the failure are traced through error stack information (such as syntax errors, permission exceptions, etc.) combined with the context request chain (such as preceding operations within a transaction). Hotspot focusing automatically filters and marks three types of high-risk business requests: 1. Performance degradation: Execution time in the second test database exceeds 120% of that in the first test database. 2. Data inconsistency: Differences exist between the result sets of the two databases. 3. Stability issues: Failure rate varies by more than 5%.

[0094] In this embodiment of the invention, if each read-only service request passes the comparison test, then the write operation service requests are subjected to consistency and performance analysis, as follows.

[0095] In some embodiments, the write operation service request file is synchronously replayed in the first test database and the second test database, and the results are compared, including steps one to four:

[0096] Step 1: Split the write operation request file according to the database tables involved in the write operation request to obtain multiple sub-files; each sub-file includes a write operation request for a single database table.

[0097] Step 2: For each sub-file, maintain the original timing of write operation requests in the sub-file; that is, the order of all write operation requests for the same database table in the sub-file must be completely consistent with the execution order in the original business traffic. This can ensure the consistency of transaction logic and data, and the sequential consistency can restore the real business pressure scenario.

[0098] Step 3: Control the concurrent execution of the multiple sub-files in the first and second test databases, and record the execution process of each write operation request, mainly including the incrementing index, execution status, execution time, number of affected rows, and exception stack information of each write operation request; among them, the write operation requests in a single sub-file are executed serially. This can improve the efficiency of evaluating write operation requests.

[0099] Step 4: Based on the execution results of each sub-file in the first test database and the execution results in the second test database, perform data consistency and performance comparison analysis; wherein, the execution results include at least the incrementing index, execution status, execution time, number of affected rows, and exception stack information of each write operation business request.

[0100] In some embodiments, based on the execution results of each sub-file in the first test database and the execution results in the second test database, data consistency and performance comparison analysis is performed, including: performing table-level data consistency verification on the first test database and the second test database using a data comparison tool; marking database tables with discrepancies as risk tables and analyzing the execution logs corresponding to the risk tables; comparing the execution time of similar write operation business requests in the first test database and the second test database according to the mode identifier of the write operation business requests, and generating a time consumption comparison trend and quantile difference report. The implementation method for generating the quantile difference report is similar to that described above and will not be repeated here.

[0101] In this embodiment of the invention, a complete diagnostic chain is also provided, which is as follows: first, the specific abnormal business request (SQL request) is located from the risk table marked by data consistency verification; then, the complete execution context of the business request (such as timing, concurrency environment, transaction association) is associated; if there is a difference in the execution results of the two databases (such as the number of rows affected), a manual confirmation step is triggered to ensure the accuracy of the difference assessment.

[0102] In this embodiment of the invention, each service request is assigned an incrementing sequence number and a pattern identifier, which facilitates subsequent evaluation and analysis based on the incrementing sequence number and pattern identifier of the service request, thereby improving the efficiency of the evaluation and analysis.

[0103] Example 3

[0104] Figure 3 This is a schematic diagram of a database migration evaluation device provided in an embodiment of the present invention. This embodiment is applicable to database migration scenarios. The device can execute any of the database migration evaluation methods of the present invention. Figure 3 As shown, the database migration evaluation device includes:

[0105] The database determination module 301 is used to determine the source database to be migrated, and to construct a first test database of the same version as the source database and a second test database of the target type; wherein, the first test database and the second test database maintain data consistency and are in a static state;

[0106] Data acquisition module 302 is used to acquire business traffic data of the source database in the production environment;

[0107] Data conversion module 303 is used to parse and structure the business traffic data to generate a business request sequence with execution order and logical relationship;

[0108] Data separation module 304 is used to perform read-write separation processing on the business request sequence to obtain a read-only business request file and a write operation business request file;

[0109] The evaluation module 305 is used to simulate real business pressure in the first test database and the second test database, concurrently replay the read-only business request file and compare the results, and after verification, synchronously replay the write operation business request file in the first test database and the second test database and compare the results to determine the database migration evaluation result.

[0110] In some embodiments, regarding simulating real business pressure and concurrently replaying the read-only business request file in the first test database and the second test database and comparing the results, the evaluation module 305 includes:

[0111] The distribution unit is used to distribute read-only service requests with the same connection identifier in the read-only service request file to the request queue maintained by the same thread.

[0112] An execution unit is configured to send read-only business requests in the request queue to the first test database and the second test database for execution in the order of business request execution, by using a thread that maintains the request queue.

[0113] The comparison unit is used to compare the execution results of the first test database and the execution results of the second test database.

[0114] In some embodiments, when distributing read-only service requests with the same connection identifier in the read-only service request file to a request queue maintained by the same thread, the distribution unit is specifically used for:

[0115] Based on the size of the preset time window, the target read-only service request to be issued within the current time window is determined from the read-only service request file;

[0116] Distribute the target read-only service requests with the same connection identifier to the request queue maintained by the same thread;

[0117] Once all threads have finished executing the target read-only business requests within the current time window, or are all executing the last target read-only business request, return to execute the operation of determining the target read-only business requests to be issued within the current time window from the read-only business request file.

[0118] In some embodiments, a preprocessing module is further included, for:

[0119] Assign an incrementing sequence number to each service request in the service request sequence;

[0120] Each business request in the business request sequence is processed into a pattern to obtain a corresponding pattern statement, and business requests with the same pattern statement are assigned the same pattern identifier.

[0121] Accordingly, the comparison unit is specifically used for:

[0122] Based on the ascending sequence number of each read-only service request, compare the execution status and result set hash value of each read-only service request in the first test database and the second test database; or,

[0123] Based on the pattern identifier of read-only business requests, compare the execution time of similar read-only business requests in the first test database and the second test database, and generate a time comparison trend chart and a quantile difference report.

[0124] In some embodiments, the evaluation module 305 includes the following for synchronously replaying the write operation service request file in the first test database and comparing the results in the second test database:

[0125] The splitting unit is used to split the write operation service request file according to the database tables involved in the write operation service request to obtain multiple sub-files; each sub-file includes a write operation service request for a single database table.

[0126] The holding unit is used to maintain the original timing of write operation service requests in each of the sub-files;

[0127] The control unit is used to control the concurrent execution of the multiple sub-files in the first test database and the second test database; wherein, write operation business requests in a single sub-file are executed serially;

[0128] The analysis unit is used to perform data consistency and performance comparison analysis based on the execution results of each sub-file in the first test database and the execution results in the second test database; wherein, the execution results include at least the incrementing index, execution status, execution time, number of affected rows, and exception stack information of each write operation business request.

[0129] In some embodiments, the analysis unit is specifically used for:

[0130] Using a data comparison tool, table-level data consistency verification was performed on the first test database and the second test database.

[0131] Mark the database tables with discrepancies as risk tables and analyze the execution logs corresponding to the risk tables;

[0132] Based on the pattern identifier of the write operation business request, compare the execution time of similar write operation business requests in the first test database and the second test database, and generate a time comparison trend and quantile difference report.

[0133] In some embodiments, the data acquisition module is specifically used for:

[0134] By using acquisition tools pre-deployed in the source database, business traffic data of the source database in the production environment is obtained and persistently stored.

[0135] The database migration assessment device provided in this embodiment of the invention can execute the database migration assessment method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0136] According to embodiments of the present invention, the present invention also provides an electronic device, a readable storage medium, and a computer program product.

[0137] Example 4

[0138] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the invention described and / or claimed herein.

[0139] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0140] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disks, optical disks, etc.; and communication unit 19, such as network cards, modems, wireless transceivers, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0141] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as performing database migration evaluation methods.

[0142] In some embodiments, the database migration assessment method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the database migration assessment method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the database migration assessment method by any other suitable means (e.g., by means of firmware).

[0143] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0144] Computer programs used to implement the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable database migration evaluation apparatus, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0145] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0146] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0147] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0148] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0149] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0150] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method of database migration assessment, the method comprising: The application comprises: determining a source database to be migrated, and constructing a first test database of the same version as the source database and a second test database of a target type; wherein the first test database and the second test database have the same hardware and network environment, the first test database and the second test database keep data consistent and are in a static state; the static state means that the first test database and the second test database have no business data write except for testing; obtaining business traffic data of the source database in a production environment; parsing and structuring the business traffic data to generate a business request sequence with execution order and logical relationship; performing read-write separation processing on the business request sequence to obtain a read-only business request file and a write operation business request file; simulating real business pressure in the first test database and the second test database, concurrently and in parallel playing back read-only business requests in the read-only business request file and comparing results, and after verification, synchronously playing back write operation business requests in the write operation business request file in the first test database and the second test database and comparing results to determine a database migration evaluation result; wherein concurrency means that multiple read-only business requests are alternately executed in the first test database or the second test database through multiple threads to simulate the scenario of multiple businesses accessing at the same time in production; parallelism means that the first test database and the second test database simultaneously process read-only business requests in the same read-only request file to ensure that they accept the same pressure at the same time dimension. parsing and structuring the business traffic data to generate a business request sequence with execution order and logical relationship, comprising: parsing the business traffic data and extracting key elements; sorting the parsed key elements according to the original timestamp, maintaining the execution order and logical relationship of the business requests, converting to a structured format, and finally obtaining a business request sequence including multiple business requests; wherein a business request refers to a database operation request composed of an SQL statement.

2. The method of claim 1, wherein, the step of simulating real business pressure in the first test database and the second test database, concurrently and in parallel playing back read-only business requests in the read-only business request file and comparing results, comprising: distributing read-only business requests with the same connection identifier in the read-only business request file to a request queue maintained by the same thread; sending read-only business requests in the request queue to the first test database and the second test database for execution in turn according to the business request execution order through the thread maintaining the request queue; comparing the execution results of the first test database and the second test database.

3. The method of claim 2, wherein, the step of distributing read-only business requests with the same connection identifier in the read-only business request file to a request queue maintained by the same thread, comprising: determining target read-only business requests to be dispatched within a current time window from the read-only business request file according to the size of a preset time window. Distribute the read-only service requests with the same connection identifier in the target read-only service request to the same thread-maintained request queue; Return the operation of determining the target read-only service requests in the current time window to be issued from the read-only service request file after all threads execute the target read-only service requests in the current time window or are executing the last target read-only service request.

4. The method of claim 2, wherein, Further comprising: Assign an incremental serial number to each service request in the service request sequence; Patternize each service request in the service request sequence to obtain a corresponding patternized statement, and assign the same pattern identifier to the service requests with the same patternized statement; Correspondingly, compare the execution results of the first test database and the second test database, including: According to the incremental serial number of the read-only service request, compare the execution state and result set hash value of the read-only service request in the first test database and the second test database; or, According to the pattern identifier of the read-only service request, compare the execution time of the same type of read-only service request in the first test database and the second test database, and generate a time consumption comparison trend chart and a quantile difference report.

5. The method of claim 1, wherein, The step of synchronously playing back the write operation service requests in the write operation service request file in the first test database and the second test database and comparing the results, comprising: Split the write operation service request file according to the database table involved in the write operation service request to obtain a plurality of sub-files; each sub-file includes write operation service requests for a single database table; For each sub-file, maintain the original time sequence of the write operation service requests in the sub-file; Control the concurrent execution of the plurality of sub-files in the first test database and the second test database; wherein the write operation service requests in a single sub-file are executed in series; According to the execution results of each sub-file in the first test database and in the second test database, perform data consistency and performance comparison analysis; wherein the execution results at least include the incremental serial number, execution state, execution time, affected row number, and exception stack information of each write operation service request.

6. The method of claim 5, wherein, According to the execution results of each sub-file in the first test database and in the second test database, perform data consistency and performance comparison analysis, including: Perform table-level data consistency verification on the first test database and the second test database through a data comparison tool; Mark the database table with differences as a risk table and analyze the execution log corresponding to the risk table; According to the pattern identifier of the write operation service request, compare the execution time of the same type of write operation service request in the first test database and the second test database, and generate a time consumption comparison trend chart and a quantile difference report.

7. The method of claim 5, wherein, The step of obtaining the business traffic data of the source database in the production environment, comprising: Obtain the business traffic data of the source database in the production environment through a collection tool pre-deployed in the source database, and persistently store it.

8. A database migration assessment apparatus, characterized by, Comprising: A database determining module is configured to determine a source database to be migrated, and construct a first test database of the same version as the source database and a second test database of a target type; wherein the first test database and the second test database have the same hardware and network environment, and the first test database and the second test database keep data consistent and are in a static state; the static state means that the first test database and the second test database have no business data write except for testing; A data acquisition module is configured to acquire business traffic data of the source database in a production environment; A data conversion module is configured to parse and structureally convert the business traffic data to generate a business request sequence with an execution order and a logical relationship; A data separation module is configured to perform read-write separation processing on the business request sequence to obtain a read-only business request file and a write operation business request file; An evaluation module is configured to simulate real business pressure in the first test database and the second test database, concurrently and in parallel replay read-only business requests in the read-only business request file, and compare results, and after verification, synchronously replay write operation business requests in the write operation business request file in the first test database and the second test database, and compare results, to determine a database migration evaluation result; wherein concurrency means that multiple read-only business requests are alternately executed in the first test database or the second test database through multiple threads to simulate a scenario of multiple businesses simultaneously accessing in production; parallelism means that the first test database and the second test database simultaneously process read-only business requests in the same read-only request file to ensure that they accept the same pressure in the same time dimension. The data conversion module is specifically configured to parse the business traffic data and extract key elements, sort the parsed key elements according to original timestamps, maintain the execution order and logical relationship of business requests, convert to a structured format, and finally obtain a business request sequence including multiple business requests; wherein a business request refers to a database operation request composed of an SQL statement.

9. An electronic device, comprising: The computer readable storage medium stores computer instructions for causing the processor to execute the method of any one of claims 1-7 when executed. The computer readable storage medium stores computer instructions for causing the processor to execute the method of any one of claims 1-7 when executed. ​ ​ ​ 10. A computer-readable storage medium, characterized in that, ​

Images