Database migration evaluation method and device, electronic equipment and storage medium

Abstract

The invention discloses a database migration evaluation method and device, electronic equipment and a storage medium, and relates to the technical field of databases. The method comprises the following steps: determining a to-be-migrated source database, and constructing a first test database with the same version as the source database and a second test database with a target type; obtaining service flow data of the source database in the production environment; carrying out analysis and structured conversion on the service flow data to generate a service request sequence with an execution sequence and a logic relationship; performing read-write separation processing on the service request sequence to obtain a read-only service request file and a write operation service request file; real service pressure is simulated in the first test database and the second test database, the read-only service request file is played back concurrently and parallelly, results are compared, and after verification is passed, the write operation service request file is played back synchronously in the first test database and the second test database, and results are compared. According to the invention, the service request of the source database can be comprehensively and accurately evaluated.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a database migration assessment method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of information technology, enterprise business systems are placing higher demands on database performance, scalability, and cost control. Database migration (such as from traditional relational databases to distributed or cloud-native databases) has become a crucial means of enhancing system architecture flexibility. However, heterogeneous databases exhibit significant differences in syntax, execution engines, function implementation, and optimizer logic, leading to numerous risks during business system migration. The most significant risk stems from the compatibility of SQL statements requested by the source database in the target database. Therefore, assessing the compatibility of SQL statements in the source database with those in the target database has become a pressing technical challenge in heterogeneous database migration. Summary of the Invention

[0003] The present invention provides a database migration assessment method, device, electronic device, storage medium and computer program product.

[0004] According to one aspect of the present invention, a database migration assessment method is provided, comprising:

[0005] Determine the source database to be migrated, and build a first test database of the same version as the source database and a second test database of the target type; wherein the first test database and the second test database maintain data consistency and are in a static state;

[0006] Obtaining business traffic data of the source database in a production environment;

[0007] Parsing and structurally converting the business traffic data to generate a business request sequence with an execution order and a logical relationship;

[0008] Performing read-write separation processing on the service request sequence to obtain a read-only service request file and a write operation service request file;

[0009] Simulate real business pressure in the first test database and the second test database and replay the read-only business request file concurrently and in parallel and compare the results. After verification, synchronously replay the write operation business request file in the first test database and the second test database and compare the results to determine the database migration evaluation result.

[0010] According to another aspect of the present invention, a database migration assessment device is provided, comprising:

[0011] Determine the source database to be migrated, and build a first test database of the same version as the source database and a second test database of the target type; wherein the first test database and the second test database maintain data consistency and are in a static state;

[0012] Obtaining business traffic data of the source database in a production environment;

[0013] Parsing and structurally converting the business traffic data to generate a business request sequence with an execution order and a logical relationship;

[0014] Performing read-write separation processing on the service request sequence to obtain a read-only service request file and a write operation service request file;

[0015] Simulate real business pressure in the first test database and the second test database and replay the read-only business request file concurrently and in parallel and compare the results. After verification, synchronously replay the write operation business request file in the first test database and the second test database and compare the results to determine the database migration evaluation result.

[0016] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by at least one processor. The computer program is executed by the at least one processor so that the at least one processor can execute the database migration assessment method according to the embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores computer instructions, which are used to enable a processor to implement the database migration evaluation method according to an embodiment of the present invention when executed.

[0021] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, which implements the steps of the above method when executed by a processor.

[0022] The technical solution of the embodiment of the present invention eliminates the interference of initial data differences on the evaluation results by constructing a first test database of the same version as the source database and a second test database of the target type, and ensuring that the initial data of the two are completely consistent and in a static state. This homogeneous benchmark environment allows any result differences in subsequent playback tests to be uniquely attributed to database type differences rather than initial data or environment configuration issues, providing an objective and reliable comparison benchmark for migration evaluation; by collecting full-volume business traffic data from the production environment, it breaks through the limitations of traditional reliance on source library system tables to collect SQL request samples; the business request sequence generated after structured conversion completely retains the original execution order and logical relationships, ensuring that all SQL requests in real business scenarios are included in the evaluation scope, effectively avoiding post-migration risks caused by omissions of SQL request samples; business requests are split into read-only files and write operation files through read-write separation processing, and read-first is adopted. Post-write step-by-step replay strategy: Read-only requests can be executed repeatedly, facilitating multiple verifications of data consistency and performance stability without the need for frequent environment recovery. Simulating real business pressure (rather than low-load or artificially constructed simple requests) can emulate the real load characteristics of actual operation, avoiding inaccurate testing caused by large differences between the test and production environments. Write operations are executed synchronously after read-only verification passes, reducing the interference of data contamination caused by write operations on the evaluation. Repeated execution of write operations can be achieved through database backup and recovery. This design improves the flexibility and repeatability of the evaluation process, reduces environmental recovery costs, and improves evaluation efficiency by replaying read-only requests in parallel.

[0023] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 This is a flowchart of a database migration assessment method provided by an embodiment of the present invention;

[0026] Figure 2 1 is a flow chart of another database migration assessment method provided by an embodiment of the present invention;

[0027] Figure 3is a structural schematic diagram of a database migration evaluation device provided by an embodiment of the present application.

[0028] Figure 4 is a structural schematic diagram of an electronic device implementing a database migration evaluation method of an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order for those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0030] Embodiment one

[0031] Figure 1 A flowchart of a database migration evaluation method provided by an embodiment of the present application, the embodiment can be applicable to the scenario of database migration, the method can be executed by a database migration evaluation device, the database migration evaluation device can be realized in the form of hardware and / or software, and the database migration evaluation device can be configured in an electronic device.

[0032] As shown in the figure, the database migration evaluation method comprises: Figure 1

[0033] S101, determine a source database to be migrated, and construct a first test database of the same version as the source database and a second test database of a target type; wherein the first test database and the second test database keep data consistent and are in a static state.

[0034] In the embodiment of the present application, the source database refers to a production environment database (such as a MySQL production library) to be migrated, which carries real business data and daily operations. The construction of the first test database comprises: using the same version as the source database, restoring data through a complete backup of the source database to ensure that it is completely consistent with the data of the source database. The second test database is constructed: using a database of a target migration type (such as migrating from MySQL to PostgreSQL), configuring the same hardware (server specifications, CPU / memory) and network environment as the first test database; migrating the data of the first test database to the second test database through logical export and import. The data of the first test database and the second test database are completely consistent, and both are in a static state (no external business writing, only for testing), which excludes the interference of initial data difference on the evaluation result.

[0035] ​S102, acquire the business traffic data of the source database in a production environment.

[0036] In the embodiment of the application, the business traffic data refers to all network packets of the interaction between the business system and the source database in the production environment, including the business request (such as SQL request) sent by the client, the response returned by the database, and the like, covering all operations (such as query, update, and transaction processing) of the real business scenario.

[0037] In some embodiments, acquiring the business traffic data of the source database in the production environment comprises: acquiring the business traffic data of the source database in the production environment through a collection tool (such as a packet capture tool) pre-deployed in the source database, and persistently storing. Specifically, the collection tool can acquire the business traffic data by directly capturing the network packets of the source database port. In addition, if the environment where the source database is located has deployed a network traffic probe (such as a switch mirror port, a server traffic monitoring, and the like), the existing network packet recording data can be directly reused as the business traffic data, without repeated packet capture.

[0038] S103, analyzing and structuring the business traffic data to generate a business request sequence with execution order and logical relationship.

[0039] In the embodiment of the application, the acquired business traffic data can be analyzed and the key elements can be extracted based on the official protocol of the source database; the analyzed key elements are sorted according to the original time stamp, the execution order (such as the timing of querying first and then updating) and the logical relationship (such as the request association within a transaction) of the business request are maintained, the business request sequence including a plurality of business requests is converted into a structured format, and finally obtained.

[0040] It should be noted that the extracted key elements include the following listed fields:

[0041] SrcIP: business IP address of the SQL issued.

[0042] SrcPort: port number used by the business to establish a connection with the database. The IP address and port number used by the business system to establish a connection with the database together constitute a unique connection identifier, and all SQL requests are issued through the database connection. In the subsequent simulation evaluation stage, the system will maintain the original business request concurrency characteristics and connection mode on the target database (the second test database) according to the connection identifier (IP:Port combination) for playback testing.

[0043] DestIP: database external service IP address.

[0044] DestPort: the port number of the database service to the outside, which together with the IP address constitutes the opposite end identification of the connection.

[0045] TimeStamp: the time stamp of the business request in the recording message, which can also be used to assist in determining the order of the request.

[0046] ReqType: SQL request type, including connection establishment / disconnection (link establishment / disconnection), transaction opening / closing, transaction submission / rollback and data operation (addition / deletion / change / query) and other business request types, which provides accurate interface calling basis for the playback component in the simulation evaluation stage, and ensures that various SQL operations can be accurately executed according to the business scenario requirements.

[0047] STMT_ID: the id of the SQL statement, when the business request adopts the prepare protocol, the id is used to accurately bind the SQL parameters on the connection with the precompiled SQL statement.

[0048] COM_Query: the specific SQL statement parsed in the message

[0049] ReqRes: execution result: success / failure.

[0050] SQLTimerWait: SQL statement execution time, which is used as an evaluation reference for evaluating the execution efficiency of SQL in the simulation environment.

[0051] S104, read-write separation processing is performed on the business request sequence to obtain a read-only business request file and a write operation business request file.

[0052] In the embodiment of the application, the business request sequence can be split into a read-only business request file and a write operation business request file based on the read-write characteristics of the business request. The read-only business request file aggregates all read-only business requests, and retains the original time sequence and concurrency characteristics; the write operation business request file aggregates all write operation business requests, and also retains the time sequence.

[0053] S105, the read-only business request file is played back in the first test database and the second test database under the simulation of real business pressure concurrency and parallelism, and the results are compared, and after verification, the write operation business request file is played back in the first test database and the second test database synchronously, and the results are compared, so as to determine the database migration evaluation result.

[0054] In this embodiment of the present invention, simulating real business pressure refers to reproducing the concurrent request volume, request frequency, and timing relationships (such as transaction order and connection mode) of the production environment to ensure that the load characteristics of the test environment are consistent with those of production. Concurrency refers to the alternating execution of multiple requests through multiple threads within a single database (such as the first test database or the second test database), simulating the simultaneous access of multiple businesses in production. Parallelism refers to the simultaneous processing of the same read-only request file by the first and second test databases, ensuring that both are subject to the same pressure over the same timeframe and ensuring fair comparison.

[0055] In some embodiments, simulating real business pressure in the first test database and the second test database and replaying the read-only business request file concurrently and comparing the results include the following steps AC:

[0056] A. Distribute the read-only service requests with the same connection identifier in the read-only service request file to a request queue maintained by the same thread.

[0057] In some embodiments, a read-only service request file is parsed to determine the connection identifier of each read-only service request. Read-only service requests with the same connection identifier are then assigned to a request queue maintained by the same thread using a consistent hashing algorithm. A thread is an execution unit that processes service requests. In embodiments of the present invention, multiple threads are used to process service requests, and dynamic scaling is supported at runtime. For example, the number of threads can be automatically adjusted based on the raw service load characteristics. As can be appreciated, assigning requests with the same connection identifier to the request queue of the same thread ensures that the execution order of requests within a single connection is consistent with real-world services, conforming to the request logic of a single connection in real-world scenarios. Furthermore, multiple threads simultaneously maintain request queues for different connections, simulating the core characteristic of multiple concurrent connections in real-world services (e.g., multiple users initiating requests simultaneously). Furthermore, the number of threads can be dynamically adjusted based on the raw service load to match fluctuations in concurrent pressure in real-world services (e.g., an increase in the number of connections during peak periods), ensuring that the pressure intensity is consistent with real-world scenarios.

[0058] In other embodiments, the read-only service requests with the same connection identifier in the read-only service request file are distributed to a request queue maintained by the same thread, including: determining, based on the size of a preset time window, a target read-only service request to be issued within the current time window from the read-only service request file; wherein the time window size may be 1 second, and the current time window is [T, T+1], where T is the current time. The target read-only service requests with the same connection identifier are distributed to a request queue maintained by the same thread; and after all threads have completed executing the target read-only service request within the current time window or are all executing the last target read-only service request, returning to the operation of determining the target read-only service request to be issued within the current time window from the read-only service request file. It is understandable that by dividing the read-only business requests to be executed into preset time windows (such as 1 second), the read-only business requests within the same window are distributed to different threads for parallel processing, reproducing the time distribution characteristics of multiple concurrent requests in the same time period in real business (such as 100 requests arriving at the same time within 1 second), avoiding time confusion during request playback (such as piling requests at different times together), and ensuring the authenticity of concurrency in the time dimension.

[0059] In this embodiment, each read-only business request is executed under the timeout control mechanism. The timeout control mechanism configures a timeout parameter for each read-only business request to prevent the read-only business request from hanging and causing the thread and evaluation process to be unable to end. The default input parameter of the timeout parameter of each read-only business request is 2 times the SQLTimerWait value.

[0060] B. Maintaining the thread of the request queue, and sending the read-only service requests in the request queue to the first test database and the second test database for execution in accordance with the service request execution order.

[0061] The execution order of the service requests is determined based on the timestamps of the read-only service requests in the request queue.

[0062] Understandably, in real-world scenarios, requests from different connections follow a clear time sequence. In step B, threads execute queued requests in the order of read-only business request timestamps. This ensures that the relative execution timing of requests between different connections aligns with real-world scenarios, reproducing the request timeline in real-world scenarios and avoiding stress distortion caused by timing mismatches. The threads simultaneously send requests to the first and second test databases, demonstrating the parallel playback feature. This ensures that both databases receive requests under the same concurrency pressure, providing a fair benchmark for subsequent result comparisons.

[0063] C. Compare the execution result of the first test database with the execution result of the second test database.

[0064] In this embodiment, the execution results of the read-only business request in the first test database and the execution results of the second test database can be compared one by one to see if they are consistent. For example, the execution status and result set hash values ​​of the read-only business request in the first test database and the second test database can be compared one by one to see if they are consistent. If they are consistent, it means that the execution result of the read-only business request in the second test database is the same as the execution result in the source database, so that after the business system is migrated from the source database to the target database of the same type as the second test database, the read-only business request is more adaptable to the target database, thus completing the compatibility test of the read-only business request. In addition, the execution results of the read-only business request in the first test database and the second test database include not only the execution status and result set hash value, but also the execution time of the read-only business request in the first test database and the second test database. In this way, the execution time of the same read-only business request in the first test database and the second test database can be compared to determine the performance of the database.

[0065] It's important to note that if the execution results of a read-only business request on the first test database differ from those on the second test database, this is primarily due to underlying differences between the heterogeneous test databases, such as syntax support, function logic, type handling, and optimizer rules. These discrepancies directly reflect a flaw in the second test database's adaptation to the first test database's business SQL requests. Log analysis is necessary to identify the specific cause (e.g., syntax errors, function differences, or differing sorting logic), and to implement targeted optimizations (e.g., rewriting SQL or adjusting the second test database configuration). Otherwise, the migrated business query will produce incorrect results or fail. The logs involved include both summary and detail logs stored in hierarchical storage. The detail logs contain the incrementing sequence number (SQLID) of each read-only business request, the schema identifier, the complete SQL text, the complete result set (field values, number of rows, and order) from the first and second test databases, execution environment parameters (e.g., character set and time zone), and the error stack of the second test database (in case of failure). For example, detailed logs can directly reveal that the result set returned by the second test database is missing two rows. Combined with the SQL text, this is due to differences in function processing in the WHERE clause. Alternatively, the error stack can be used to pinpoint the specific location of the syntax error (such as a unique function name). Summary logs can quickly identify the pattern identifier of the abnormal request, calculate the inconsistency rate of this type of read-only business request, and determine whether it is a sporadic issue or a batch compatibility issue (for example, a certain function type is always handled abnormally in the second test database).

[0066] In the embodiment of the present application, if each read-only service request passes the comparison test, consistency and performance analysis is performed on the write operation service request. It should be noted that, due to the possibility of accidents (such as server downtime, network interruption, program abnormal crash, etc.) during the test process, these accidents will cause the current write operation service request playback to be interrupted or the result to be unreliable. Therefore, by performing full backup on the first test database and the second test database before the write operation service request playback, when an accident occurs, the two databases can be restored to the initial data state before the test by using the backup, so as to re-execute the write operation service request playback test, ensure that the entire verification process can be reliably repeated, and avoid test interruption or invalid results due to accidents. That is, the playback of the write operation service request can also be repeated.

[0067] In some embodiments, synchronously playing back the write operation service request file in the first test database and the second test database and comparing the results includes steps one to four:

[0068] Step one, splitting the write operation service request file according to the database tables involved in the write operation service request to obtain a plurality of sub-files; each of the sub-files includes a write operation service request for a single database table.

[0069] Step two, for each of the sub-files, maintaining the original time sequence of the write operation service requests in the sub-file. That is, the order of all write operation service requests for the same database table in the sub-file must be completely consistent with the execution order in the original service traffic, so as to ensure the consistency of transaction logic and data, and the order consistency can restore the real service pressure scenario.

[0070] Step three, controlling the plurality of sub-files to be executed concurrently in the first test database and the second test database, and recording the execution process of each write operation service request, mainly recording the incremental identifier, execution state, execution time consumption, affected row number, and exception stack information of each write operation service request; wherein the write operation service requests in a single sub-file are executed serially. In this way, the efficiency of evaluating the write operation service request can be improved.

[0071] Step four, performing data consistency and performance comparison analysis according to the execution results of each of the sub-files in the first test database and in the second test database; wherein the execution results at least include the incremental identifier, execution state, execution time consumption, affected row number, and exception stack information of each write operation service request.

[0072] In some embodiments, data consistency and performance comparison analysis is performed based on the execution results of each sub-file in the first test database and the execution results in the second test database, including: performing table-level data consistency verification on the first test database and the second test database through a data comparison tool; marking the database table with differences as a risk table and analyzing the execution log corresponding to the risk table; comparing the execution time of similar write operation business requests in the first test database and the second test database according to the pattern identifier of the write operation business request, and generating a time comparison trend and quantile difference report.

[0073] In the embodiment of the present invention, by constructing a first test database of the same version as the source database and a second test database of the target type, and ensuring that the initial data of the two are completely consistent and in a static state, the interference of initial data differences on the evaluation results is eliminated. This homogeneous benchmark environment ensures that any discrepancies in subsequent playback test results can be uniquely attributed to differences in database types, rather than issues with the initial data or environment configuration, providing an objective and reliable comparison benchmark for migration assessments. By collecting full business traffic data from the production environment, this overcomes the limitations of traditional SQL request sample collection based on source database system tables. The business request sequence generated after structured conversion fully preserves the original execution order and logical relationships, ensuring that all SQL requests (i.e., business requests) in real business scenarios are included in the assessment scope, effectively avoiding post-migration risks caused by missed SQL requests. Through read-write separation, business requests are split into read-only files and write operation files, and a step-by-step replay strategy of read-first and write-later is adopted. Read-only requests can be executed repeatedly, facilitating multiple verifications of data consistency and performance stability without the need for frequent environment restores. Write operations are executed synchronously after read-only verification passes, reducing the impact of data contamination caused by write operations on the assessment. Repeated execution testing of write business requests is possible through database backup and recovery. This design enhances the flexibility and repeatability of the assessment process, reduces environment recovery costs, and improves assessment efficiency through parallel replay of read-only requests.

[0074] Example 2

[0075] Figure 2 The present invention provides a flowchart of a database migration evaluation method. Figure 2 , the method comprises the following steps:

[0076] S201: Determine a source database to be migrated, and construct a first test database of the same version as the source database and a second test database of the target type; wherein the first test database and the second test database maintain data consistency and are in a static state.

[0077] S202, acquire the business traffic data of the source database in a production environment.

[0078] S203, parse and structure convert the business traffic data to generate a business request sequence with execution order and logical relationship.

[0079] The specific implementation process of steps S201-S203 can refer to the description of the above embodiments, which will not be repeated here.

[0080] S204, assign an incremental serial number to each business request in the business request sequence; mode processing is performed on each business request in the business request sequence to obtain a corresponding mode statement, and the same mode statement is assigned the same mode identification.

[0081] In the embodiment of the application, an incremental serial number (such as SQLID) is assigned to each business request (such as SQL request) in the business request sequence; for each business request, mode processing is performed on the business request by a SQL template tool to obtain a corresponding mode statement (digesttext), and the same mode statement is assigned the same mode identification (such as digestid). It can be understood that the incremental serial number and the mode identification of the business request can be used for comparison of subsequent execution results between the two databases.

[0082] For example, after mode processing of the business request "update table1 set a=2 where b=1" and the business request "update table1 set a=3 where b=2", the same mode statement "update table1 set a=? where b=?" is generated, and the same mode identification is assigned to the two business requests, so as to ensure that the same mode business request (SQL statement) has consistent mode identification, and different mode business requests correspond to unique mode identification.

[0083] S205, read-write separation processing is performed on the business request sequence to obtain a read-only business request file and a write operation business request file.

[0084] S206, simulate real business pressure in the first test database and the second test database, concurrently and in parallel, play back the read-only business request file and compare the results, and after verification, synchronously play back the write operation business request file in the first test database and the second test database and compare the results to determine the database migration evaluation result.

[0085] In some embodiments, simulating real business pressure in the first test database and the second test database and replaying the read-only business request file concurrently and comparing the results include the following steps AC:

[0086] A. Distribute the read-only service requests with the same connection identifier in the read-only service request file to a request queue maintained by the same thread.

[0087] In some embodiments, a read-only business request file is parsed to determine the connection identifier of each read-only business request therein, and the read-only business requests with the same connection identifier are assigned to a request queue maintained by the same thread through a consistent hashing algorithm; wherein, a thread refers to an execution unit for processing business requests. In an embodiment of the present invention, there are multiple threads for processing business requests, and dynamic expansion is supported at runtime. For example, the number of threads can be automatically adjusted according to the original business load characteristics.

[0088] In other embodiments, the read-only service requests with the same connection identifier in the read-only service request file are distributed to a request queue maintained by the same thread, including: determining, from the read-only service request file, target read-only service requests to be issued within the current time window based on the size of a preset time window; wherein the time window size may be 1 second, and the current time window is [T, T+1], where T is the current time. The target read-only service requests with the same connection identifier are distributed to the request queue maintained by the same thread; and after all threads have completed executing the target read-only service requests within the current time window or are currently executing the last target read-only service request, the operation of determining the target read-only service requests to be issued within the current time window from the read-only service request file is returned to execution. It is understood that requests within each preset time window (e.g., 1 second) are considered concurrent service requests within the same time segment and must be processed simultaneously by each thread within that window period; only when all threads have completed the requests in the current window (or are currently processing the last request) do they proceed to the next window. This mechanism forces all threads to maintain a similar execution progress, preventing any one thread from running too fast or too slow. This ensures that requests within the same time period are processed concurrently within the same time window, maintaining concurrency synchronization. The window length (such as 1 second or 5 seconds) is essentially the time granularity for simulating concurrency. The shorter the window, the higher the simulation time precision, the more accurate the simulation of concurrency pressure, and the more accurately it can replicate the instantaneous characteristics of concurrent requests in real business.

[0089] B. Maintaining the thread of the request queue and sending the read-only service requests in the request queue to the first test database and the second test database for execution in the order of service request execution. The order of service request execution is determined based on the timestamps of the read-only service requests in the request queue.

[0090] C. Compare the execution result of the first test database with the execution result of the second test database.

[0091] Specifically, according to the ascending serial number of the read-only business request, the execution status and result set hash value of the read-only business request in the first test database and the second test database are compared one by one; or, according to the mode identifier of the read-only business request, the execution time of the same type of read-only business request in the first test database and the second test database is compared, and a time comparison trend chart and a quantile difference report are generated.

[0092] In this embodiment of the present invention, the core of generating a quantile difference report is to statistically analyze the execution time distribution of similar read-only business requests in the first and second test databases, quantifying the performance differences between the two at different stress quantiles. The specific process is as follows: First, data preparation is performed to aggregate the execution time data of similar read-only business requests. Specifically, the data is grouped by digestID: a digestID is a unique identifier for similar SQL business requests. All read-only business requests in the read-only business request file are aggregated by digestID, generating a list of execution times in the first and second test databases corresponding to each digestID. For example, for digestID 1, the aggregation yields 1000 execution time records for this type of request in the first test database (e.g., [10ms, 15ms, ..., 30ms]) and 1000 execution time records in the second test database (e.g., [12ms, 18ms, ..., 40ms]). Furthermore, execution failures and timeouts (e.g., exceeding a preset threshold) are eliminated to ensure that only valid requests are included in the statistics and to prevent outliers from interfering with quantile calculations. Next, quantile calculations are performed, primarily generating quantile values ​​for the first and second test databases. Specifically, for each digestID, key quantiles (typically including P50, P90, and P99, but expandable as needed) are calculated for the time consumption lists of the first and second test databases. This involves sorting the time consumption lists in ascending order (e.g., sorting the time consumption of the first test database: [8ms, 10ms, 12ms, …, 50ms]). The index of the corresponding position is calculated according to the quantile definition (index = quantile × total number of samples, rounded or rounded to the nearest nearest integer). For example, P50 (50% quantile) corresponds to the value at the 50th percentile after sorting. If the number of samples is 1000, then index = 1000 × 50% = 500, and the 500th value is taken; P90 corresponds to index = 1000 × 90% = 900, and the 900th value is taken; P99 corresponds to index = 1000 × 99% = 990, and the 990th value is taken. Record the quantiles of the first and second test databases. Furthermore, quantify the performance deviation of the quantiles of the two test databases. Specifically, for each pattern identifier, calculate the difference between the second test database and the first test database at each quantile, such as the absolute difference or relative difference. Finally, generate a quantile difference report based on the calculated quantiles and difference values.

[0093] In this embodiment, when the read-only service request has a difference in the execution results of the first test data and the second test database, the problem can be located, the root cause can be analyzed, and the high-risk SQL can be focused on, thereby providing specific basis for migration risk control. Specifically, first, difference positioning is performed, that is, when the hash values of the execution results of the two databases do not match (that is, the data consistency check fails), the specific difference content is obtained by associating the detail log (recording the complete result set, SQL text, etc.), and the specific performance of the inconsistent result set is determined. Further, root cause analysis is performed, which mainly aims at the difference or exception, and the complete business request (SQL statement) is retrieved by the incremental serial number (SQLID) of the business request, and the problem is reproduced by twice execution in the two databases to verify whether the difference exists stably; if the business request execution fails, the direct cause and chain effect of the failure are traced back by the error stack information (such as syntax error, permission exception, etc.) and the context request chain (such as the previous operation in the transaction). Hot spot focusing mainly automatically screens and marks three types of high-risk business requests: 1. Performance degradation: the execution time in the second test database is more than 120% of the execution time in the first test database. 2. Data inconsistency: the result sets of the two databases have differences. 3. Stability problem: the failure rate difference is more than 5%.

[0094] In the embodiment of the application, if each read-only business request passes the comparison test, consistency and performance analysis is performed on the write operation business request, specifically as follows.

[0095] In some embodiments, the write operation business request file is synchronously played back in the first test database and the second test database and the results are compared, including steps 1 to 4:

[0096] Step 1, the write operation business request file is split according to the database tables involved in the write operation business request to obtain a plurality of sub-files; each of the sub-files includes write operation business requests for a single database table.

[0097] Step 2, for each of the sub-files, the original time sequence of the write operation business requests in the sub-file is maintained; that is, the order of all write operation business requests for the same database table in the sub-file must be completely consistent with the execution order in the original business traffic, so as to ensure the consistency of transaction logic and data, and the order consistency can restore the real business pressure scenario.

[0098] Step 3, the plurality of sub-files are controlled to be executed concurrently in the first test database and the second test database, and the execution process of each write operation business request is recorded, mainly the incremental serial number, execution state, execution time, affected row number, and exception stack information of each write operation business request; wherein the write operation business requests in a single sub-file are executed in series. In this way, the efficiency of evaluating the write operation business request can be improved.

[0099] Step 4: Perform data consistency and performance comparison analysis based on the execution results of each sub-file in the first test database and the execution results in the second test database; wherein the execution results include at least the incremental label, execution status, execution time, number of affected rows, and exception stack information of each write operation business request.

[0100] In some embodiments, based on the execution results of each sub-file in the first test database and the execution results in the second test database, a data consistency and performance comparison analysis is performed, including: performing table-level data consistency verification on the first test database and the second test database through a data comparison tool; marking the database table with differences as a risk table and analyzing the execution log corresponding to the risk table; comparing the execution time of similar write operation business requests in the first test database and the second test database according to the pattern identifier of the write operation business request, and generating a time comparison trend and quantile difference report. The implementation method of generating the quantile difference report here is similar to the above and will not be repeated here.

[0101] In an embodiment of the present invention, a complete diagnostic link is also provided. Specifically, the specific abnormal business request (SQL request) is first located from the risk table marked by the data consistency check; then the complete execution context of the business request (such as timing, concurrency environment, and transaction association) is associated; if there is a difference in the execution results of the two databases (such as the number of affected rows), a manual confirmation link is triggered to ensure the accuracy of the difference assessment.

[0102] In the embodiment of the present invention, an increasing sequence number and a mode identifier are assigned to each service request, so as to facilitate subsequent evaluation and analysis based on the increasing sequence number and the mode identifier of the service request, thereby improving the efficiency of the evaluation and analysis.

[0103] Example 3

[0104] Figure 3 This is a schematic diagram of the structure of a database migration evaluation device provided by an embodiment of the present invention. This embodiment is applicable to the scenario of database migration. The device can execute any database migration evaluation method of the present invention. Figure 3 As shown, the database migration assessment device includes:

[0105] The database determination module 301 is configured to determine a source database to be migrated and construct a first test database of the same version as the source database and a second test database of the target type; wherein the first test database and the second test database maintain data consistency and are in a static state;

[0106] A data acquisition module 302 is used to acquire business flow data of the source database in a production environment;

[0107] The data conversion module 303 is used to parse and structure the service traffic data to generate a service request sequence with an execution order and a logical relationship;

[0108] A data separation module 304 is configured to perform read-write separation processing on the service request sequence to obtain a read-only service request file and a write operation service request file;

[0109] The evaluation module 305 is used to simulate real business pressure in the first test database and the second test database, and to replay the read-only business request file concurrently and in parallel and compare the results. After verification, the write operation business request file is synchronously replayed in the first test database and the second test database and the results are compared to determine the database migration evaluation result.

[0110] In some embodiments, in terms of simulating real business pressure in the first test database and the second test database and replaying the read-only business request file concurrently and comparing the results, the evaluation module 305 includes:

[0111] A distribution unit, configured to distribute the read-only service requests having the same connection identifier in the read-only service request file to a request queue maintained by the same thread;

[0112] An execution unit, configured to send the read-only service requests in the request queue to the first test database and the second test database for execution in sequence according to the service request execution order by maintaining the thread of the request queue;

[0113] A comparing unit is used to compare the execution result of the first test database with the execution result of the second test database.

[0114] In some embodiments, in the step of distributing the read-only service requests having the same connection identifier in the read-only service request file to the request queue maintained by the same thread, the distributing unit is specifically configured to:

[0115] Determining, from the read-only service request file, a target read-only service request to be issued within the current time window according to the size of the preset time window;

[0116] Distributing the target read-only service requests with the same connection identifier to a request queue maintained by the same thread;

[0117] After all threads have finished executing the target read-only service request in the current time window or are all executing the last target read-only service request, the operation of determining the target read-only service request to be issued in the current time window from the read-only service request file is returned.

[0118] In some embodiments, a pre-processing module is further included for:

[0119] Assigning an increasing sequence number to each service request in the service request sequence;

[0120] Performing pattern processing on each service request in the service request sequence to obtain a corresponding patterned statement, and assigning the same pattern identifier to the service requests with the same patterned statement;

[0121] Accordingly, the comparison unit is specifically used for:

[0122] Comparing the execution status and result set hash values ​​of the read-only business requests in the first test database and the second test database one by one according to the increasing sequence number of the read-only business requests; or

[0123] According to the mode identifier of the read-only business request, the execution time of the same type of read-only business request in the first test database and the second test database is compared, and a time comparison trend graph and a quantile difference report are generated.

[0124] In some embodiments, in terms of synchronously replaying the write operation service request file in the first test database and the second test database and comparing the results, the evaluation module 305 includes:

[0125] a splitting unit, configured to split the write operation service request file according to the database tables involved in the write operation service request to obtain a plurality of sub-files; each of the sub-files includes a write operation service request for a single database table;

[0126] A maintaining unit, configured to maintain, for each sub-file, an original time sequence of a write operation service request in the sub-file;

[0127] A control unit, configured to control the concurrent execution of the plurality of subfiles in the first test database and the second test database; wherein write operation service requests in a single subfile are executed serially;

[0128] An analysis unit is used to perform data consistency and performance comparison analysis based on the execution results of each sub-file in the first test database and the execution results in the second test database; wherein the execution results at least include the incremental label, execution status, execution time, number of affected rows, and exception stack information of each write operation business request.

[0129] In some embodiments, the analysis unit is specifically configured to:

[0130] Performing table-level data consistency verification on the first test database and the second test database using a data comparison tool;

[0131] Marking the database table with differences as a risk table and analyzing the execution log corresponding to the risk table;

[0132] According to the pattern identifier of the write operation business request, the execution time of the similar write operation business request in the first test database and the second test database is compared, and a time consumption comparison trend and quantile difference report is generated.

[0133] In some embodiments, the data acquisition module is specifically configured to:

[0134] The service flow data of the source database in the production environment is obtained through the collection tool pre-deployed in the source database, and is persistently stored.

[0135] The database migration assessment device provided in the embodiment of the present invention can execute the database migration assessment method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0136] According to an embodiment of the present invention, the present invention further provides an electronic device, a readable storage medium and a computer program product.

[0137] Example 4

[0138] Figure 4 The components, their connections and relationships, and their functions shown herein are merely examples and are not intended to limit implementation of the invention described and / or claimed herein.

[0139] like Figure 4 As shown, electronic device 10 includes at least one processor 11 and memory, such as read-only memory (ROM) 12 and random access memory (RAM) 13, communicatively connected to at least one processor 11. The memory stores computer programs executable by the at least one processor. Processor 11 can perform various appropriate actions and processes based on the computer programs stored in ROM 12 or loaded from storage unit 18 into RAM 13. RAM 13 can also store various programs and data required for the operation of electronic device 10. Processor 11, ROM 12, and RAM 13 are interconnected via bus 14. An input / output (I / O) interface 15 is also connected to bus 14.

[0140] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0141] The processor 11 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as performing the database migration assessment method.

[0142] In some embodiments, the database migration assessment method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the database migration assessment method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the database migration assessment method by any other appropriate means, such as by means of firmware.

[0143] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0144] Computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable database migration assessment device, such that, when executed by the processor, the computer programs implement the functions / operations specified in the flowcharts and / or block diagrams. The computer programs can be executed entirely on the machine, partially on the machine, as a standalone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0145] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or apparatus. A computer-readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0146] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device that has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0147] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0148] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0149] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0150] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A database migration assessment method, characterized in that: include: Determine the source database to be migrated, and build a first test database of the same version as the source database and a second test database of the target type; wherein the first test database and the second test database maintain data consistency and are in a static state; Obtaining business traffic data of the source database in a production environment; Parsing and structurally converting the business traffic data to generate a business request sequence with an execution order and a logical relationship; Performing read-write separation processing on the service request sequence to obtain a read-only service request file and a write operation service request file; Simulate real business pressure in the first test database and the second test database and replay the read-only business request file concurrently and in parallel and compare the results. After verification, synchronously replay the write operation business request file in the first test database and the second test database and compare the results to determine the database migration evaluation result.

2. The method according to claim 1, characterized in that The simulating real business pressure in the first test database and the second test database and replaying the read-only business request file concurrently and comparing the results include: Distributing the read-only service requests with the same connection identifier in the read-only service request file to a request queue maintained by the same thread; By maintaining the thread of the request queue, the read-only service requests in the request queue are sent to the first test database and the second test database for execution in sequence according to the service request execution order; Compare the execution result of the first test database with the execution result of the second test database.

3. The method according to claim 2, characterized in that The distributing the read-only service requests with the same connection identifier in the read-only service request file to the request queue maintained by the same thread includes: Determining, from the read-only service request file, a target read-only service request to be issued within the current time window according to the size of the preset time window; Distributing the target read-only service requests with the same connection identifier to a request queue maintained by the same thread; After all threads have finished executing the target read-only service request in the current time window or are all executing the last target read-only service request, the operation of determining the target read-only service request to be issued in the current time window from the read-only service request file is returned.

4. The method according to claim 2, characterized in that Also includes: Assigning an increasing sequence number to each service request in the service request sequence; Performing pattern processing on each service request in the service request sequence to obtain a corresponding patterned statement, and assigning the same pattern identifier to the service requests with the same patterned statement; Accordingly, comparing the execution result of the first test database with the execution result of the second test database includes: Comparing the execution status and result set hash values ​​of the read-only business requests in the first test database and the second test database one by one according to the increasing sequence number of the read-only business requests; or According to the mode identifier of the read-only business request, the execution time of the same type of read-only business request in the first test database and the second test database is compared, and a time comparison trend graph and a quantile difference report are generated.

5. The method according to claim 1, wherein The synchronously replaying the write operation service request file in the first test database and the second test database and comparing the results include: Splitting the write operation business request file according to the database tables involved in the write operation business request to obtain multiple sub-files; each of the sub-files includes a write operation business request for a single database table; For each of the sub-files, maintaining the original time sequence of the write operation service request in the sub-file; Controlling the multiple sub-files to be executed concurrently in the first test database and the second test database; wherein write operation business requests in a single sub-file are executed serially; A data consistency and performance comparison analysis is performed based on the execution results of each sub-file in the first test database and the execution results in the second test database; wherein the execution results at least include the incremental label, execution status, execution time, number of affected rows, and exception stack information of each write operation business request.

6. The method according to claim 5, characterized in that Performing data consistency and performance comparison analysis based on the execution results of each sub-file in the first test database and the execution results in the second test database includes: Performing table-level data consistency verification on the first test database and the second test database using a data comparison tool; Marking the database table with differences as a risk table and analyzing the execution log corresponding to the risk table; According to the pattern identifier of the write operation business request, the execution time of the similar write operation business request in the first test database and the second test database is compared, and a time consumption comparison trend and quantile difference report is generated.

7. The method according to claim 5, characterized in that The obtaining of the service flow data of the source database in the production environment includes: The service flow data of the source database in the production environment is obtained through the collection tool pre-deployed in the source database, and is persistently stored.

8. A database migration assessment device, characterized in that: include: A database determination module is configured to determine a source database to be migrated and construct a first test database of the same version as the source database and a second test database of the target type; wherein the first test database and the second test database maintain data consistency and are in a static state; A data acquisition module is used to acquire business flow data of the source database in a production environment; A data conversion module is used to parse and structure the service traffic data to generate a service request sequence with an execution order and a logical relationship; A data separation module is used to perform read-write separation processing on the service request sequence to obtain a read-only service request file and a write operation service request file; An evaluation module is configured to simulate real business pressure in the first test database and the second test database, concurrently replay the read-only business request file and compare the results, and after verification, synchronously replay the write operation business request file in the first test database and the second test database and compare the results to determine a database migration evaluation result.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method according to any one of claims 1 to 7 when executed.

Images