Flow replication pressure measurement method and device for heterogeneous database, equipment and medium

Abstract

The invention relates to the technical field of flow processing of heterogeneous databases, and discloses a flow replication pressure measurement method, device and equipment for a heterogeneous database and a medium, and the method comprises the steps: loading a byte code enhancement module in a Spring Boot starting stage through a self-defined Java Agent, and carrying out non-intrusive transformation on byte codes based on an ASM framework; a flow weight routing mechanism is introduced in a message queue level, and the flow of the SQL statement is obtained through gradual adjustment based on a dynamic replication proportion adjustment strategy of the flow weight routing mechanism in a time window; and constructing an SQL grammar conversion engine for the obtained SQL statements in the data consumption end, and defining a grammar mapping rule from Oracle to TiDB through a YAML configuration file. The method can be applied to development of business systems such as financial science and technology and medical health care, and seamless migration from Oracle 11g to TiDB 5.0 is supported.

Description

Technical Field

[0001] This invention relates to the technical field of traffic processing for heterogeneous databases, and more particularly to a method, apparatus, device, and medium for traffic replication stress testing of heterogeneous databases. Background Technology

[0002] In tests conducted in financial scenarios such as fintech, healthcare, and elderly care, mainstream traffic replication technologies fall into two main categories: capturing and replaying data changes by parsing the Redo Log or Binlog of databases like Oracle / MySQL, with tools like Alibaba Cloud DTS and DataX being typical examples. However, Oracle and TiDB differ significantly in SQL syntax, function support (such as sequences and partitioned tables), and transaction isolation levels. This means that SQL statements parsed from the logs cannot be directly executed in TiDB. For instance, Oracle's `SELECT * FROM DUAL` has no corresponding implementation in TiDB and requires manual adaptation. Tests in a specific financial scenario showed that traffic replication for 100+ microservice nodes required an investment of over 200 person-days for buffer development and load testing. Summary of the Invention

[0003] This invention provides a method, apparatus, computer equipment, and medium for traffic replication stress testing of heterogeneous databases, in order to solve the technical problem that privilege escalation vulnerabilities in the prior art basically require manual inspection during testing.

[0004] Firstly, a traffic replication stress testing method for heterogeneous databases is provided, including:

[0005] By using a custom Java Agent to load the bytecode enhancement module during the Spring Boot startup phase, the bytecode is non-intrusively modified based on the ASM framework to obtain the modified SQL statement;

[0006] A traffic weight routing mechanism is introduced at the message queue level. Based on the traffic weight routing mechanism, the dynamic replication ratio adjustment strategy within the time window is used to gradually adjust the traffic of SQL statements. When the TP99 response time of the target database exceeds 500ms, the system automatically triggers degradation logic and switches to asynchronous compensation mode.

[0007] On the data consumption side, the acquired SQL statements are used to build an SQL syntax conversion engine, which then...

[0008] YAML configuration files define the syntax mapping rules from Oracle to TiDB, building a complete SQL syntax adaptation system for heterogeneous databases.

[0009] Secondly, a traffic replication stress testing device for heterogeneous databases is provided, comprising:

[0010] An intercepting module is configured to load a bytecode enhancement module at a Spring Boot starting stage through a custom Java Agent, to perform non-invasive modification on bytecode based on an ASM framework, and to obtain modified SQL statements;

[0011] An optimization module is configured to introduce a traffic weight routing mechanism at a message queue level, to gradually adjust traffic of the SQL statements based on a dynamic replication ratio adjustment strategy of the traffic weight routing mechanism in a time window, and to automatically trigger a degradation logic and switch to an asynchronous compensation mode when a TP99 response time of a target database exceeds 500 ms.

[0012] A conversion module is configured to construct a SQL syntax conversion engine from the obtained SQL statements at a data consumption end, to define Oracle-to-TiDB syntax mapping rules through a YAML configuration file, and to construct a complete heterogeneous database SQL syntax adaptation system.

[0013] In a third aspect, a computer device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the traffic replication stress testing method for the heterogeneous database when executing the computer program.

[0014] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program implements the steps of the traffic replication stress testing method for the heterogeneous database when executed by a processor.

[0015] The scheme realized by the traffic replication stress testing method and device for the heterogeneous database, the computer device, and the storage medium can load the bytecode enhancement module in the Spring Boot starting stage through the custom Java Agent, perform non-invasive modification on the bytecode based on the ASM framework, and obtain the modified SQL statement; introduce a traffic weight routing mechanism at the message queue level, gradually adjust the traffic of the obtained SQL statement based on the traffic weight routing mechanism and the dynamic replication ratio adjustment strategy of the time window, wherein when the TP99 response time of the target database is monitored to be more than 500 ms, the system automatically triggers the degradation logic and switches to the asynchronous compensation mode; and constructs the SQL syntax conversion engine for the obtained SQL statement in the data consumption end, defines the syntax mapping rule from Oracle to TiDB through the YAML configuration file, and constructs a complete heterogeneous database SQL syntax adaptation system. The application realizes seamless migration from Oracle 11g to TiDB 5.0; the volume of the third-party package is less than 5MB, and only the Spring Boot starting script needs to be modified to add the -javaagent parameter; the application supports scenarios such as traffic mirroring (Mirror), gray release (1%->10%->100%), A / B testing, and the risk control system can dynamically set the replication strategy, such as preferentially replicating the business with the insurance amount greater than 1 million yuan, and realizes real-time synchronization of cross-machine room heterogeneous databases, desensitizes the production traffic for machine learning model training, and quickly constructs a fault injection test environment through traffic replication. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the application. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0017] Figure 1 is an application environment schematic diagram of the traffic replication stress testing method for the heterogeneous database in an embodiment of the application;

[0018] Figure 2 is a flowchart of the traffic replication stress testing method for the heterogeneous database in an embodiment of the application;

[0019] Figure 3 is a structure schematic diagram of the traffic replication stress testing device for the heterogeneous database in an embodiment of the application;

[0020] Figure 4 is a structure schematic diagram of the computer device in an embodiment of the application;

[0021] Figure 5is another structural schematic diagram of the computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0023] The traffic replication stress testing method for heterogeneous databases provided by the embodiments of the present application can be applied in application environments such as Figure 1 , wherein the client loads a bytecode enhancement module in a Spring Boot startup phase through a custom Java Agent, performs non-invasive modification on bytecode based on an ASM framework to obtain modified SQL statements, introduces a traffic weight routing mechanism at a message queue level, and gradually adjusts traffic of the obtained SQL statements based on a dynamic replication ratio adjustment strategy of the traffic weight routing mechanism in a time window. When the TP99 response time of the target database is monitored to be more than 500 ms, the system automatically triggers a degradation logic to switch to an asynchronous compensation mode. The obtained SQL statements are constructed into a SQL syntax conversion engine in a data consumption end, syntax mapping rules from Oracle to TiDB are defined through a YAML configuration file, and a complete heterogeneous database SQL syntax adaptation system is constructed. The present application realizes seamless migration from Oracle 11g to TiDB 5.0, the volume of a third-party package is less than 5 MB, and only the -javaagent parameter needs to be added in a Spring Boot startup script. The present application supports scenarios such as traffic mirroring (Mirror), gray release (1%->10%->100%), A / B testing, and the like. A risk control system can dynamically set a replication strategy, such as preferentially replicating a business with a policy amount greater than 1 million yuan, and realizes real-time synchronization of cross-machine-room heterogeneous databases. The production traffic is desensitized and used for machine learning model training, a fault injection test environment is quickly constructed through traffic replication, and the like. The client can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers. The present application will be described in detail below through specific embodiments.

[0024] Please refer to Figure 2 , Figure 2 is a flowchart of the traffic replication stress testing method for heterogeneous databases provided by the embodiments of the present application, comprising the following steps:

[0025] S10: Load bytecode enhancement module through custom Java Agent during Spring Boot startup phase, perform non-invasive modification on bytecode based on ASM framework, and obtain modified SQL statements; specifically, create a Java project as a custom Java Agent, define a class containing a premain method in the project, and set the premain method as the entry point of the Java Agent; receive external configuration information through the agentArgs parameter; in the startup script or configuration of the Spring Boot application, add the -javaagent parameter to specify the path and related parameters of the custom Java Agent. According to the importance of different businesses, data size, real-time requirements and other factors, allocate different traffic weights to each message queue or topic; in the configuration file or management interface of the message queue, set the weight parameter corresponding to the traffic weight.

[0026] Load bytecode enhancement module through custom Java Agent during Spring Boot startup, modify the bytecode of JdbcTemplate / MyBatis related classes using ASM framework, and achieve "non-invasive" SQL interception:

[0027] First, create a Java project as a custom Java Agent. Define a class containing a premain method in the project, which is the entry point of the Java Agent. For example:

[0028]

[0029] Receive external configuration information through the agentArgs parameter, such as specifying the type of SQL to be intercepted, the name of the related class, etc. The agentArgs parameter can be passed in through the -javaagent parameter when starting the Spring Boot application.

[0030] In the bytecode enhancement module loaded during the Spring Boot startup, associate Java Agent with Spring Boot application: in the startup script or configuration of the Spring Boot application, add the -javaagent parameter to specify the path and related parameters of the custom Java Agent, and obtain the bytecode. For example:

[0031] java -javaagent:path / to / my-agent.jar=arg1=value1,arg2=value2 -jar my-spring-boot-app.jar

[0032] In this way, when the Spring Boot application starts, the JVM will first load the Java Agent and execute the premain method. Instrumentation is used to register class file transformers: In the premain method, use the Instrumentation object to register a class file transformer. This transformer will intercept and modify the class bytecode before the class is loaded.

[0033] Modify the bytecode of JdbcTemplate / MyBatis related classes using the ASM framework: Add the ASM framework dependency in the Java Agent project. You can add the following dependency in the pom.xml file:

[0034] Implement the ClassFileTransformer interface: Create a class that implements the ClassFileTransformer interface, such as MyClassFileTransformer. In the transform method, use the ASM framework to modify the bytecode of JdbcTemplate / MyBatis related classes.

[0035] First, determine whether the class name belongs to JdbcTemplate or MyBatis related classes. Then iterate through the methods in the class, find the query and update methods, and use ASM's MethodVisitor to insert custom logic in the method body, printing a message to the console indicating that the SQL operation has been intercepted.

[0036] Through the above steps, when the Spring Boot application starts and loads the JdbcTemplate / MyBatis related classes, the Java Agent will intercept the class loading process and modify the bytecode using the ASM framework. In this way, when the related SQL operation method is executed, the inserted custom logic will be executed, thus achieving SQL interception. At the same time, since it is implemented through bytecode enhancement, there is no need to modify the source code of the application, achieving a non-invasive effect. More complex functions can be added in the custom logic, such as recording SQL statements, analyzing SQL performance, implementing SQL throttling, etc., to meet different business needs.

[0037] Among them, after this step, the standardization of the message body structure is also designed. First, determine the target and purpose of the message body, and clarify what the message body is used for, such as whether it is used for data exchange between internal systems, information sharing with external partners, or task instruction transmission in message queues. Different targets and purposes will determine the overall architecture and content of the message body.

[0038] Define the basic components of the message body, the message header: message identifier, used to uniquely identify the type or function of the message, such as "ORDER_CREATED" for an order creation message,

[0039] "PAYMENT_CONFIRMED" for a payment confirmation message, etc., which helps the recipient quickly identify the nature of the message and handle it accordingly. Sender information mainly records the identity of the message sender, such as the name of the sending system, module name or specific sender identification code, etc., to facilitate the recipient to understand the source of the message and trace the problem. The recipient information is the intended recipient of the message, which can be a specific system, module or recipient identifier, ensuring that the message can be accurately routed to the target location. The timestamp records the time of message generation, which is used to track the processing order and timeliness of the message, such as determining whether the message is overdue or processing is delayed. The message body content is the part that carries specific business data, and its structure and content vary according to the type of message.

[0040] For an order creation message, it may contain order number, order time, customer information, product list, order amount, etc. For a device status update message, it may contain device ID, device name, status code, status description, etc. For real-time data acquisition messages, it may contain sensor ID, acquisition time, acquisition data value, etc.

[0041] Fields.

[0042] Message trailer: Checksum, usually generated by CRC (Cyclic Redundancy Check) or other check algorithms, used to verify whether the message has been error or damaged during transmission, and the recipient can judge the integrity of the message through the checksum. Extension field, reserve some extension fields for future possible business demand expansion, such as adding some custom labels, tags or additional information.

[0043] Specify the data format and encoding method of the message body, specify the data format: you can choose common data formats such as JSON, XML, Protocol Buffers, etc. JSON has the characteristics of simplicity, readability, easy parsing and generation, and is widely used in web applications and mobile applications. XML has strict syntax rules and good extensibility, suitable for scenarios that require strict data structure and complex data exchange. Protocol Buffers is a high-efficiency binary data format with small storage space and fast encoding and decoding speed, suitable for scenarios with high performance requirements.

[0044] Specify the encoding method: determine the character encoding used, such as UTF-8, which can support multiple languages and character sets, ensuring that messages can be correctly transmitted and parsed in different systems and environments.

[0045] As the business develops and changes, the structure of the message body may need to be modified and extended. Therefore, a version management strategy needs to be developed. When the structure of the message body changes incompatibly, such as deleting or modifying important fields, adding new mandatory fields, etc., the version number of the message body needs to be upgraded. When processing messages, the receiving party needs to use the corresponding parsing logic according to the version number of the message to ensure that it can correctly process messages of different versions.

[0046] Establish and write specification documents for the message body, clearly describe each part of the message body, including field name, data type, length limit, value range, whether mandatory, etc. At the same time, provide some example messages to help developers better understand and use the message body structure. Specification documents not only help developers follow uniform standards during system development, but also facilitate subsequent maintenance and extension work.

[0047] In actual application, fully test and verify the message body structure to ensure that it can meet business requirements and performance requirements. Through functional testing, verify whether the message body can be correctly transmitted and parsed in different business scenarios to ensure that the data in the message can be accurately extracted and used. Performance testing: evaluate the encoding and decoding performance, transmission efficiency, and storage space occupation of the message body under different load conditions to ensure that it does not become a performance bottleneck of the system. Through compatibility testing, test in different environments and systems to ensure that the message body can work normally on various platforms and devices and maintain compatibility with other related systems and components.

[0048] S20: Introduce a traffic weight routing mechanism at the message queue level, based on the dynamic replication ratio adjustment strategy of the traffic weight routing mechanism in the time window, gradually adjust the traffic of the obtained SQL statement, wherein when the TP99 response time of the target database is monitored to exceed 500 ms, the system automatically triggers the degradation logic and switches to the asynchronous compensation mode;

[0049] Specifically, a suitable time window is determined according to the characteristics of the business and the frequency of data processing; the replication ratio of the message is dynamically adjusted according to the system load, message processing speed, and resource utilization rate in the time window. Set the TP99 threshold of the target database to a preset value, and use a database monitoring tool or the monitoring function of the message queue system to monitor the TP99 index of the database in real time. When it is found that the TP99 exceeds the preset value, the automatic degradation mechanism is triggered; in the asynchronous compensation mode, the message is first placed in an asynchronous queue; the asynchronous processing thread or task takes the message from the queue and performs a database write operation on the taken message; if the write fails, relevant information is recorded and compensation operations are performed subsequently.

[0050] According to the importance of different services, the size of data, real-time requirements and other factors, different traffic weights are allocated to each message queue or topic. For example, for the message queue of core business such as order processing, payment confirmation, etc., a higher weight is given to ensure that the messages of these key businesses can be processed in priority; while for some non-core businesses such as user behavior statistics, log recording, etc., a relatively low weight is allocated. In the configuration file or management interface of the message queue, the corresponding weight parameter is set. The weight can be represented by a number, such as 1-10, and the larger the number, the higher the weight. When the message producer sends a message, the message queue routes the message to different queues or topics for processing according to the pre-set weight rules. For example, the queue with a weight of 8 will have more message processing resources than the queue with a weight of 3.

[0051] A suitable time window is determined, such as 5 minutes, 10 minutes, etc. The length of this time window needs to be determined according to the characteristics of the business and the frequency of data processing. If the business data changes frequently, the time window can be set shorter to adjust the replication ratio in time; if the business data is relatively stable, the time window can be appropriately extended. According to the system load, message processing speed, resource utilization and other indicators within the time window, the replication ratio of the message is dynamically adjusted. For example, within a certain time window, if it is found that the message processing speed of a certain queue is faster, while other queues have message backlog, the message replication ratio of the queue can be appropriately increased, and a part of the messages are replicated to the queue with faster processing speed for processing, to achieve load balancing. At the same time, combined with the performance indicators of the database, such as TP99 (representing 99% of the request response time), when the TP99 of the database approaches or exceeds the set threshold, the replication ratio can also be adjusted to reduce the pressure on the database. For example, when the database TP99 reaches 400ms (close to the threshold of 500ms), the message replication ratio for writing to the database can be appropriately reduced to reduce the concurrent write operations to the database and avoid further deterioration of the database performance.

[0052] When the target database TP99 is greater than 500ms, automatically downgrade to asynchronous compensation mode, first set the TP99 threshold of the target database to 500ms. Through the database monitoring tool or the monitoring function of the message queue system, real-time monitoring of the TP99 index of the database. When it is found that TP99 exceeds 500ms, trigger the automatic degradation mechanism. In asynchronous compensation mode, the message processing flow changes. Originally, the message may be sent synchronously and the database confirmation of successful writing is waited for. Now, the message is put into an asynchronous queue first. Then, a special asynchronous processing thread or task takes the message from the queue and performs database writing operation. If the writing fails, record the relevant information and perform compensation operation in the future. For example, a timing task can be set to check the list of messages that have not been successfully written to the database every certain period of time, and retry writing until success. In this way, it can avoid a large number of message sending failures or backlog due to database performance problems, and ensure the stability and reliability of the system. At the same time, in asynchronous compensation mode, some preprocessing or caching of messages can be performed to reduce the pressure on the database. For example, some repeated messages are processed by merging, or some non-critical information is temporarily cached and written to the database after the database performance recovers. In MQ, traffic weight routing rules are introduced to support dynamic adjustment of replication ratio according to time window, and when the target database TP99 is greater than 500ms, automatically downgrade to asynchronous compensation mode. The following explains the design principle, implementation method and specific strategy in detail:

[0053] Weight allocation basis, first according to business priority allocation, according to the importance of business weight division, such as core transaction business (order creation, payment processing) weight set to 8-10, log collection, statistical analysis and other non-core business weight set to 1-3, ensure that the key business message is processed first. According to the size of data allocation, for large data volume of message flow (such as batch data synchronization), reduce its weight to avoid occupying too many resources, protect the processing of small data volume and high timeliness message. According to the service resource consumption allocation, for the message type that consumes more database, CPU and other resources, appropriately reduce the weight to prevent system overload.

[0054] In the configuration file of MQ (such as RabbitMQ's rabbitmq.conf, Kafka's server.properties extended configuration) or management console, set the weight parameter for each message queue or topic. For example, in Kafka, a topic weight can be identified by customizing the attribute weight = 5. Weighted round robin algorithm is adopted to allocate messages according to weights. Assuming that there are A (weight 3) and B (weight 2) queues, 3 out of every 5 messages are routed to A queue and 2 to B queue, realizing reasonable allocation of resources.

[0055] Determine the window length according to the characteristics of business traffic fluctuations, such as during financial trading, set a short window of 5-10 minutes to perceive traffic changes in real time; regular business scenarios can set a 15-30 minute window. In each time window, statistics such as message queue backlog, processing rate, resource utilization (such as CPU, memory usage) and other indicators are collected. For example, use Prometheus+Grafana combination to monitor Kafka's consumer_lag(consumer backlog), message_bytes_in(inbound byte count) and other indicators.

[0056] When the backlog of a certain queue is too high, dynamically increase its message replication ratio. For example, if the backlog of queue A reaches the threshold, increase the original 30% replication ratio to 50%, and divert part of the messages to queue A for processing. Adjust the replication ratio in combination with database performance indicators. When the database TP99 approaches the 500ms threshold (such as reaching 450ms), reduce the message replication ratio for writing to the database to avoid further increasing the pressure. For example, reduce the message replication ratio for writing to the database from 80% to 60%, reducing the number of concurrent writes.

[0057] Collect TP99 indicators in real time through database monitoring tools (such as MySQL's SHOW STATUS, TiDB's performance_schema) or middleware monitoring modules. When the target database TP99 response time exceeds 500ms, trigger the degradation mechanism. At the same time of triggering the degradation, send an alarm through email, SMS or internal IM system (such as DingTalk, Feishu) to inform the operation and development personnel of the database performance anomaly and degradation situation.

[0058] Change the original synchronous message writing to the database to first store it in an asynchronous queue (such as Redis's List, Kafka's specific Topic), and asynchronously consume and write it to the database by an independent consumer thread pool. For example, in a Spring Boot application, use the @Async annotation to mark the consumer method to achieve asynchronous processing. If the message fails to write to the database, use a retry mechanism (such as exponential backoff retry, initial interval 1 second, double the interval each time) and record the failed message to the log or compensation table, and periodically scan and reprocess it through a timing task (such as Quartz) to ensure data consistency. In the asynchronous compensation phase, if the database performance continues to deteriorate, you can limit the message consumption rate (such as token bucket algorithm to limit the consumption rate), or even shut down the writing of non-critical business messages to prioritize the stable operation of core business.

[0059]

[0060] ​Select the MQ that supports dynamic configuration and expansion, such as Kafka (create Topic dynamically through AdminClient, adjust partition) and RocketMQ (support traffic control plug-in). Use Prometheus+Grafana for index visualization, and combine Alertmanager to realize alarm pushing; or use cloud native monitoring services (such as Aliyun ARMS and Tencent cloud Prometheus service). In the asynchronous compensation mode, adjust the database connection pool parameters (such as HikariCP's maximumPoolSize and connectionTimeout) to avoid exhausting resources due to a large number of concurrent connections.

[0061] S30: In the data consumption end, build a SQL syntax conversion engine for the obtained SQL statement, define the syntax mapping rules from Oracle to TiDB through a YAML configuration file, and build a complete heterogeneous database SQL syntax adaptation system.

[0062] Specifically, based on the open source parser, the Oracle SQL statement is parsed into an abstract syntax tree (AST); based on the parsed abstract syntax tree (AST), combined with the mapping rules in the YAML configuration file, the Oracle syntax elements that need to be converted are located; the Oracle syntax node is replaced with a TiDB compatible node; according to the converted abstract syntax tree (AST), the TiDB executable SQL statement is reassembled. Determine the corresponding relationship between Oracle functions and TiDB functions; define the conversion logic for function parameters; establish the corresponding relationship between Oracle data types and TiDB data types.

[0063] The SQL syntax conversion engine is built at the consumer end, and the syntax mapping rules from Oracle to TiDB are defined with the help of YAML files, providing key technical support for database migration. First, open-source parsers such as JSqlParser, Apache Calcite, or ANTLR are used. Taking JSqlParser as an example, it can parse Oracle SQL statements into an abstract syntax tree (AST), providing structured data for subsequent conversion. For example, SELECT * FROM employees WHERE department_id = 10 will be broken down into a collection of nodes representing table names, fields, and conditions. Based on the parsed AST, combined with the mapping rules in the YAML configuration file, the syntax elements that need to be converted are located. When encountering the ROWNUM pagination syntax specific to Oracle, the corresponding conversion logic is found through rule matching. Replace the Oracle syntax node with a TiDB-compatible node, such as converting the Oracle DECODE function to the TiDB CASE WHEN structure. According to the converted AST, reassemble to generate TiDB executable SQL statements, ensuring syntax correctness and functional consistency.

[0064] The corresponding relationship between Oracle functions and TiDB functions is determined, such as TO_DATE corresponding to STR_TO_DATE and SYSDATE corresponding to NOW(). For function parameters, define conversion logic. For example, in the TO_DATE function, the Oracle date format YYYY needs to be converted to TiDB's %Y, which can be configured in YAML as transform: "replace(oracle_format,'YYYY','%Y')".

[0065] For SELECT * FROM DUAL syntax that can be directly used in TiDB, mark it as no need to convert; for Oracle pagination syntax SELECT * FROM (SELECT t.*, ROWNUM rn FROM table_name t WHERE ROWNUM <= 20) WHERE rn > 10, map it to TiDB's SELECT * FROM table_name LIMIT 10, 20. Support setting conversion conditions through the conditions field, for example, only when the statement contains the SYSDATE function, replace it with NOW(). Establish the corresponding relationship between Oracle and TiDB data types, such as NUMBER(10) converted to INT and VARCHAR2(255) converted to VARCHAR(255), to ensure data storage and query compatibility.

[0066] Read the rule file using the SnakeYAML library, parse it into a Map structure for subsequent query and application. With the help of JSqlParser, parse the SQL statement into AST, traverse the nodes and apply the mapping rules. Write auxiliary methods to implement complex conversion logic, such as handling nested functions, multi-condition statements, etc. For example, encapsulate the applyFunctionMapping method to handle the dynamic conversion of function parameters.

[0067] Write test cases for each mapping rule, use JUnit and other frameworks to verify the correctness of the conversion results, for example, input Oracle SQL containing the TO_DATE function, assert that the output is the correctly converted TiDB SQL. Integrate the conversion engine into the consumer-side business process, test with actual business SQL to ensure that it does not affect message consumption and database operations. Simulate high-concurrency scenarios to test the impact of the engine on message processing performance, optimize parsing and conversion efficiency, and avoid becoming a system bottleneck.

[0068] Integrate the conversion engine code directly into the consumer-side project and deploy it with the business application. Encapsulate the engine as a standalone REST service to provide conversion functions through HTTP interfaces, making it easier to manage and maintain. Store the YAML configuration file in a configuration center (such as Nacos, Apollo) to support dynamic updating of mapping rules without restarting the service, quickly adapting to new syntax conversion requirements. Record failed SQL statements and error information, and use log analysis tools or monitoring platforms to provide real-time alerts, making it easier to promptly identify omissions or errors in syntax mapping rules.

[0069] It can be seen that in the above scheme, for the traffic processing of the heterogeneous database, the bytecode enhancement module is loaded in the Spring Boot starting stage through the self-defined Java Agent, the bytecode is modified in a non-invasive manner based on the ASM framework, and the modified SQL statement is obtained; the traffic weight routing mechanism is introduced at the message queue level, the traffic weight routing mechanism is used to adjust the dynamic replication ratio strategy in the time window, and the traffic of the obtained SQL statement is gradually adjusted, wherein when the TP99 response time of the target database is monitored to be more than 500ms, the system automatically triggers the degradation logic and switches to the asynchronous compensation mode; the obtained SQL statement is constructed into a SQL syntax conversion engine in the data consumption end, the syntax mapping rules from Oracle to TiDB are defined through a YAML configuration file, and a complete heterogeneous database SQL syntax adaptation system is constructed. The application realizes the seamless migration from Oracle 11g to TiDB 5.0; the volume of the third-party package is less than 5MB, and only the Spring Boot starting script needs to be modified to add the '-javaagent' parameter; the application supports scenarios such as traffic mirroring (Mirror), gray release (1%->10%->100%), A / B testing, and the like, and the risk control system can dynamically set the replication strategy, such as preferentially replicating the business with the insurance amount greater than 1 million yuan, and realizes real-time synchronization of cross-machine room heterogeneous databases, desensitizes the production traffic and uses it for machine learning model training, and quickly constructs a fault injection test environment through traffic replication.

[0070] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the application.

[0071] In an embodiment, a traffic replication stress testing device for a heterogeneous database is provided, which corresponds to the traffic replication stress testing method for a heterogeneous database in the above embodiment. As shown in the figure, the traffic replication stress testing device for a heterogeneous database includes an interception module 101, an optimization module 102, and a conversion module 103. The functions of each module are described in detail as follows: Figure 3

[0072] The interception module 101 is configured to load a bytecode enhancement module in the Spring Boot starting stage through a self-defined Java Agent, modify the bytecode in a non-invasive manner based on an ASM framework, and obtain a modified SQL statement;

[0073] ​The optimization module 102 is used for introducing a traffic weight routing mechanism at a message queue level, and gradually adjusting the traffic of the obtained SQL statement based on a dynamic replication ratio adjustment strategy of the traffic weight routing mechanism in a time window. When the TP99 response time of the target database is monitored to exceed 500 ms, the system automatically triggers a degradation logic to switch to an asynchronous compensation mode.

[0074] The conversion module 103 is used for constructing a SQL syntax conversion engine in a data consumption end, defining an Oracle-to-TiDB syntax mapping rule through a YAML configuration file, and constructing a complete heterogeneous database SQL syntax adaptation system.

[0075] In an embodiment, the interception module 101 is specifically used for:

[0076] A Java project is created as a custom Java Agent, a class containing a premain method is defined in the project, and the premain method is set as an entry point of the Java Agent;

[0077] External configuration information is received through an agentArgs parameter;

[0078] In a startup script or configuration of a Spring Boot application, a-javaagent parameter is added to specify a path of the custom JavaAgent and related parameters.

[0079] In an embodiment, the optimization module 102 is specifically used for:

[0080] Different traffic weights are allocated to each message queue or topic according to importance, data size, real-time requirement and other factors of different businesses;

[0081] In a configuration file or management interface of the message queue, a weight parameter corresponding to the traffic weight is set.

[0082] A suitable time window is determined according to characteristics of the business and a data processing frequency;

[0083] The replication ratio of the message is dynamically adjusted according to system load, message processing speed and resource utilization rate in the time window.

[0084] The TP99 threshold of the target database is set as a preset value, the TP99 index of the database is monitored in real time through a database monitoring tool or a monitoring function of the message queue system, and when it is found that the TP99 exceeds the preset value, an automatic degradation mechanism is triggered;

[0085] In the asynchronous compensation mode, the message is first put into an asynchronous queue.

[0086] Retrieve an asynchronous processing thread or task from a queue message, and perform a database write operation on the retrieved message;

[0087] If the write fails, record the relevant information and perform a compensation operation subsequently.

[0088] In an embodiment, the conversion module 103 is specifically configured to:

[0089] Parse the Oracle SQL statement into an abstract syntax tree (AST) based on an open source parser;

[0090] Based on the parsed abstract syntax tree (AST), combined with the mapping rules in the YAML configuration file, the Oracle syntax elements to be converted are located;

[0091] Replace the Oracle syntax node with a TiDB compatible node;

[0092] According to the converted abstract syntax tree (AST), reassemble and generate a TiDB executable SQL statement.

[0093] Determine the corresponding relationship between the Oracle function and the TiDB function;

[0094] Define the conversion logic for the function parameters;

[0095] Establish the corresponding relationship between the Oracle data type and the TiDB data type.

[0096] The application provides a kind of flow replication stress testing device for heterogeneous database, by customizing

[0097] The Java Agent loads the bytecode enhancement module in the Spring Boot startup stage, the bytecode is modified in a non-invasive manner based on the ASM framework, and the modified SQL statement is obtained; a traffic weight routing mechanism is introduced at the message queue level, the traffic weight routing mechanism is used for dynamic copy ratio adjustment in a time window, and the traffic of the obtained SQL statement is gradually adjusted, wherein when the TP99 response time of the target database is monitored to be more than 500 ms, the system automatically triggers the degradation logic and switches to an asynchronous compensation mode; the obtained SQL statement is constructed into a SQL syntax conversion engine in the data consumption end, the syntax mapping rules from Oracle to TiDB are defined by using a YAML configuration file, and a complete heterogeneous database SQL syntax adaptation system is constructed. The application realizes seamless migration from Oracle 11g to TiDB 5.0; the volume of the third-party package is less than 5MB, and only the '-javaagent' parameter needs to be added in the Spring Boot startup script; the application supports scenarios such as traffic mirroring (Mirror), gray release (1%->10%->100%), A / B testing, and the like, and the risk control system can dynamically set the copy strategy, such as preferentially copying the business with a policy amount greater than 1 million yuan, and realizes real-time synchronization of cross-machine room heterogeneous databases, desensitizes the production traffic and uses the desensitized production traffic for machine learning model training, and quickly constructs a fault injection test environment through traffic replication.

[0098] The specific limitations of the traffic replication stress testing device for heterogeneous databases can refer to the limitations of the traffic replication stress testing method for heterogeneous databases in the foregoing, and will not be described here. Each module in the traffic replication stress testing device for heterogeneous databases can be realized by software, hardware and combinations thereof, in whole or in part. Each module can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so that the processor calls and executes the operations corresponding to each module.

[0099] In one embodiment, a computer device can be a server, and an internal structure diagram thereof can be as shown in Figure 4 The computer device includes a processor, a memory, a network interface and a database connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium, an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with the external client through the network connection. The computer program is executed by the processor to implement the functions or steps of the traffic replication stress testing method for heterogeneous databases on the server side.

[0100] In one embodiment, a computer device is provided, which can be a client, and an internal structure diagram thereof can be as shown in Figure 5 The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is configured to communicate with an external server through a network connection. The computer program is executed by the processor to implement the functions or steps of a traffic replication stress testing method for a heterogeneous database on the client side

[0101] In one embodiment, a computer device is provided, which includes a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the following steps when executing the computer program:

[0102] A bytecode enhancement module is loaded in the Spring Boot startup phase through a custom Java Agent, and the bytecode is modified in a non-invasive manner based on an ASM framework to obtain modified SQL statements;

[0103] A traffic weight routing mechanism is introduced at the message queue level, and a dynamic replication ratio adjustment strategy based on the traffic weight routing mechanism is used to gradually adjust the traffic of the obtained SQL statements. When the TP99 response time of the target database is monitored to exceed 500 ms, the system automatically triggers a degradation logic to switch to an asynchronous compensation mode;

[0104] A SQL syntax conversion engine is constructed from the obtained SQL statements at the data consumption end, and the SQL statements are converted into TiDB syntax through

[0105] A YAML configuration file defines the syntax mapping rules from Oracle to TiDB to construct a complete heterogeneous database SQL syntax adaptation system.

[0106] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0107] A bytecode enhancement module is loaded in the Spring Boot startup phase through a custom Java Agent, and the bytecode is modified in a non-invasive manner based on an ASM framework to obtain modified SQL statements;

[0108] Introducing a traffic weight routing mechanism at the message queue level, based on the dynamic replication ratio adjustment strategy of the traffic weight routing mechanism in the time window, gradually adjusting the traffic of the obtained SQL statement, wherein when the TP99 response time of the target database is monitored to exceed 500 ms, the system automatically triggers the degradation logic and switches to the asynchronous compensation mode;

[0109] In the data consumption end, the obtained SQL statement is constructed into a SQL syntax conversion engine, and the SQL statement is converted into a TiDB SQL statement through

[0110] The YAML configuration file defines the syntax mapping rules from Oracle to TiDB, and constructs a complete heterogeneous database SQL syntax adaptation system.

[0111] It should be noted that the functions or steps that the computer readable storage medium or the computer device can implement correspond to the related descriptions of the server side and the client side in the foregoing method embodiments, and to avoid repetition, they will not be described one by one here.

[0112] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. In the embodiments provided in the present application, any reference to memory, storage, database or other medium can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0113] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0114] The above-described embodiments are only used to illustrate the technical solutions of the present application, but not limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those ordinarily skilled in the art should understand: the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for traffic replication stress testing of heterogeneous databases, characterized in that, include: By using a custom Java Agent to load the bytecode enhancement module during the Spring Boot startup phase, the bytecode is non-intrusively modified based on the ASM framework to obtain the modified SQL statement; A traffic weight routing mechanism is introduced at the message queue level. Based on the traffic weight routing mechanism, the dynamic replication ratio adjustment strategy within the time window is used to gradually adjust the traffic of SQL statements. When the TP99 response time of the target database exceeds 500ms, the system automatically triggers degradation logic and switches to asynchronous compensation mode. On the data consumption side, the acquired SQL statements are used to build an SQL syntax conversion engine. The syntax mapping rules from Oracle to TiDB are defined through YAML configuration files to build a complete heterogeneous database SQL syntax adaptation system.

2. The traffic replication stress testing method for heterogeneous databases as described in claim 1, characterized in that, The steps described above for loading the bytecode enhancement module during the Spring Boot startup phase using a custom Java Agent, performing non-intrusive bytecode modification based on the ASM framework, and obtaining the modified SQL statement include: Create a Java project as a custom Java Agent, and define a class in the project that contains a premain method, which is set as the entry point of the Java Agent; Receive configuration information from external sources via the agentArgs parameter; In the startup script or configuration of a Spring Boot application, add the -javaagent parameter to specify the path and related parameters of a custom JavaAgent and obtain the bytecode; The bytecode is modified non-intrusively using the ASM framework to obtain the modified SQL statement.

3. The traffic replication stress testing method for heterogeneous databases as described in claim 1, characterized in that, The introduction of a traffic weight routing mechanism at the message queue level, and the gradual adjustment of the traffic for acquiring SQL statements based on the dynamic replication ratio adjustment strategy within the time window using this mechanism, wherein the system automatically triggers degradation logic and switches to asynchronous compensation mode when the TP99 response time of the target database exceeds 500ms, includes the following steps: Different traffic weights are assigned to each message queue or topic based on factors such as the importance of different businesses, the size of data, and the real-time requirements. In the message queue's configuration file or management interface, set the corresponding weight parameters for the traffic weight.

4. The traffic replication stress testing method for heterogeneous databases as described in claim 3, characterized in that, After the step of setting the corresponding weight parameters for traffic weights in the message queue configuration file or management interface, the method further includes: Determine an appropriate time window based on the characteristics of the business and the frequency of data processing; The message replication ratio is dynamically adjusted based on system load, message processing speed, and resource utilization metrics within the time window.

5. The traffic replication stress testing method for heterogeneous databases as described in claim 4, characterized in that, After the step of dynamically adjusting the message replication ratio based on system load, message processing speed, and resource utilization indicators within the time window, the method further includes: Set the TP99 threshold of the target database to a preset value, and monitor the database's TP99 index in real time through database monitoring tools or the monitoring function built into the message queue system. When the TP99 exceeds the preset value, trigger the automatic degradation mechanism. In asynchronous compensation mode, messages are first placed into an asynchronous queue; Retrieve messages from the queue of asynchronous processing threads or tasks, and write the retrieved messages to the database. If the write operation fails, record the relevant information and perform compensation operations later.

6. The traffic replication stress testing method for heterogeneous databases as described in claim 1, characterized in that, The steps involved in building an SQL syntax conversion engine from the acquired SQL statements at the data consumption end, defining the Oracle-to-TiDB syntax mapping rules through a YAML configuration file, and constructing a complete heterogeneous database SQL syntax adaptation system include: This method uses an open-source parser to parse Oracle SQL statements into an Abstract Syntax Tree (AST). Based on the parsed Abstract Syntax Tree (AST) and the mapping rules in the YAML configuration file, the Oracle syntax elements that need to be converted are located. Replace the Oracle syntax nodes with TiDB-compatible nodes; Based on the transformed Abstract Syntax Tree (AST), the executable SQL statements for TiDB are reassembled.

7. The traffic replication stress testing method for heterogeneous databases as described in claim 6, characterized in that, After the step of reassembling and generating TiDB executable SQL statements based on the transformed Abstract Syntax Tree (AST), the method further includes: Determine the correspondence between Oracle functions and TiDB functions; Define the conversion logic for the function parameters; Establish the correspondence between Oracle data types and TiDB data types.

8. A traffic replication stress testing device for heterogeneous databases, characterized in that, include: The interception module is used to load the bytecode enhancement module during the Spring Boot startup phase through a custom Java Agent, and to perform non-intrusive modification of the bytecode based on the ASM framework to obtain the modified SQL statement; The optimization module is used to introduce a traffic weight routing mechanism at the message queue level. Based on the traffic weight routing mechanism, the dynamic replication ratio adjustment strategy within the time window is used to gradually adjust the traffic of SQL statements. When the TP99 response time of the target database exceeds 500ms, the system automatically triggers degradation logic and switches to asynchronous compensation mode. The conversion module is used to build an SQL syntax conversion engine from the acquired SQL statements on the data consumption side. It defines the syntax mapping rules from Oracle to TiDB through YAML configuration files to build a complete SQL syntax adaptation system for heterogeneous databases.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the traffic replication stress testing method for heterogeneous databases as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the traffic replication stress testing method for heterogeneous databases as described in any one of claims 1 to 7.

Images