Distributed system processing method and apparatus, electronic device, and storage medium
By creating shadow tables in the database management system and intercepting and rewriting SQL statements, the problems of resource waste and high code maintenance costs in parallel development by multiple teams are solved, and efficient multi-lane test environment isolation and transparent routing of business code are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING QIYI CENTURY SCI & TECH CO LTD
- Filing Date
- 2026-02-12
- Publication Date
- 2026-06-02
AI Technical Summary
Existing technologies suffer from resource waste, low testing efficiency, and high code maintenance costs when multiple teams are developing in parallel. This is especially true in test scenarios involving database configuration modifications, where setting up a complete test pipeline is time-consuming, the code structure is chaotic, and there is a lack of a unified management mechanism, which affects the continuity and flexibility of the testing process.
By receiving swimlane configuration requests, generating data processing instructions to create shadow tables, copying target data, intercepting and rewriting SQL statements using preset swimlane configuration rules, and replacing online table names with shadow table names, data environment isolation is achieved, avoiding modification of business code and deployment of new services.
It achieves resource conservation and improved testing efficiency through multi-lane parallel testing, ensures the maintainability of business code and the continuity of the testing process, and reduces operational complexity and resource consumption.
Smart Images
Figure CN122132307A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of distributed processing technology, and in particular to a distributed system processing method, apparatus, electronic device and storage medium. Background Technology
[0002] In modern software development processes, parallel development of different functionalities by multiple teams has become the mainstream approach. To ensure the stability and compatibility of new functionalities, testing and verification need to be conducted under conditions close to production environments. Swimlane technology, by constructing multiple logically isolated traffic channels within the same environment, achieves the goal of parallel testing of multiple features without interference, becoming a key technological direction for solving the needs of parallel testing.
[0003] For test scenarios involving database configuration modifications in the testing process, existing technologies mainly offer two implementation schemes: First, build a complete test process that includes all services, and use the same full data as the online test process; second, embed swimlane routing logic rules in the business code, and after receiving swimlane requests, redirect database requests to the specified test table or test database according to the rules.
[0004] However, all of the above-mentioned existing solutions have significant drawbacks: For Solution 1, building a complete test link requires a large amount of computing and storage resources, and the link deployment takes a long time. When there are multiple functions that need to be tested in parallel, multiple test links need to be built repeatedly, resulting in resource waste and low testing efficiency. For Solution 2, a large amount of routing code unrelated to the core business logic needs to be added to the business code, resulting in a messy code structure. Frequent code modifications due to non-business needs will increase system maintenance costs. At the same time, the routing rules are scattered in various services and lack a unified management mechanism. After the rules are changed, the service needs to be restarted to take effect, which seriously affects the continuity and flexibility of the testing process.
[0005] Therefore, there is an urgent need to develop a distributed system processing method, device, electronic equipment, and storage medium to solve one or more of the aforementioned problems. Summary of the Invention
[0006] In view of this, to solve the above-mentioned technical problems or some of the technical problems, embodiments of the present invention provide a distributed system processing method, apparatus, electronic device and storage medium. The method receives a swimlane configuration request initiated by a user, generates a data processing instruction based on the request and sends it to a database management system. The database management system creates a swimlane shadow table with the same structure as the online table and copies the target data within a specified range. At the same time, based on preset swimlane configuration rules, when a database access request carrying a swimlane identifier is received, the SQL statement is intercepted and rewritten, and the online table name is replaced with the corresponding shadow table name before routing to the swimlane shadow table. This achieves the goal of providing independent data environments for different swimlanes through the shadow table mechanism without modifying the business code, effectively isolating parallel test traffic, reducing resource consumption, and improving test efficiency and system maintainability.
[0007] In a first aspect, this application provides a distributed system processing method, the method comprising: Receive a swimlane configuration request initiated by a user. The swimlane configuration request carries a business application identifier, a swimlane identifier, and shadow table data requirement parameters. In response to the swimlane configuration request, a corresponding data processing instruction is generated, which includes the online table identifier, swimlane identifier, and data copy range of the business application; The data processing instruction is sent to the database management system, so that the database management system creates a swimlane shadow table with the same structure as the online table according to the data processing instruction, copies the target data of a specified range from the corresponding online table to the swimlane shadow table, and returns the swimlane shadow table after the copy is completed. Obtain preset lane configuration rules, which include the mapping relationship between lane identifiers and shadow table names; When a database access request carrying a swimlane identifier is received, the SQL statement corresponding to the database access request is intercepted. According to the swimlane configuration rules, the online table name in the SQL statement is rewritten to the corresponding shadow table name, and the rewritten SQL request is routed to the swimlane shadow table.
[0008] In one possible implementation, the method further includes: When a database access request without a lane identifier is received, the database access request is routed to the corresponding online table.
[0009] In one possible implementation, rewriting the online table name in the SQL statement to the corresponding shadow table name according to the swimlane configuration rules includes: Perform syntax parsing on the intercepted SQL statements to identify table name keywords in the SQL statements and the online table names corresponding to the table name keywords; According to the lane configuration rules, obtain the shadow table name corresponding to the lane identifier; Rewrite the online table names in the SQL statement with the corresponding shadow table names, and verify the compliance of the rewritten SQL statement.
[0010] In one possible implementation, the method further includes: When no shadow table name matches the swimlane identifier, the database access request is routed to the corresponding online table, and an alarm notification is triggered. The alarm notification carries the swimlane identifier for which no shadow table name was matched.
[0011] In one possible implementation, the method further includes: Obtain the database access logs for each lane, including access timestamps, SQL statements, number of rows affected by the SQL statements, and the dataset returned by the database. A log isolation and analysis mechanism is established based on swimlane identifiers to isolate and analyze database access logs for different swimlanes. When a risk of cross-lane data access or shadow table data contamination is detected, an alarm is triggered and abnormal information is logged.
[0012] In one possible implementation, the method further includes: Obtain the validity period for each lane, where the validity period includes the start time and end time; After the validity period of the swimlane expires, a shadow table cleanup command is sent to the database management system, so that the database management system can perform data backup and deletion operations on the shadow table corresponding to the expired swimlane according to the shadow table cleanup command, thereby releasing database storage resources.
[0013] In one possible implementation, the method further includes: At preset time intervals, initiate swimlane configuration rule version verification requests to determine whether there are any version changes to the current swimlane configuration rules; When the swimlane configuration rules are changed, the updated swimlane configuration rules are used as the current swimlane configuration rules.
[0014] Secondly, this application provides a distributed system processing apparatus, the apparatus comprising: The receiving module is used to receive swimlane configuration requests initiated by users. The swimlane configuration requests carry business application identifiers, swimlane identifiers, and shadow table data requirement parameters. The generation module is used to generate corresponding data processing instructions in response to the swimlane configuration request. The data processing instructions include the online table identifier, swimlane identifier, and data copy range of the business application. The copy module is used to send the data processing instructions to the database management system, so that the database management system can create a swimlane shadow table with the same structure as the online table according to the data processing instructions, copy the target data of a specified range from the corresponding online table to the swimlane shadow table, and return the swimlane shadow table after the copy is completed. The acquisition module is used to acquire preset lane configuration rules, which include the mapping relationship between lane identifiers and shadow table names; The processing module is used to intercept the SQL statement corresponding to the database access request when it receives a database access request carrying a swimlane identifier, rewrite the online table name in the SQL statement to the corresponding shadow table name according to the swimlane configuration rules, and route the rewritten SQL request to the swimlane shadow table.
[0015] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the distributed system processing method described in any embodiment of the first aspect.
[0016] Fourthly, this application also provides a computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the distributed system processing method described in any embodiment of the first aspect.
[0017] Compared with the prior art, the technical solution provided in this application has the following advantages: The method provided in this application receives a swimlane configuration request initiated by a user, generates a data processing instruction containing the online table identifier of the business application, the swimlane identifier, and the data copy range based on the business application identifier, swimlane identifier, and shadow table data requirement parameters carried in the request, and sends it to the database management system. This enables the database management system to create a swimlane shadow table with the same structure as the online table and copy the target data of the specified range from the corresponding online table to the swimlane shadow table. This achieves automated creation and data initialization of the shadow table, eliminating the need for manual intervention in database table structure creation and data migration, greatly reducing operational complexity and improving configuration efficiency.
[0018] Meanwhile, through preset swimlane configuration rules, which include the mapping relationship between swimlane identifiers and shadow table names, when a database access request carrying a swimlane identifier is received, the corresponding SQL statement can be automatically intercepted. According to the configuration rules, the online table name in the SQL statement is accurately rewritten to the corresponding shadow table name, and the rewritten SQL request is routed to the swimlane shadow table. This achieves automated and transparent data routing for different swimlane requests, avoids embedding a large amount of routing logic in the business code, and ensures the maintainability of the business code. Attached Figure Description
[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0021] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.
[0022] Figure 1 A flowchart illustrating a distributed system processing method provided in an embodiment of this application; Figure 2 This is a schematic diagram illustrating the steps of a distributed system processing method provided in an embodiment of this application; Figure 3 A schematic diagram illustrating the steps of an online table name changing method provided in this application embodiment; Figure 4 A schematic diagram of the structure of dynamic routing for a shadow table provided in an embodiment of this application; Figure 5 A schematic diagram of the internal execution of a Rover provided for an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a distributed system processing device provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0024] The following disclosure provides numerous different embodiments or examples for implementing various structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of the invention. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.
[0025] To address the problems existing in current technologies, conducting simultaneous testing of multiple functions requires building multiple test pipelines, which not only consumes significant resources and time but also leads to the writing of a large amount of code unrelated to business logic, resulting in code clutter. Furthermore, code modifications are frequently made for non-business needs, and routing rules are scattered across various services, making unified management difficult. Changes to swimlane rules also require service restarts. This application provides a distributed system processing method, apparatus, electronic device, and storage medium. By employing shadow tables, data from multiple swimlanes can be isolated and prevented from interfering with each other. Simultaneously, new features can be verified in an online environment without modifying business code or deploying new services, thus achieving this goal.
[0026] Java agents are Java probe technologies provided by the JDK, implemented based on the JVM Tool Interface. By combining Java agent technology with the Java Instrumentation API (Application Programming Interface), it is possible to dynamically modify application functionalities, thereby enabling a wide range of rich features. Common applications include code debugging, diagnostic tools, and application enhancement logic.
[0027] By building a Java agent, a proxy program independent of the business logic is constructed. This proxy program listens for database requests from the application. Upon intercepting a request, it determines whether it matches the configured rules. If a match is found, it parses the SQL statement according to the user-configured rules, replacing the table names in the SQL statement with the user-configured shadow table names. This allows different swimlanes to access different shadow databases, fulfilling the requirement for multi-swimlane parallel testing. Figure 1 This is a flowchart illustrating a distributed system processing method provided in an embodiment of this application, as shown below. Figure 1 As shown, the method specifically includes: S101. Receive a swimlane configuration request initiated by the user. The swimlane configuration request carries a business application identifier, a swimlane identifier, and shadow table data requirement parameters.
[0028] Swimlane configuration requests refer to configuration instructions initiated by users in distributed system testing scenarios to independently test new functions or modules of specific business applications.
[0029] Among them, the business application identifier is used to uniquely identify the business system to be tested, such as "e-commerce transaction platform" or "financial payment system"; the swimlane identifier is an independent identifier assigned to the current test task, such as "test swimlane 001" or "new function verification swimlane A". This identifier can isolate the data and traffic of different test tasks; the shadow table data requirement parameters specify in detail the source range and filtering conditions of the data required by the shadow table. For example, it can be "transaction order data of the last 7 days", "user information with user IDs between 1000 and 2000", or "order records with a status of 'pending payment'". The requirement parameters ensure that the shadow table can simulate the real online data environment and provide accurate data support for testing.
[0030] S102. In response to the swimlane configuration request, generate a corresponding data processing instruction, which includes the online table identifier of the business application, the swimlane identifier, and the data copy range.
[0031] In this embodiment, upon receiving a swimlane configuration request, the business application identifier carried in the request is parsed to determine its corresponding online database and specific online table, thereby obtaining the online table identifier. The swimlane identifier is directly obtained from the configuration request for subsequent naming and identification of shadow tables. The data copy range is determined based on the shadow table data requirement parameters. For example, if the requirement parameter is "transaction order data for the past 7 days", it will be converted into a specific time range condition, such as "create_time>= '2025-01-01 00:00:00' AND create_time<= '2025-01-07 23:59:59'", which serves as the data copy range in the data processing instruction, ensuring that the database management system can accurately copy the target data that meets the conditions.
[0032] S103. Send the data processing instruction to the database management system so that the database management system creates a swimlane shadow table with the same structure as the online table according to the data processing instruction, copies the target data of a specified range from the corresponding online table to the swimlane shadow table, and returns the swimlane shadow table after the copy is completed.
[0033] In this embodiment, after receiving a data processing instruction, the database management system first reads the metadata information of the corresponding online table based on the online table identifier in the instruction. This metadata includes field names, data types, lengths, constraints, and configuration parameters such as the table's storage engine and character set. Using this as a template, a swimlane shadow table with a completely identical structure is created. Then, the database management system filters out the target data that meets the conditions from the corresponding online table according to the data copy range specified in the data processing instruction.
[0034] It should be noted that during the data copying process, to ensure data integrity and consistency, for example, for tables containing auto-incrementing primary keys, the original primary key values will be retained to maintain the relationships between data; for fields with default values or triggers, they will also be processed according to the rules of the online tables. After the data copy is complete, the database management system will return relevant information about the copied swimlane shadow tables to the distributed system processing unit, such as the shadow table name, data volume, copy time, etc., so that the unit can perform subsequent swimlane configuration and request routing operations.
[0035] Shadow tables are typically named with swimlane identifiers for easy differentiation and management. For example, if the online table name is "order_info" and the swimlane identifier is "test_001", then the shadow table name can be "order_info_shadow_test_001".
[0036] S104. Obtain preset lane configuration rules, wherein the lane configuration rules include the mapping relationship between lane identifiers and shadow table names.
[0037] The preset swimlane configuration rules can be predefined and dynamically updated by the administrator through a visual interface or configuration file. The rules contain the mapping relationship between swimlane identifiers and shadow table names, which precisely associates each independent swimlane identifier with one or more corresponding shadow table names. For example, swimlane identifier "test_001" corresponds to shadow table names "order_info_shadow_test_001" and "user_info_shadow_test_001", swimlane identifier "feature_A_002" corresponds to shadow table name "goods_info_shadow_feature_A_002", and so on.
[0038] In this embodiment, the latest swimlane configuration rules are obtained from the configuration center through periodic polling or event listening to ensure that the processing module uses the currently effective mapping relationship when rewriting SQL, thus avoiding routing errors caused by untimely rule updates.
[0039] S105. When a database access request carrying a swimlane identifier is received, the SQL statement corresponding to the database access request is intercepted, the online table name in the SQL statement is rewritten to the corresponding shadow table name according to the swimlane configuration rules, and the rewritten SQL request is routed to the swimlane shadow table.
[0040] In this embodiment, when a database access request carrying a swimlane identifier is received, the request is first parsed to extract the swimlane identifier. Then, using Java agent technology combined with the Java Instrumentation API, a proxy program independent of the business program is constructed. This proxy program listens for and intercepts database requests from the business program, thereby obtaining the SQL statement corresponding to the database access request. Further, based on the obtained preset swimlane configuration rules, which contain the mapping relationship between swimlane identifiers and shadow table names, after intercepting the SQL statement, it is determined whether the SQL statement matches the current configuration rules. If it matches, the structure of the SQL statement is parsed to locate the online table name. Then, according to the correspondence between swimlane identifiers and shadow table names in the swimlane configuration rules, the online table name in the SQL statement is automatically rewritten to the corresponding shadow table name.
[0041] For example, if a received database access request carries a swimlane identifier of "test_001", and the online table involved in the SQL statement is named "order_info", according to the preset mapping relationship that "test_001" corresponds to "order_info_shadow_test_001", then "order_info" in the SQL statement will be rewritten as "order_info_shadow_test_001". After rewriting the SQL statement, the rewritten SQL request is routed to the swimlane shadow table corresponding to that swimlane identifier, enabling the database access request to accurately access the shadow table data of that swimlane. This achieves automated and transparent data routing for different swimlane requests, and the entire process does not require embedding additional routing logic in the business code.
[0042] The distributed system processing method provided in this application receives a swimlane configuration request initiated by a user. Based on the business application identifier, swimlane identifier, and shadow table data requirement parameters carried in the request, it generates a data processing instruction containing the online table identifier of the business application, swimlane identifier, and data copy range. This instruction is then sent to the database management system, enabling the database management system to create a swimlane shadow table with the same structure as the online table and copy the target data within a specified range from the corresponding online table to the swimlane shadow table. This achieves automated creation and data initialization of the shadow table, eliminating the need for manual intervention in database table structure creation and data migration, greatly reducing operational complexity and improving configuration efficiency.
[0043] Meanwhile, through preset swimlane configuration rules, which include the mapping relationship between swimlane identifiers and shadow table names, when a database access request carrying a swimlane identifier is received, the corresponding SQL statement can be automatically intercepted. According to the configuration rules, the online table name in the SQL statement is accurately rewritten to the corresponding shadow table name, and the rewritten SQL request is routed to the swimlane shadow table. This achieves automated and transparent data routing for different swimlane requests, avoids embedding a large amount of routing logic in the business code, and ensures the maintainability of the business code.
[0044] Figure 2 This is a schematic diagram illustrating the steps of a distributed system processing method provided in an embodiment of this application, as shown below. Figure 2 As shown, the distributed system processing method specifically includes the following implementation steps: Step 1: After completing their tasks, developers will submit a request to the configuration management platform for swimlane configuration of the business application. This request includes the information and parameters required to create the swimlane.
[0045] Step 2: After receiving the business application swimlane configuration request submitted by the developer, the configuration management platform will further submit a requirement to the data DBA system based on the content of the request. This requirement mainly concerns the creation of a shadow table corresponding to the swimlane, and also involves the operation requirement of copying the original data into this shadow table.
[0046] Step 3: After receiving the requirements from the configuration management platform, the DBA system begins to execute the operation of creating a swimlane shadow table, accurately constructing the shadow table according to the established rules and format requirements.
[0047] Step 4: After the swimlane shadow table is created, the next task of the DBA system is to copy the relevant data from the original location to this newly created shadow table to ensure the integrity and accuracy of the data.
[0048] Step 5: After the above series of operations are completed, the entire process will enter the stage of returning the scenario results, which is a summary and feedback of all previous operations, so that relevant personnel can understand the current status.
[0049] Step Six: Subsequently, the system will notify the corresponding business applications to obtain the configured swimlanes, ensuring that the business applications can know and use the new swimlane configuration in a timely manner.
[0050] Step 7: Finally, return a message indicating that the configuration change is complete.
[0051] Among them, the configuration management platform is a platform for centralized management of swimlane configurations; the DBA system is a database management system that can realize functions such as database table creation and data copying; Rover is a specific implementation of Java agent technology, which can pull data from the configuration management platform and perform functions such as shadow table dynamic routing; business applications are a general term for business systems.
[0052] In an optional embodiment of the present invention, the method further includes: when a database access request without a lane identifier is received, routing the database access request to the corresponding online table.
[0053] In this embodiment, database access requests without swimlane identifiers are identified as regular online business requests by default. In this case, no table name rewriting operation is performed on the SQL statement corresponding to the request. Instead, it is directly routed to the original online database table to ensure that normal production business data is not affected by the test environment and to maintain the stable operation of online business.
[0054] The above processing method effectively isolates test traffic from production traffic, ensuring that while using shadow tables for multi-lane parallel testing, it does not interfere with the reading and writing of real online business data.
[0055] Figure 3 This application provides a schematic diagram illustrating the steps of an online table name changing method, as shown in the embodiments below. Figure 3 As shown, rewriting the online table name in the SQL statement to the corresponding shadow table name according to the swimlane configuration rules includes: S301. Perform syntax parsing on the intercepted SQL statement to identify the table name keywords in the SQL statement and the online table names corresponding to the table name keywords.
[0056] In this embodiment, an SQL syntax parser is used to perform lexical and syntactic analysis on the intercepted SQL statements, decomposing the SQL statements into an abstract syntax tree (AST) structure.
[0057] In the abstract syntax tree, by traversing the nodes, the keywords representing table names, such as "FROM" and "JOIN", are identified, followed by table name identifiers, thereby accurately locating all online table names involved in the SQL statement.
[0058] S302. According to the lane configuration rules, obtain the shadow table name corresponding to the lane identifier.
[0059] Swimlane configuration rules are stored in key-value pairs, where the key is the swimlane identifier and the value is a set of all related shadow table names under that swimlane.
[0060] In this embodiment, based on the swimlane identifier extracted from the database access request, a precise matching query is performed within the preset swimlane configuration rules. By traversing this set, the shadow table name corresponding to the online table name parsed from the current SQL statement is found.
[0061] S303. Rewrite the online table name in the SQL statement with the corresponding shadow table name, and verify the compliance of the rewritten SQL statement.
[0062] In this embodiment, the online table name node that needs to be replaced is located in the abstract syntax tree and replaced with the corresponding shadow table name obtained in step S302. After the replacement is completed, the rewritten SQL statement is checked for syntax compliance, such as checking whether there are syntax errors after the table name is replaced, whether the fields match the table structure, and whether the logic of the SQL statement is complete.
[0063] If the verification passes, the final executable rewritten SQL statement is generated; if the verification fails, the error message is recorded and the original SQL statement is returned or the exception handling mechanism is triggered to ensure that the SQL request routed to the shadow table is legal and valid, and to avoid database access errors caused by SQL rewriting errors.
[0064] The online table name modification method provided in this application performs syntax parsing on the intercepted SQL statement to identify the table name keywords and their corresponding online table names. Then, it obtains the shadow table name corresponding to the swimlane identifier according to the swimlane configuration rules, replaces the online table name in the SQL statement with the shadow table name, and performs strict compliance verification on the rewritten SQL statement. This ensures the accuracy and effectiveness of the SQL statement rewriting and lays a solid foundation for correctly routing the rewritten SQL request to the swimlane shadow table, effectively avoiding database access problems caused by incorrect table name rewriting.
[0065] In an optional embodiment of the present invention, the method further includes: When no shadow table name matches the swimlane identifier, the database access request is routed to the corresponding online table, and an alarm notification is triggered. The alarm notification carries the swimlane identifier for which no shadow table name was matched.
[0066] In this embodiment, when querying the corresponding shadow table name from the swimlane configuration rules based on the swimlane identifier, if no matching record is found, it is determined that the shadow table corresponding to the current swimlane identifier does not exist or there is an omission in the configuration rules.
[0067] At this point, to avoid any unforeseen impact on online data from the test request, the distributed system processing unit will route the database access request to the corresponding online table by default, ensuring data access security. Simultaneously, the system will immediately trigger an alarm notification mechanism, sending an alarm message to the administrator through pre-defined alarm channels. This alarm notification will explicitly include the swimlane identifier for the unmatched shadow table name, enabling the administrator to quickly locate the problem, check for errors in swimlane configuration rules, successful creation of the shadow table, or correct execution of data processing instructions, and promptly repair and adjust accordingly, ensuring the stability and reliability of the swimlane testing environment.
[0068] In an optional embodiment of the present invention, the method further includes: Obtain database access logs for each lane, including access timestamps, SQL statements, number of rows affected by the SQL, and database returned datasets; establish a log isolation analysis mechanism based on lane identifiers to isolate and analyze database access logs for different lanes; when cross-lane data access or shadow table data pollution risks are detected, trigger an alarm and record abnormal information.
[0069] In this embodiment, database access logs for each lane are collected in real time. Each log entry records the precise timestamp of the access, the full text of the executed SQL statement, the number of rows affected by the SQL statement in the database, and the result dataset returned by the database.
[0070] Then, a complete log isolation and analysis mechanism is built based on swimlane identifiers, allocating independent log storage and analysis space for each swimlane to ensure that access logs from different swimlanes are not mixed up.
[0071] By conducting in-depth analysis of these isolated logs, such as comparing table name access patterns and data modification traces in SQL statements across different swimlanes, potential risky behaviors can be effectively identified. When an SQL request from one swimlane attempts to access shadow table data in other swimlanes, or when unexpected modifications are made to shadow table data in one swimlane, or when abnormal associations are established with online table data or other shadow table data, an alarm process is immediately triggered. Detailed information, including the time of the anomaly, the swimlane identifier involved, the specific SQL statement, and the anomaly type, is recorded in the system anomaly log, allowing administrators to intervene promptly and ensure the independence and security of data in each swimlane.
[0072] In an optional embodiment of the present invention, the method further includes: Obtain the validity period of each lane, which includes the start time and end time; after the validity period of the lane expires, send a shadow table cleanup instruction to the database management system, so that the database management system can perform data backup and deletion operations on the shadow table corresponding to the expired lane according to the shadow table cleanup instruction, thereby releasing database storage resources.
[0073] When creating a swimlane shadow table, the validity period of each swimlane is configured synchronously. This validity period is generally set by the administrator in the swimlane configuration request according to the test requirements, and specifically includes the test start time and end time.
[0074] In this embodiment, all created swimlanes are periodically scanned, and the current system time is compared with the validity period of each swimlane to determine whether the swimlane has expired. When the validity period of a swimlane is detected to have expired, a shadow table cleanup command is automatically sent to the database management system.
[0075] The cleanup command includes explicit identifiers for expired swimlanes and a list of corresponding shadow table names. Upon receiving the cleanup command, the database management system (DBMS) performs a complete backup of the shadow table data corresponding to these expired swimlanes. The backup file is stored in a designated backup storage path according to preset naming rules, allowing for data recovery or auditing when needed. After completing the data backup and confirming the integrity of the backup file, the DBMS then performs the shadow table deletion operation, completely removing these unused shadow tables and their related indexes, constraints, and other database objects.
[0076] By performing the above-mentioned checks on validity periods, the occupied database storage space can be released in a timely manner, avoiding storage pressure caused by invalid data occupying resources for a long time, thereby improving the overall operating efficiency and resource utilization of the database.
[0077] In an optional embodiment of the present invention, the method further includes: At preset time intervals, initiate swimlane configuration rule version verification requests to determine whether there are any version changes to the current swimlane configuration rules; when there are version changes to the swimlane configuration rules, obtain the updated swimlane configuration rules and determine them as the current swimlane configuration rules.
[0078] In this embodiment, the agent program will proactively send swimlane configuration rule version verification requests to the configuration management platform at preset time intervals. This request includes the configuration rule version number currently used by the agent program.
[0079] Upon receiving a request, the configuration management platform compares the version number in the request with the latest version number stored on the platform. If the comparison results match, meaning the current configuration rules have not changed, the configuration management platform returns a response indicating no update is needed. If the comparison results do not match, indicating that the swimlane configuration rules have been updated, the configuration management platform returns the updated complete swimlane configuration rules to the agent program. After obtaining the updated swimlane configuration rules, the agent program immediately replaces the locally cached old version with the new rules and establishes the new rules as the currently effective swimlane configuration rules. This ensures that subsequent interception of database access requests and SQL rewriting logic can be executed based on the latest configuration rules, thereby adapting to dynamically changing swimlane testing needs and guaranteeing the accuracy and flexibility of data routing.
[0080] Dynamic routing technology intercepts database SQL requests using bytecode enhancement techniques in a Java agent, then dynamically rewrites the table names in the SQL statements, routing swimlane requests to shadow tables.
[0081] Figure 4 This application provides a schematic diagram of the structure of a dynamic routing table for a shadow table, as shown in the embodiment of the present application. Figure 4 As shown, the implementation process of the online actual execution process and the swimlane test execution process of the operation management system is as follows: Operations or product personnel configure operational data through the back-end management system. Upon receiving a request, the back-end management system modifies the online database. Online users access the configuration data through the business system, which reads the data from the online tables.
[0082] The execution process for the lane test is as follows: When QA / testing personnel send a request to the backend management system, they include a swimlane identifier. Rover will then route the database request to the corresponding swimlane table based on this identifier. Similarly, when QA / testing personnel read data from the business system, they also include the swimlane identifier, and Rover will again route the database request to the corresponding swimlane table based on this identifier.
[0083] This allows online requests and swimlane requests to read different data within the same service without affecting each other, thus enabling the verification and testing of new features in an online environment.
[0084] Figure 5 A flowchart illustrating the internal execution of a Rover instance is provided as an embodiment of this application, such as... Figure 5 As shown, Rover's internal execution flow includes the following steps: Step 1: When the business system receives a request and needs to perform related operations on database resources, it should first determine whether the request carries a swimlane identifier; if the check finds that the request does not carry a swimlane identifier, proceed directly to step 5; if it is confirmed that the request has carried a swimlane identifier, proceed to step 2.
[0085] Step 2: In this step, the corresponding configuration rules need to be obtained from the configuration management platform, and then these configuration rules are matched with the current request. If a corresponding shadow resource is found through matching, proceed to Step 3; otherwise, skip the subsequent intermediate steps and directly execute Step 5.
[0086] Step 3: After confirming the matching of shadow resources, bytecode enhancement technology is needed to modify the table names in the SQL statement. Specifically, the original table name pointing to the online table is changed to the shadow table name of the corresponding swimlane. After completing the modification, continue execution to step 4.
[0087] Step 4: After the table name has been correctly changed to the shadow table name, send a request to the database for the shadow table to obtain the required data or perform related operations.
[0088] Step 5: If, in the previous steps, it was found that the request did not carry a swimlane identifier or failed to match a shadow resource, then it is necessary to send a request to the database for the online table to ensure that the business can run normally and obtain the required data.
[0089] Step Six: At this point, the entire process is complete, and both the shadow table and the online table have received the corresponding processing results.
[0090] Figure 6 This is a schematic diagram of the structure of a distributed system processing device provided in an embodiment of this application, as shown below. Figure 6 As shown, the device specifically includes: The receiving module 601 is used to receive a swimlane configuration request initiated by a user. The swimlane configuration request carries a business application identifier, a swimlane identifier, and shadow table data requirement parameters. The generation module 602 is used to generate a corresponding data processing instruction in response to the swimlane configuration request. The data processing instruction includes the online table identifier, swimlane identifier, and data copy range of the business application. The copy module 603 is used to send the data processing instruction to the database management system, so that the database management system creates a swimlane shadow table with the same structure as the online table according to the data processing instruction, copies the target data of a specified range from the corresponding online table to the swimlane shadow table, and returns the swimlane shadow table after the copy is completed. The acquisition module 604 is used to acquire preset lane configuration rules, the lane configuration rules including the mapping relationship between lane identifiers and shadow table names; The processing module 605 is used to intercept the SQL statement corresponding to the database access request when it receives a database access request carrying a swimlane identifier, rewrite the online table name in the SQL statement to the corresponding shadow table name according to the swimlane configuration rules, and route the rewritten SQL request to the swimlane shadow table.
[0091] In one possible implementation, the processing module 605 is further configured to route the database access request to the corresponding online table when a database access request without a lane identifier is received.
[0092] In one possible implementation, the processing module 605 is further configured to perform syntax parsing on the intercepted SQL statement, identify table name keywords in the SQL statement and the online table names corresponding to the table name keywords; obtain the shadow table names corresponding to the swimlane identifiers according to the swimlane configuration rules; rewrite the online table names in the SQL statement to the corresponding shadow table names, and verify the compliance of the rewritten SQL statement.
[0093] In one possible implementation, the processing module 605 is further configured to route the database access request to the corresponding online table and trigger an alarm notification when no shadow table name matching the swimlane identifier exists, the alarm notification carrying the swimlane identifier for which no shadow table name was matched.
[0094] In one possible implementation, the device further includes an isolation module 606 (not shown in the figure) for acquiring database access logs for each lane, the database access logs including access timestamps, SQL statements, number of rows affected by the SQL, and database returned datasets; establishing a log isolation analysis mechanism based on lane identifiers to perform isolated analysis on database access logs for different lanes; and triggering an alarm and recording abnormal information when a risk of cross-lane data access or shadow table data contamination is detected.
[0095] In one possible implementation, the device further includes a period monitoring module 607 (not shown in the figure) for obtaining the validity period of each lane, the validity period including the start time and the end time; after the validity period of the lane expires, a shadow table cleanup instruction is sent to the database management system, so that the database management system performs data backup and deletion operations on the shadow table corresponding to the expired lane according to the shadow table cleanup instruction, thereby releasing database storage resources.
[0096] In one possible implementation, the device further includes an update module 608 (not shown in the figure), which is used to initiate a swimlane configuration rule version verification request at a preset time interval to determine whether there is a version change in the current swimlane configuration rule; when there is a version change in the swimlane configuration rule, the updated swimlane configuration rule is obtained and determined as the current swimlane configuration rule.
[0097] The distributed system processing device provided in this embodiment can be as follows: Figure 6 The distributed system processing device shown can perform, for example... Figure 1-5 All steps of the distributed system processing, thereby achieving Figure 1-5 For details on the technical effects of the distributed system processing shown, please refer to [link / reference]. Figure 1-5 The relevant descriptions are presented concisely and will not be elaborated upon here.
[0098] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0099] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 7 As shown, this application embodiment provides an electronic device, including a processor 701, a communication interface 702, a memory 703, and a communication bus 704. The processor 701, communication interface 702, and memory 703 communicate with each other via the communication bus 704. The memory 703 stores computer programs. When the processor 701 executes the program stored in the memory 703, it implements the steps of the distributed system processing method provided in any of the aforementioned method embodiments. The system receives a swimlane configuration request initiated by a user, the request carrying a business application identifier, a swimlane identifier, and shadow table data requirement parameters; in response to the swimlane configuration request, it generates a corresponding data processing instruction, the instruction including the online table identifier of the business application, the swimlane identifier, and the data copy range; it sends the data processing instruction to the database management system, so that the database management system creates a swimlane shadow table with the same structure as the online table according to the instruction, copies the target data of the specified range from the corresponding online table to the swimlane shadow table, and returns the completed swimlane shadow table; it obtains a preset swimlane configuration rule, the rule including the mapping relationship between swimlane identifiers and shadow table names; when a database access request carrying a swimlane identifier is received, it intercepts the SQL statement corresponding to the database access request, rewrites the online table name in the SQL statement to the corresponding shadow table name according to the swimlane configuration rule, and routes the rewritten SQL request to the swimlane shadow table.
[0100] In one possible implementation, when a database access request without a lane identifier is received, the database access request is routed to the corresponding online table.
[0101] In one possible implementation, the intercepted SQL statement is parsed to identify table name keywords in the SQL statement and the corresponding online table names; according to the swimlane configuration rules, the shadow table name corresponding to the swimlane identifier is obtained; the online table name in the SQL statement is rewritten to the corresponding shadow table name, and the compliance of the rewritten SQL statement is verified.
[0102] In one possible implementation, when no shadow table name matches the swimlane identifier, the database access request is routed to the corresponding online table, and an alarm notification is triggered, the alarm notification carrying the swimlane identifier for which no shadow table name was matched.
[0103] In one possible implementation, database access logs for each swimlane are obtained, including access timestamps, SQL statements, number of rows affected by the SQL statements, and database returned datasets; a log isolation analysis mechanism is established based on swimlane identifiers to isolate and analyze the database access logs of different swimlanes; when cross-swimlane data access or shadow table data pollution risks are detected, an alarm is triggered and abnormal information is recorded.
[0104] In one possible implementation, the validity period of each lane is obtained, the validity period including the start time and the end time; after the validity period of the lane expires, a shadow table cleanup instruction is sent to the database management system, so that the database management system performs data backup and deletion operations on the shadow table corresponding to the expired lane according to the shadow table cleanup instruction, thereby releasing database storage resources.
[0105] In one possible implementation, a swimlane configuration rule version verification request is initiated at a preset time interval to determine whether there is a version change in the current swimlane configuration rule; when there is a version change in the swimlane configuration rule, the updated swimlane configuration rule is obtained and determined as the current swimlane configuration rule.
[0106] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0107] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.
[0108] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.
Claims
1. A distributed system processing method, characterized in that, The method includes: Receive a swimlane configuration request initiated by a user. The swimlane configuration request carries a business application identifier, a swimlane identifier, and shadow table data requirement parameters. In response to the swimlane configuration request, a corresponding data processing instruction is generated, which includes the online table identifier, swimlane identifier, and data copy range of the business application; The data processing instruction is sent to the database management system, so that the database management system creates a swimlane shadow table with the same structure as the online table according to the data processing instruction, copies the target data of a specified range from the corresponding online table to the swimlane shadow table, and returns the swimlane shadow table after the copy is completed. Obtain preset lane configuration rules, which include the mapping relationship between lane identifiers and shadow table names; When a database access request carrying a swimlane identifier is received, the SQL statement corresponding to the database access request is intercepted. According to the swimlane configuration rules, the online table name in the SQL statement is rewritten to the corresponding shadow table name, and the rewritten SQL request is routed to the swimlane shadow table.
2. The method according to claim 1, characterized in that, The method further includes: When a database access request without a lane identifier is received, the database access request is routed to the corresponding online table.
3. The method according to claim 1, characterized in that, The step of rewriting the online table name in the SQL statement to the corresponding shadow table name according to the swimlane configuration rules includes: Perform syntax parsing on the intercepted SQL statements to identify table name keywords in the SQL statements and the online table names corresponding to the table name keywords; According to the lane configuration rules, obtain the shadow table name corresponding to the lane identifier; Rewrite the online table names in the SQL statement with the corresponding shadow table names, and verify the compliance of the rewritten SQL statement.
4. The method according to claim 3, characterized in that, The method further includes: When no shadow table name matches the swimlane identifier, the database access request is routed to the corresponding online table, and an alarm notification is triggered. The alarm notification carries the swimlane identifier for which no shadow table name was matched.
5. The method according to claim 1, characterized in that, The method further includes: Obtain the database access logs for each lane, including access timestamps, SQL statements, number of rows affected by the SQL statements, and the dataset returned by the database. A log isolation and analysis mechanism is established based on swimlane identifiers to isolate and analyze database access logs for different swimlanes. When a risk of cross-lane data access or shadow table data contamination is detected, an alarm is triggered and abnormal information is logged.
6. The method according to claim 5, characterized in that, The method further includes: Obtain the validity period for each lane, where the validity period includes the start time and end time; After the validity period of the swimlane expires, a shadow table cleanup command is sent to the database management system, so that the database management system can perform data backup and deletion operations on the shadow table corresponding to the expired swimlane according to the shadow table cleanup command, thereby releasing database storage resources.
7. The method according to claim 1, characterized in that, The method further includes: At preset time intervals, initiate swimlane configuration rule version verification requests to determine whether there are any version changes to the current swimlane configuration rules; When the swimlane configuration rules are changed, the updated swimlane configuration rules are used as the current swimlane configuration rules.
8. A distributed system processing device, characterized in that, The device includes: The receiving module is used to receive swimlane configuration requests initiated by users. The swimlane configuration requests carry business application identifiers, swimlane identifiers, and shadow table data requirement parameters. The generation module is used to generate corresponding data processing instructions in response to the swimlane configuration request. The data processing instructions include the online table identifier, swimlane identifier, and data copy range of the business application. The copy module is used to send the data processing instructions to the database management system, so that the database management system can create a swimlane shadow table with the same structure as the online table according to the data processing instructions, copy the target data of a specified range from the corresponding online table to the swimlane shadow table, and return the swimlane shadow table after the copy is completed. The acquisition module is used to acquire preset lane configuration rules, which include the mapping relationship between lane identifiers and shadow table names; The processing module is used to intercept the SQL statement corresponding to the database access request when it receives a database access request carrying a swimlane identifier, rewrite the online table name in the SQL statement to the corresponding shadow table name according to the swimlane configuration rules, and route the rewritten SQL request to the swimlane shadow table.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the distributed system processing method according to any one of claims 1 to 7.
10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the distributed system processing method according to any one of claims 1 to 7.