Data processing method and apparatus

Abstract

Embodiments of the present specification provide a data processing method and device, wherein the data processing method comprises: in response to an operation test request for a target program, determining a reference database and a non-reference database in at least two databases associated with the target program; determining a data identifier associated with the operation test request in the reference database, and determining to-be-processed data identifiers associated with the operation test request in the non-reference database; updating the to-be-processed data identifiers based on the data identifier, obtaining a target database, and executing a test task associated with the operation test request based on the reference database and the target database.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to data processing methods and apparatus. Background Technology

[0002] In software development and quality assurance, operational testing of application systems with multi-database architectures, such as traffic replay, regression testing, or integration testing, is a crucial step in ensuring functional correctness and data consistency. Existing testing methods typically initiate operational requests to the target program in an isolated environment, relying on each database to independently generate response data. However, in scenarios involving multiple interconnected databases, each database often automatically generates random values ​​during testing, causing subsequent test steps that rely on these random values ​​to fail to correctly correlate data, resulting in invalid tests. Therefore, a more effective data processing method is urgently needed to address these issues. Summary of the Invention

[0003] In view of the above, embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0004] According to a first aspect of the embodiments of this specification, a data processing method is provided, comprising:

[0005] In response to an operational test request for the target program, a baseline database and a non-baseline database are determined from at least two databases associated with the target program;

[0006] The data identifier associated with the operation test request is determined in the benchmark database, and the data identifier to be processed associated with the operation test request is determined in the non-benchmark database.

[0007] The data identifier to be processed is updated based on the data identifier to obtain the target database, and the test task associated with the operation test request is executed based on the benchmark database and the target database.

[0008] According to a second aspect of the embodiments of this specification, a data processing apparatus is provided, comprising:

[0009] The first determining module is configured to determine a benchmark database and a non-benchmark database in response to an operational test request for the target program in at least two databases associated with the target program;

[0010] The second determining module is configured to determine the data identifier associated with the operation test request in the benchmark database, and to determine the data identifier to be processed associated with the operation test request in the non-benchmark database.

[0011] The execution module is configured to update the data identifier to be processed based on the data identifier, obtain the target database, and execute the test task associated with the operation test request based on the benchmark database and the target database.

[0012] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising:

[0013] Memory and processor;

[0014] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0015] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the data processing method described above.

[0016] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0017] This specification provides a data processing method according to one embodiment, which, in response to an operation test request for a target program, determines a baseline database and a non-baseline database in at least two databases associated with the target program. A data identifier associated with the operation test request is determined in the baseline database, and a pending data identifier associated with the operation test request is determined in the non-baseline database. The pending data identifier is updated based on the data identifier to obtain the target database, achieving data identifier synchronization between the baseline and non-baseline databases. Executing the test task associated with the operation test request based on the baseline and target databases avoids the problem of the non-baseline database generating random pending data identifiers in response to the operation test request, leading to subsequent test task execution failures, thus improving the test success rate. Even if the execution of the operation test request results in different data identifiers being added to the baseline and non-baseline databases, subsequent test tasks executed based on the data identifiers can still be completed through data identifier synchronization, ensuring the effectiveness of traffic testing based on the operation test request and avoiding interference from random values. Attached Figure Description

[0018] Figure 1This is a schematic diagram of a data processing method provided in one embodiment of this specification;

[0019] Figure 2 This is a flowchart illustrating a data processing method provided in one embodiment of this specification;

[0020] Figure 3 This is a schematic diagram illustrating a data processing method provided in one embodiment of this specification.

[0021] Figure 4 This is an interactive schematic diagram of a data processing method provided in one embodiment of this specification;

[0022] Figure 5 This is a schematic diagram of the structure of a data processing apparatus provided in one embodiment of this specification;

[0023] Figure 6 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0024] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0025] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0026] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0027] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0028] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0029] MySQL: A widely used open-source relational database management system.

[0030] DM: Dameng Database is a relational database management system.

[0031] MariaDB: An open-source relational database management system created by the founder of MySQL. As a community-driven fork of MySQL, it aims to remain open-source, free, and continuously innovate.

[0032] OceanBase: A distributed relational database with financial-grade high availability, strong consistency, and horizontal scalability.

[0033] Traffic mirroring / traffic replay testing refers to capturing and copying the actual request traffic (including HTTP requests, database operations, message queue events, etc.) generated in the production environment (or by real users) without loss, and then replaying it in the test environment, shadow system, or new version of the service without affecting online business, in order to verify the functional correctness, performance, and stability of the target system.

[0034] Traffic Multi-Sending / Multi-Target Traffic Replay refers to sending (or distributing) the same original request traffic to multiple target systems or service instances simultaneously (such as different versions of services, different databases, or different middleware) to verify the consistency of behavior or performance differences among multiple targets in parallel.

[0035] When testing a target program based on data processing requests within an application service scenario, the testing objective can be to test the database's functionality or data processing logic, or it can be to test the target program's functionality. Operational test requests for the target program can be data processing requests within the application service scenario, including data insertion requests, as well as data query requests, data deletion requests, and / or data modification requests. Testing the target program can also be a high-volume traffic test within a traffic replication test scenario.

[0036] Figure 1 This diagram illustrates a data processing method according to an embodiment of this specification. In response to an operation test request for a target program, a baseline database and a non-baseline database are determined in at least two databases associated with the target program. A data identifier associated with the operation test request is determined in the baseline database, and a data identifier to be processed associated with the operation test request is determined in the non-baseline database. The data identifier to be processed is updated based on the data identifier to obtain the target database, achieving data identifier synchronization between the baseline and non-baseline databases. Executing the test task associated with the operation test request based on the baseline and target databases avoids the problem of the non-baseline database generating random data identifiers to be processed in response to the operation test request, leading to subsequent test task execution failures, thus improving the test success rate. Even if the execution of the operation test request results in different data identifiers being added to the baseline and non-baseline databases, subsequent test tasks executed based on the data identifiers can still be completed through data identifier synchronization, ensuring the effectiveness of traffic testing based on the operation test request and avoiding interference from random values.

[0037] This specification provides a data processing method, and also relates to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0038] See Figure 2 , Figure 2 A flowchart of a data processing method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0039] Step S202: In response to an operational test request for the target program, determine a baseline database and a non-baseline database in at least two databases associated with the target program.

[0040] Specifically, the target program can be any application or application system that involves multiple database operations. Operation test requests are used to test the target program in terms of functionality, execution logic, and other dimensions. The target program must be associated with at least two databases used to store business data related to the target program. These at least two databases can be of different types, including but not limited to MySQL, OceanBase, SQL Server, MariaDB, etc. Any one of the at least two databases can be selected as the baseline database; databases not selected from the at least two databases are considered non-baseline databases. Operation test requests can correspond to data storage operations, i.e., data insertion operations, used to insert key-value pairs of data into at least two databases.

[0041] Based on this, in response to an operational test request for the target program, at least two databases associated with the target program are identified. A baseline database and a non-baseline database are then determined from these two databases according to a database selection strategy. The database selection strategy can be to select any one of the at least two databases as the baseline database, or to specify a target database type and select the database matching that type from the at least two databases as the baseline database.

[0042] Furthermore, considering that the target program corresponds to at least two target subroutines, and each target subroutines correspond to one of at least two databases, upon receiving an operation test request, each target subroutine must first execute the data operation task corresponding to the operation test request. After the data operation task is completed and data is received from at least two target subroutines, the benchmark database is then determined. The specific implementation is as follows:

[0043] The system receives the operation test request for the target program and sends the operation test request to at least two target subroutines corresponding to the target program; upon receiving the execution feedback data from the at least two target subroutines, the system designates the databases corresponding to the at least two target subroutines as the at least two databases associated with the target program; and determines the benchmark database and the non-benchmark database from the at least two databases.

[0044] Specifically, the target program can be an application system, which contains at least two target subroutines with independent databases. The databases of the at least two target subroutines are independent of each other, and the databases corresponding to each target subroutine can be of different types. Execution feedback data indicates that each target subroutine has successfully completed the data operation corresponding to the operation test request.

[0045] Based on this, an operation test request for the target program is received and sent to at least two target subroutines corresponding to the target program. Each target subroutine executes the data operation corresponding to the operation test request and, upon completion of the data operation, returns execution completion data. The execution completion data from the at least two target subroutines constitutes the execution feedback data. Receiving execution feedback data from at least two target subroutines indicates that each target subroutine has completed the data operation corresponding to the operation test request. The databases corresponding to the at least two target subroutines are designated as the at least two databases associated with the target program. A baseline database and a non-baseline database are determined from these at least two databases.

[0046] For example, in a high-traffic testing scenario, an operation test request can be an HTTP request executed during a traffic replication test. Upon receiving an operation test request initiated through a browser, the gateway first sends the request to each target subroutine corresponding to the target program. These subroutines, acting as backend programs, need to execute the data operations corresponding to the operation test request, updating their respective databases. After each target subroutine completes its data operations, data indicating the completion of the data operations for the operation test request is generated, obtaining execution feedback data from at least two target subroutines. For each target subroutine's database, one database is selected as the baseline database, and the unselected databases are designated as non-baseline databases. Figure 3 As shown in (a), the operation test request can be submitted through the display page. The MySQL database, as the penetrating database corresponding to the display page, can have data operations performed by the backend program corresponding to the MySQL database. The operation test request can be sent by the gateway to the target subprograms corresponding to the MySQL database, DM database, and OB database, respectively. Each target subprogram performs data operations, storing the value 1 in the MySQL database through the MySQL data service layer, storing the value 10 in the DM database through the DM data service layer, and storing the value 3 in the OB database through the OB data service layer.

[0047] In summary, upon receiving execution feedback data from at least two target subroutines, the databases corresponding to each of the at least two target subroutines are designated as the at least two databases associated with the target program. This ensures that the operation test request is completed by each target subroutine, and that at least two databases have completed the data processing associated with the operation test request, before executing subsequent test tasks for the target program, thereby increasing the probability of successful test task execution.

[0048] Furthermore, considering that not all types of operation test requests involve data operations on at least two databases, it is necessary to detect the operation type of the operation test request before processing it. The specific implementation is as follows:

[0049] Receive an operation test request for the target program; perform operation type detection on the operation test request to obtain the target operation type corresponding to the operation test request; if the target operation type is an insertion operation type, determine the baseline database and non-baseline database in at least two databases associated with the target program.

[0050] Specifically, performing operation type detection on operation test requests involves detecting the data operation type corresponding to the operation test request. Data operation types include, but are not limited to, insert, delete, update, and query operations.

[0051] Based on this, an operation test request for the target program is received, and the operation type of the request is detected to obtain the target operation type corresponding to the operation test request. If the target operation type is an insert operation, data insertion operations need to be performed in each of at least two databases. For the data to be inserted, if an identifier or key (id) needs to be assigned, each database can add the identifier using an auto-incrementing key-value pair, and the newly added identifiers may be different for each database. If the newly added identifiers are different for each database, subsequent data query and update tests based on the newly added identifiers will fail, ultimately causing the high-traffic test to fail. Therefore, upon receiving an operation test request, it is necessary to determine the insert operation type corresponding to the operation test request, and thus identify a baseline database and a non-baseline database among at least two databases associated with the target program.

[0052] In summary, when the target operation type is an insert operation, a baseline database and a non-baseline database are determined in at least two databases associated with the target program. This ensures that in high-traffic testing scenarios, the problem of invalid high-traffic tests caused by the insertion of random data identifiers into the database can be resolved.

[0053] Step S204: Determine the data identifier associated with the operation test request in the benchmark database, and determine the data identifier to be processed associated with the operation test request in the non-benchmark database.

[0054] Specifically, in response to an operational test request for the target program, after determining the baseline database and the non-baseline database in at least two databases associated with the target program, the data identifier associated with the operational test request can be determined in the baseline database, and the data identifier to be processed associated with the operational test request can be determined in the non-baseline database. The data identifier associated with the operational data request can be a data identifier newly added to the baseline database in response to the operational test request. The data identifier to be processed can be a data identifier newly added to the non-baseline database in response to the operational test request, and the data identifier and the data identifier to be processed are different.

[0055] Based on this, after determining the baseline database and non-baseline database in at least two databases associated with the target program in response to an operation test request for the target program, a data identifier associated with the operation test request is determined in the baseline database, and a data identifier to be processed associated with the operation test request is determined in the non-baseline database. The data identifier is stored in the baseline database in response to the operation test request, and the data identifier to be processed is stored in the non-baseline database in response to the operation test request. The data identifier can be a key-value pair determined in the baseline database using an auto-incrementing id. The data identifier to be processed can be a key-value pair determined in the non-baseline database using an auto-incrementing id.

[0056] Furthermore, considering that the benchmark database stores a large amount of data, to ensure that data identifiers can be quickly determined in the benchmark database, a trigger module can be configured for the data tables in the benchmark database to retrieve data identifiers. The specific implementation is as follows:

[0057] The data table associated with the operation test request is determined in the benchmark database, and the trigger module configured in the data table is used to obtain the data identifier associated with the operation test request.

[0058] Specifically, the data table associated with the operation test request refers to the data table that performs data operations in response to the operation test request. In this embodiment, when the operation test request corresponds to a data insertion operation, after executing the operation test request, data and its corresponding data identifier are inserted into the data table. The data identifier can be a UUID or an ID determined by an auto-incrementing ID in the benchmark database. The triggering module can be a trigger used to obtain the data identifier when an insertion operation type corresponding to the operation test request is detected.

[0059] Based on this, the data table associated with the operation test request that has undergone data operation is determined in the benchmark database. The data operation type of the operation test request is detected. If the data operation type corresponding to the operation test request is detected to be an insert operation type, the trigger module configured in the data table is used to obtain the data identifier associated with the operation test request.

[0060] Continuing with the previous example, with the MySQL database designated as the baseline database, trigger modules are configured for each table within the MySQL database; that is, triggers are configured. After the MySQL database completes the data insertion operation for the operation test request, it is determined that the table in which the operation test request was performed contains the data insertion operation. The table that performed the data insertion operation in response to the operation test request is identified, and the data identifier id=1 stored in the table in response to the operation test request is retrieved using the trigger configured for that table.

[0061] In summary, when the data operation type corresponding to the operation test request is detected to be an insert operation type, the trigger module configured in the data table is used to obtain the data identifier associated with the operation test request, thereby improving the accuracy of data identifier determination.

[0062] Step S206: Update the data identifier to be processed based on the data identifier to obtain the target database, and execute the test task associated with the operation test request based on the benchmark database and the target database.

[0063] Specifically, after determining the data identifier associated with the operation test request in the benchmark database and the data identifier to be processed associated with the operation test request in the non-benchmark database, the data identifier to be processed can be updated based on the data identifier to obtain the target database. Then, the test task associated with the operation test request can be executed based on the benchmark database and the target database. The test task is a data processing task that performs data operations on the data identifier. By executing the data processing task, the benchmark database and the non-benchmark database can be determined whether they pass the test based on the results of the data processing task. The test task can be used to test the data read and write functions of the database.

[0064] Based on this, after determining the data identifier associated with the operation test request in the benchmark database and the data identifier to be processed associated with the operation test request in the non-benchmark database, the data identifier is updated based on the data identifier to obtain the target database. Then, the test task associated with the operation test request is executed based on the benchmark database and the target database. The test results of functional testing on the benchmark database and the non-benchmark database are determined based on the test task execution results.

[0065] Furthermore, considering that the benchmark database and non-benchmark databases simultaneously executed the data insertion operation corresponding to the operation test request, the specific implementation is as follows:

[0066] In the non-benchmark database, determine the target data table associated with the operation test request, and in the target data table, determine the target record associated with the operation test request; update the data identifier to be processed in the target record to the data identifier, and obtain the target database.

[0067] Specifically, the target data table associated with the test request refers to the data table in which data operations are performed in non-benchmark data in response to the operation test request. In this embodiment, when the operation test request corresponds to a data insertion operation, after the operation test request is executed, data and the corresponding data identifier to be processed are inserted into the target data table of the non-benchmark database. The data identifier to be processed can be a UUID or an ID determined by auto-incrementing the ID in the non-benchmark database.

[0068] Based on this, the target data table associated with the operation test request and the data operation performed is identified in the non-benchmark database, and the target record of the target data table that responded to the operation test request is identified in the target data table. The pending data identifier in the target record is updated to the data identifier, thereby updating the non-benchmark database and obtaining the target database. The data operation performed in response to the operation test request can be a data insertion operation, inserting data and its corresponding pending data identifier into the target data table.

[0069] Continuing with the previous example, such as Figure 3As shown in (a), the DM and OB databases are non-baseline databases. In these databases, data insertion operations assign a data identifier (id value) to the data to be inserted using an auto-incrementing id method. The DM database inserts a data identifier of 10 using an auto-incrementing method, storing the id value 10 and the data to be inserted into the corresponding target data table. Similarly, the OB database inserts a data identifier of 3 using an auto-incrementing method, storing the id value 3 and the data to be inserted into the corresponding target data table. Therefore, given that the DM and OB databases are non-baseline databases, the target data tables related to the data insertion operation—that is, the tables where the data to be inserted—can be found in both the DM and OB databases. Further, the data identifier to be processed, 10, is determined in the target data table of the DM database, and the data identifier to be processed, 3, is determined in the target data table of the OB database. The data identifier to be processed, 10, in the DM database is replaced with the data identifier 1 in the MySQL database, and the data identifier to be processed, 3, in the OB database is replaced with the data identifier 1 in the MySQL database. At this point, the MySQL database, the DM database, and the OB database have the same data identifier 1 (id) for the operation test request, realizing the unification of random id values. This facilitates subsequent database logic tests and database function tests based on the unified data identifier.

[0070] In summary, updating the pending data identifier in the target record with a data identifier enables the update of the non-benchmark database, thereby obtaining the target database. This ensures that the benchmark and non-benchmark databases have the same data identifier for operation test requests, facilitating subsequent database logic testing and database function testing based on a unified data identifier, and avoiding interference from random ID values ​​in high-traffic tests.

[0071] Furthermore, historical data identifiers stored in the non-benchmark database are all stored in the representation record table. This indicates that historical data identifiers have been stored in the non-benchmark database. Therefore, when determining the target record in the target data table, the representation record table can be searched. The specific implementation is as follows:

[0072] At least one candidate data record is determined in the target data table, and a candidate data identifier corresponding to each of the at least one candidate data record is determined; a target data identifier matching the identifier record table is determined from the candidate data identifiers, and the target data record corresponding to the target data identifier is used as the target record associated with the operation test request.

[0073] Specifically, all records stored in the target data table can be used as candidate data records, and the data identifiers stored in the target candidate data records serve as candidate data identifiers. The identifier record table records the used data identifiers in chronological order, which are used for identifier synchronization between databases. The target data identifier that matches the identifier record table among the candidate data identifiers is the candidate data identifier that is not in the identifier record table.

[0074] Based on this, at least one candidate data record is identified in the target data table, and the data identifier corresponding to each candidate data record is used as the candidate data identifier. An identifier record table is searched based on the candidate data identifiers, and the target data identifier that matches the identifier record table is determined from the candidate data identifiers. The candidate data identifier not found in the identifier record table is then used as the target data identifier. The target data record corresponding to the target data identifier is then used as the target record associated with the operation test request.

[0075] The identifier record table is a configuration table for the random number synchronization program. It records data identifier synchronization between the baseline database and non-baseline databases. The data identifier synchronization record contains the tables that need ID synchronization between the baseline and non-baseline databases, as well as a list of fields. The fields in the identifier record table can be identified by recognizing the primary keys in the tables that need ID synchronization between the baseline and non-baseline databases, and configuring the primary keys accordingly. The field list can be all fields with auto-incrementing keys and UUID values. For example, the configuration file for identifying the tables to be synchronized and related fields is named syncKeys.conf. Each line represents one table and its corresponding field, such as log (UUID), app (id.appkey), and artifact (id), representing three tables and their corresponding fields. The random number synchronization program is called by the gateway. Calling the random number synchronization program updates the data identifier pushed by the trigger to the target data table in the non-baseline database; that is, it updates the target record in the target data table, replacing the data identifier to be processed in the target record.

[0076] Continuing with the previous example, after the backends of each target subroutine have completed the data insertion operations in the test requests, the gateway, configured with a list of URLs containing insert logic with UUIDs or random value generation, can determine whether the current request includes an operation that inserts a UUID or random value as a query condition field. If so, the gateway calls the random number synchronization program to update the most recent record in other databases (non-baseline databases) with the primary key value id=1 pushed from the trigger in the baseline database. The most recent record is found based on whether it is not in the list of used IDs. Because the gateway controls the serial execution of requests for the same URL at any given time, at most one record in the same table is waiting to be updated.

[0077] In summary, by identifying the target data identifier that matches the identifier record table from the candidate data identifiers, and using the target data record corresponding to the target data identifier as the target record associated with the operation test request, the target record can be quickly found.

[0078] Furthermore, to ensure the real-time performance and accuracy of the data stored in the identifier record table, after updating the data identifier to be processed based on the data identifier and obtaining the target database (i.e., after the data identifier is synchronized to the non-baseline database), the data identifier is stored in the identifier record table. This indicates that the data identifier has been used for identifier synchronization between the baseline database and the non-baseline database, and will no longer be used subsequently. The specific implementation is as follows:

[0079] The data identifier is stored in the identifier record table, and the data identifier is used to find the data record in the non-benchmark database.

[0080] Based on this, after updating the data identifier to be processed based on the data identifier and obtaining the target database, the data identifier is stored in the identifier record table. This indicates that the data identifier has been used for identifier synchronization between the baseline database and the non-baseline database, and will no longer be used subsequently. The data identifier stored in the identifier record table can be used to find data records in the non-baseline database. When searching for the target record in the non-baseline database later, it can accurately find unused data identifiers among the candidate data identifiers.

[0081] Following the previous example, after updating the data identifier to be processed based on the data identifier and obtaining the target database, the random number synchronization program adds the current UUID value (data identifier) ​​to the used ID set (representing the record table) of the database table of the baseline database, so that when searching for the latest inserted record next time, records with UUID equal to this value will be filtered out.

[0082] In summary, storing the data identifier in the identifier record table indicates that the data identifier has been used for identifier synchronization between the baseline database and the non-baseline database. Subsequently, the data identifier will no longer be used, making it convenient to filter out the data identifier when searching for the target record in the non-baseline database.

[0083] Furthermore, after updating the benchmark and non-benchmark databases, test tasks associated with operation test requests can be executed based on the benchmark and target databases, as specifically implemented below:

[0084] Based on the operation test request, determine the data operation test task associated with the data identifier, and use the data operation test task as the test task; determine the data query instruction, data modification instruction, and / or data deletion instruction contained in the test task, and execute the data query instruction, the data modification instruction, and / or the data deletion instruction for the benchmark database and the target database.

[0085] Specifically, data manipulation test tasks are data operations scheduled after the data insertion operations corresponding to the operation test requests. The execution of these tasks depends on data identifiers in the benchmark database and updated data identifiers to be processed in the non-benchmark database. Data query, data modification, and data deletion commands are all data processing commands associated with data identifiers in the benchmark database and updated data identifiers to be processed in the non-benchmark database.

[0086] Based on this, data operation test tasks are determined based on the operation test request and associated with the data identifier. These data operation test tasks are arranged after the data insertion operation corresponding to the operation test request. The data operation test tasks perform data operations such as data querying, data modification, and data deletion on data identifiers uniformly stored in both the benchmark and non-benchmark databases. Using these data operation test tasks as test tasks, the data query instructions, data modification instructions, and / or data deletion instructions included in the test tasks are determined. These instructions are then executed on both the benchmark and target databases to complete data operations such as data querying, data modification, and data deletion based on the data identifier.

[0087] Continuing with the previous example, such as Figure 3As shown in (b), after updating the pending data identifiers in the non-baseline database, it indicates that the newly added id=1 in the MySQL database has been synchronized to the DM and OB databases. At this point, for the data insertion operation corresponding to the operation test request, the MySQL, DM, and OB databases store the same data identifier (id). When executing the data operation test task associated with the operation test request, the data operation test task can be a data operation such as querying, modifying, or deleting data for id=1. Querying the MySQL, DM, and OB databases based on the same id will yield the same data, thus realizing the data operation function test of the MySQL, DM, and OB databases.

[0088] In summary, the tests performed on the benchmark and target databases by executing data query, data modification, and / or data deletion commands ensured that data operations such as data querying, data modification, and data deletion were performed on the benchmark and target databases based on a unified data identifier. This ensured that the benchmark and target databases successfully responded to data operations based on the data identifier and guaranteed the effectiveness of the high-traffic test.

[0089] This specification provides a data processing method according to one embodiment, which, in response to an operation test request for a target program, determines a baseline database and a non-baseline database in at least two databases associated with the target program. A data identifier associated with the operation test request is determined in the baseline database, and a pending data identifier associated with the operation test request is determined in the non-baseline database. The pending data identifier is updated based on the data identifier to obtain the target database, achieving data identifier synchronization between the baseline and non-baseline databases. Executing the test task associated with the operation test request based on the baseline and target databases avoids the problem of the non-baseline database generating random pending data identifiers in response to the operation test request, leading to subsequent test task execution failures, thus improving the test success rate. Even if the execution of the operation test request results in different data identifiers being added to the baseline and non-baseline databases, subsequent test tasks executed based on the data identifiers can still be completed through data identifier synchronization, ensuring the effectiveness of traffic testing based on the operation test request and avoiding interference from random values.

[0090] The following is in conjunction with the appendix Figure 4 Taking the application of the data processing method provided in this specification in traffic surge testing as an example, the data processing method will be further explained. Among other things, Figure 4 An interactive schematic diagram of a data processing method provided in one embodiment of this specification is shown, which specifically includes the following steps.

[0091] Step S402: The browser's business operation initiates a data processing request and submits the data processing request to the multi-send gateway.

[0092] Step S404: The multi-gateway checks whether the data processing request involves data insertion operations on each database.

[0093] The multi-send gateway sends data processing requests to the backend programs for execution and waits for each backend program to complete its execution.

[0094] Step S406: The configuration file returns the check results to the multi-send gateway.

[0095] Step S408: The multi-send gateway sends id=uuid1 to the MySQL database for storage.

[0096] Step S410: The multi-send gateway sends id=uuid2 to the OB database for storage.

[0097] Step S412: The multi-send gateway sends id=uuid3 to the DM database for storage.

[0098] Step S414: The multi-gateway determines that the data processing request involves data insertion operations in each database and informs the random number synchronization program to perform data identification synchronization.

[0099] Step S416: Obtain UUID1 using a MySQL database trigger.

[0100] Step S418: The MySQL database uses id=uuid1 as a data identifier and synchronizes it to the random number synchronization program.

[0101] Step S420: The random number synchronization program updates id=uuid1 to the OB database.

[0102] Step S422: The random number synchronization program updates id=uuid1 to the DM database.

[0103] Step S424: The random number synchronization program informs the multi-sender gateway that the data identifier update is complete.

[0104] Step S426: The multi-send gateway informs the browser that the data identifier update is complete.

[0105] After the data identifier synchronization between the MySQL database, OB database, and DM database is completed, subsequent high-traffic tests such as data query, data deletion, and data modification can be performed based on id=uuid1.

[0106] In summary, the data processing method provided in this embodiment obtains the value of a unique field in a specific table of a benchmark database (MySQL database) based on a trigger, then synchronizes it to the newly generated record in the corresponding table of other databases, updating the unique field value of that record to the value (data identifier) ​​in the benchmark database. After all updates are complete or a timeout occurs, the multi-traffic gateway returns the execution result of the request with the insert operation to the browser / client, ensuring the consistent execution of subsequent requests. This achieves consistency of business IDs in multiple tested databases in a non-intrusive manner, improving the practicality of high-traffic scenarios. This method can be applied to databases such as Oracle / MySQL / PostgreSQL, and can be applied non-intrusively to applications written in C / Java / Go. It solves the problem of high-traffic testing failure caused by randomly generated values ​​(ids) in high-traffic testing scenarios.

[0107] Corresponding to the above method embodiments, this specification also provides data processing apparatus embodiments. Figure 5 A schematic diagram of the structure of a data processing apparatus according to one embodiment of this specification is shown. Figure 5 As shown, the device includes:

[0108] The first determining module 502 is configured to determine a benchmark database and a non-benchmark database in response to an operational test request for the target program in at least two databases associated with the target program;

[0109] The second determining module 504 is configured to determine the data identifier associated with the operation test request in the benchmark database and to determine the data identifier to be processed associated with the operation test request in the non-benchmark database.

[0110] The execution module 506 is configured to update the data identifier to be processed based on the data identifier, obtain the target database, and execute the test task associated with the operation test request based on the benchmark database and the target database.

[0111] In an optional embodiment, the first determining module 502 is further configured to:

[0112] Receive the operation test request for the target program, and send the operation test request to at least two target subroutines corresponding to the target program;

[0113] Upon receiving execution feedback data from the at least two target subroutines, the databases corresponding to the at least two target subroutines are designated as the at least two databases associated with the target program.

[0114] The benchmark database and the non-benchmark database are determined from the at least two databases.

[0115] In an optional embodiment, the second determining module 504 is further configured to:

[0116] The data table associated with the operation test request is determined in the benchmark database, and the trigger module configured in the data table is used to obtain the data identifier associated with the operation test request.

[0117] In an optional embodiment, the execution module 506 is further configured to:

[0118] In the non-benchmark database, determine the target data table associated with the operation test request, and in the target data table, determine the target record associated with the operation test request;

[0119] The data identifier to be processed in the target record is updated to the data identifier to obtain the target database.

[0120] In an optional embodiment, the execution module 506 is further configured to:

[0121] In the target data table, at least one candidate data record is determined, and a candidate data identifier is determined for each of the at least one candidate data record.

[0122] A target data identifier that matches the identifier record table is determined from the candidate data identifiers, and the target data record corresponding to the target data identifier is used as the target record associated with the operation test request.

[0123] In an optional embodiment, the first determining module 502 is further configured to:

[0124] Receive operation test requests for the target program;

[0125] The operation test request is subjected to operation type detection to obtain the target operation type corresponding to the operation test request;

[0126] When the target operation type is an insert operation type, a baseline database and a non-baseline database are determined from at least two databases associated with the target program.

[0127] In an optional embodiment, the execution module 506 is further configured to:

[0128] The data identifier is stored in the identifier record table, and the data identifier is used to find the data record in the non-benchmark database.

[0129] In an optional embodiment, the execution module 506 is further configured to:

[0130] Based on the operation test request, determine the data operation test task associated with the data identifier, and use the data operation test task as the test task;

[0131] The test task includes data query instructions, data modification instructions, and / or data deletion instructions, which are then executed on the benchmark database and the target database.

[0132] One embodiment of this specification provides a data processing apparatus that, in response to an operation test request for a target program, determines a baseline database and a non-baseline database in at least two databases associated with the target program. A data identifier associated with the operation test request is determined in the baseline database, and a pending data identifier associated with the operation test request is determined in the non-baseline database. The pending data identifier is updated based on the data identifier to obtain the target database, achieving data identifier synchronization between the baseline and non-baseline databases. Executing test tasks associated with the operation test request based on the baseline and target databases avoids the problem of the non-baseline database generating random pending data identifiers in response to the operation test request, leading to subsequent test task execution failures, thus improving the test success rate. Even if the execution of the operation test request results in different data identifiers being added to the baseline and non-baseline databases, subsequent test tasks executed based on the data identifiers can still be completed through data identifier synchronization, ensuring the effectiveness of traffic testing based on the operation test request and avoiding interference from random values.

[0133] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.

[0134] Figure 6 A structural block diagram of a computing device 600 according to one embodiment of this specification is shown. The components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and a database 650 is used to store data.

[0135] The computing device 600 also includes an access device 640, which enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 640 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0136] In one embodiment of this specification, the above-described components of the computing device 600 and Figure 6 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 6 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0137] The computing device 600 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 600 can also be a mobile or stationary server.

[0138] The processor 620 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0139] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the data processing method described above.

[0140] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0141] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the data processing method described above.

[0142] An embodiment of this specification also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0143] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the data processing method described above.

[0144] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0145] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0146] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0147] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0148] The preferred embodiments disclosed above are merely illustrative of this specification. Optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described in this specification. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification.

Claims

1. A data processing method, characterized by, The method comprises the following steps: In response to an operation test request for a target program, determining a reference database and a non-reference database in at least two databases associated with the target program; Determining a data identifier associated with the operation test request in the reference database, and determining a target record associated with the operation test request in the non-reference database, wherein the target record is not a to-be-processed data identifier in an identifier record table, and the identifier record table records data identifiers that have been used for database synchronization in chronological order; Updating the to-be-processed data identifier to the data identifier to obtain a target database, and executing a test task associated with the operation test request based on the reference database and the target database.

2. The data processing method according to claim 1, characterized in that, The step of determining the reference database and the non-reference database in the at least two databases associated with the target program in response to the operation test request for the target program comprises the following steps: Receiving the operation test request for the target program, and sending the operation test request to at least two target sub-programs corresponding to the target program; In the case that execution feedback data of the at least two target sub-programs is received, taking databases corresponding to the at least two target sub-programs as the at least two databases associated with the target program; Determining the reference database and the non-reference database in the at least two databases.

3. The data processing method of claim 1, wherein, The step of determining the data identifier associated with the operation test request in the reference database comprises the following steps: Determining a data table associated with the operation test request in the reference database, and obtaining the data identifier associated with the operation test request by using a trigger module configured by the data table.

4. The data processing method of claim 1, wherein, The step of updating the to-be-processed data identifier to the data identifier to obtain a target database comprises the following steps: Determining a target data table associated with the operation test request in the non-reference database, and determining a target record associated with the operation test request in the target data table; Updating the to-be-processed data identifier in the target record to the data identifier to obtain the target database.

5. The data processing method according to claim 4, characterized in that, The step of determining the target record associated with the operation test request in the target data table comprises the following steps: Determining at least one candidate data record in the target data table, and determining candidate data identifiers corresponding to the at least one candidate data record respectively; Determining a target data identifier not in an identifier record table in the candidate data identifiers, and taking a target data record corresponding to the target data identifier as the target record associated with the operation test request.

6. The data processing method of claim 1, wherein, The step of determining the reference database and the non-reference database in the at least two databases associated with the target program in response to the operation test request for the target program comprises the following steps: Receiving an operation test request for a target program; Performing operation type detection on the operation test request to obtain a target operation type corresponding to the operation test request; In the case that the target operation type is an insertion operation type, determining a reference database and a non-reference database in at least two databases associated with the target program.

7. The data processing method according to claim 5, characterized in that, The method further comprises: storing the data identifier into the identifier record table, the data identifier being used to search the data record in the non-reference database.

8. The data processing method of claim 1, wherein, The method further comprises: determining the data operation test task associated with the data identifier based on the operation test request, and taking the data operation test task as the test task; determining the data query instruction, the data modification instruction and / or the data deletion instruction contained in the test task, and executing the data query instruction, the data modification instruction and / or the data deletion instruction on the reference database and the target database.

9. A data processing apparatus, characterized by, The method further comprises: a first determining module configured to determine a reference database and a non-reference database in at least two databases associated with a target program in response to an operation test request for the target program; a second determining module configured to determine a data identifier associated with the operation test request in the reference database, and determine a to-be-processed data identifier associated with the operation test request in a target record of the operation test request in the non-reference database, the to-be-processed data identifier not being in an identifier record table, the identifier record table recording data identifiers used for synchronization of identifiers between databases in chronological order; an executing module configured to update the to-be-processed data identifier to the data identifier, obtain the target database, and execute a test task associated with the operation test request based on the reference database and the target database.

10. A computing device, comprising: The method further comprises: a memory and a processor; the memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions, the computer executable instructions being used to implement the steps of the data processing method in any one of claims 1 to 8.

11. A computer readable storage medium, characterized in that, The computer executable instructions are stored in the memory, and the computer executable instructions are used to implement the steps of the data processing method in any one of claims 1 to 8.

12. A computer program product, characterised in that, The computer program or instructions are stored in the memory, and the computer program or instructions are used to implement the steps of the data processing method in any one of claims 1 to 8.

Images