Security and service non-sensing integrated system switching method after reconstruction and security and service non-sensing integrated system switching system

Abstract

The invention relates to a security and service non-sensing integrated switching method after system reconstruction, which belongs to the field of system switching and comprises the following steps of: a, data migration; b, running accompanied by double systems, placing spy objects through a gateway interceptor, recording api calling, database change and messages in a new system consumption message queue, executing a corresponding api service method, and comparing the consistency of api response messages of the new system and the old system; c, primary data verification; d, switching between a new system and an old system; and e, performing data verification again. The method has the advantages that the spy entity object is processed through the ThreadLoal, and the problem of complexity of protocol assembly of new and old business systems is solved. And meanwhile, the quality of a new service system after reconstruction is ensured through production dual-system accompanying running and data verification. And finally, flow distribution is carried out through the gateway, and the flow is migrated from the old service system to the new service system. The new business system stably supports online business, and the old business system completes offline mission.

Description

Technical Field

[0001] The invention relates to a system post-reconstruction switching method and a system integrating security and service senselessness, and belongs to the field of system switching. Background Art

[0002] As the system technical debt accumulates, old systems will face the fate of system reconstruction. The reconstruction of high-traffic systems faces the problem of system switching. The old system executes the business logic through the API interface, which ultimately has two business impacts:

[0003] 1. The resulting business data changes are stored in the old database;

[0004] 2. The API calls the API of the peripheral system are nested inside the API, which indirectly causes the business data of the peripheral system to change;

[0005] After the new business system is restructured, the business impact is also the same as the previous two points. Due to the high traffic of the old system, downtime for upgrading will result in high business losses. Therefore, the problem of switching between the old and new business systems is caused by the lack of downtime. Currently, there is no feasible, safe, reliable and business-insensitive system switching method.

[0006] In order to solve this problem, this method is provided from the two goals of security and business awareness. Summary of the invention

[0007] In order to overcome the defects of the prior art, the present invention provides a system post-reconstruction switching method that integrates security and service awareness. The technical solution of the present invention is:

[0008] A system post-reconstruction switching method integrating security and business awareness, comprising:

[0009] a. Data migration: the original business system is used as the origin of business data, and the data from the old business system database is migrated to the new business system database;

[0010] b. The two systems run together. Spy objects are placed through the gateway interceptor to record API calls and database changes. The new system consumes messages in the message queue and executes corresponding API business methods. The consistency of API response messages between the new and old systems is compared.

[0011] c. Initial data verification: during the running-in period between the new system and the original business system, the data consistency between the new and old databases is compared to ensure the consistency of the business rule code;

[0012] d. Switch between the old and new systems, and divert API traffic to the new business system through the gateway to complete the switch between the old and new systems;

[0013] e. Verify the data again. After switching the system, repeat the initial data verification steps to ensure data consistency during the switching process.

[0014] The step a combines the binlog file of the database and the open source component canal to monitor and extract old data changes; at the same time, the extracted data is stored in the kafka queue and consumed by the data synchronization program, specifically:

[0015] 1.1Kafka queues are split according to business rules to ensure that the same business data is processed by a consumer data synchronization program to avoid the problem of sequential storage caused by concurrent consumption;

[0016] 1.2 The data synchronization program performs serial consumption processing with a single thread corresponding to a single queue. The number of queues is calculated according to the consumption speed and the expected data migration completion time. The specific formula is as follows:

[0017] The variable avg represents the average time from when the original business system database is changed to when it is finally included in the new business system data, in milliseconds;

[0018] Total data sum, indicating the total number of business data records in the original business system database;

[0019] Data increment deta indicates the average number of incremental data changes that occur per second in the original business system;

[0020] Data migration time t, which indicates the time it takes to complete the migration of the old business system database data, in milliseconds;

[0021] The number of queues count indicates the number of Kafka queues;

[0022] Within the migration time t, the total number of data changes is calculated as all = sum + deta * t; therefore t = all / count * avg; thus the final count is calculated as t / (avg * (cum + deta * t))

[0023] 1.3 During the data synchronization process, the following data is processed:

[0024] If the log is insert data, it is directly added to the corresponding business table according to the business primary key; if the log is update data, and there is a record in the new business table, it is directly updated according to the business primary key;

[0025] For log data that is updated and no records exist in the new business table, this situation can be avoided through the design of the previous two steps.

[0026] Described step b is:

[0027] 2.1 Make an API call to the original business system through the gateway, and insert the built-in spy object through the interceptor before the specific business rules of the API are executed; at the same time, considering that the API method involves nested methods, in order to reduce the complexity of object transfer, ThreadLocal technology is fully used here to store thread variables. The main protocol fields of the indirect object are as follows:

[0028] API number, used to represent API identity; business number, used to distinguish business identity; use Map to store peripheral API numbers and access input parameters; use Map to record other more business numbers; request input parameters; the API response message;

[0029] 2.2 The original system business involves the change of the original database data through the execution of business rule code; it involves the call of the peripheral system API;

[0030] 2.3 After the call to the peripheral system API is completed, the response message of the peripheral API is stored in redis, and at the same time stored in the Map peripheral API through the ThreadLocal variable;

[0031] 2.4 After the original business system API is executed, the spy object is packaged with the protocol and then stored in MQ;

[0032] 2.5 When the new business system consumes MQ, it finds the corresponding API business method according to the API number of the indirect object and executes it;

[0033] 2.6 The logical execution of the new business system API involves the change of the new database data; the call of the peripheral system API is obtained through the redis cache, rather than directly calling and executing from the peripheral system;

[0034] 2.7 After the logic execution of the new business system API is completed, the consistency of the two API messages of the new business system and the original business system is compared. If there is any inconsistency, the cause is investigated and repaired.

[0035] Described step b is:

[0036] Compare and verify the data before the day to complete the comparison and verification of data migration; for data changes that occur on the day, perform data verification and comparison on the next day;

[0037] Data verification and comparison are always postponed for one day. This way, through long-term observation, the full amount of data can be compared and verified to ensure the consistency of the implementation of the new and old business rule codes.

[0038] The steps for comparing the consistency of the new and old data are as follows:

[0039] 3.1 Define the comparison data range according to the specified data update time;

[0040] 3.2 For the circled data, read the old data in pages according to the specified PageSize size, and store the PageSize data in the blocking queue;

[0041] 3.3 Establish a thread pool of specified PoolSize size, and each thread in the thread pool compares the old and new data on the PageSize old data set;

[0042] 3.4 A single thread traverses and compares the old PageSize data. The core of the comparison is to serialize the old data into json. At the same time, the migration data of the old data is queried from the new database and serialized into json. The new and old json serializations are compared for string-level consistency.

[0043] Among them, regarding the determination of PageSize and PoolSize, the consistency of new and old data is compared with the program running time; if the expected program running time is T, then the process of determining the size of PageSize and PoolSize is as follows:

[0044] Y is the time required for a single thread to process a consistency comparison between new and old data;

[0045] S is the total number of old data records that need to be compared for consistency within the specified time range;

[0046] The data set range of the PageSize is determined according to the order of magnitude of 1000, 10000, and 100000;

[0047] The PoolSize calculation formula is SUM / (T)*PageSize*Y.

[0048] The step d is specifically as follows: through the gateway, the new API traffic is directed to the new business system; when the old business system completes the existing API traffic and no new traffic flows in, in this step, the production system's dependence on the old system is completely transferred to the new system.

[0049] A post-reconstruction switching system based on a system integrating security and business senselessness, comprising: a data migration module, used to use the original business system as the original point of business data, and migrate data from the old business system database to the new business system database;

[0050] The dual-system running module is used to insert spy objects through the gateway interceptor, record API calls and database changes, and the new system consumes messages in the message queue and executes the corresponding API business methods, and compares the consistency of the API response messages of the new and old systems;

[0051] The initial data verification module is used to compare the data consistency between the new system and the original business system during the running-in period to ensure the consistency of business rule code implementation;

[0052] The new and old system switching module is used to direct API traffic to the new business system through the gateway to complete the switch between the new and old systems;

[0053] The second data verification module is used to repeat the initial data verification steps after switching the system to ensure the consistency of data during the switching process.

[0054] The advantages of the present invention are: the spy entity object is processed by ThreadLoal to solve the complex problem of protocol assembly of the new and old business systems. At the same time, the quality of the new business system after reconstruction is ensured by running with the production dual system and data verification. Finally, the process is distributed through the gateway to complete the migration of traffic from the old business system to the new business system. The new business system stably supports the online business, and the old business system completes the offline mission. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a schematic diagram of the data migration process of the present invention.

[0056] Figure 2 It is a schematic diagram of the process of the dual-system accompanying running of the present invention.

[0057] Figure 3 It is a schematic diagram of the process of initial data verification of the present invention.

[0058] Figure 4 It is a schematic diagram of the new and old system switching process of the present invention.

[0059] Figure 5 It is a structural block diagram of the system of the present invention. DETAILED DESCRIPTION

[0060] The present invention will be further described below in conjunction with specific embodiments, and the advantages and features of the present invention will become clearer as the description proceeds. However, these embodiments are exemplary only and do not constitute any limitation to the scope of the present invention. It should be understood by those skilled in the art that the details and forms of the technical solution of the present invention may be modified or replaced without departing from the spirit and scope of the present invention, but these modifications and replacements all fall within the scope of protection of the present invention.

[0061] See also Figures 1 to 5The present invention relates to a system post-reconstruction switching method that integrates security and service awareness, comprising the following steps:

[0062] a. Data migration: the original business system is used as the origin of business data, and the data from the old business system database is migrated to the new business system database;

[0063] b. The two systems run together. Spy objects are placed through the gateway interceptor to record API calls and database changes. The new system consumes messages in the message queue and executes corresponding API business methods. The consistency of API response messages between the new and old systems is compared.

[0064] c. Initial data verification: during the running-in period between the new system and the original business system, the data consistency between the new and old databases is compared to ensure the consistency of the business rule code;

[0065] d. Switch between the old and new systems, and divert API traffic to the new business system through the gateway to complete the switch between the old and new systems;

[0066] e. Verify the data again. After switching the system, repeat the initial data verification steps to ensure data consistency during the switching process.

[0067] Based on the settings of steps a to e above, the following are achieved:

[0068] Data integrity and consistency assurance: Through data migration and two data verification steps, the integrity and consistency of data during the transition from the old system to the new system are ensured. This reduces the risk of business interruption caused by data problems.

[0069] Business continuity: The dual-system running-in phase allows the old and new systems to run in parallel, which means that the old system can continue to handle business before the new system completely takes over, thus ensuring business continuity.

[0070] Risk reduction: Before completely switching to a new system, by comparing the consistency of API response messages between the old and new systems, potential problems can be discovered and fixed in advance, thereby reducing the risk of switching.

[0071] Seamless switching: By directing API traffic to the new business system through the gateway, seamless switching can be achieved. For users, the system switching process is transparent and does not affect the user experience.

[0072] Timely discovery and repair of problems: The placement of spy objects and the consumption mechanism of message queues allow the new system to fully test and verify the business logic before actually taking over the business, ensuring the stability and reliability of the new system.

[0073] Flexibility and scalability: This method allows you to adjust strategies based on actual conditions at different stages, such as adjusting the speed of data migration, the length of time the dual systems run together, and the frequency of data verification, providing a high degree of flexibility and scalability.

[0074] The step a combines the binlog file of the database and the open source component canal to monitor and extract old data changes; at the same time, the extracted data is stored in the kafka queue and consumed by the data synchronization program, specifically:

[0075] 1.1Kafka queues are split according to business rules to ensure that the same business data is processed by a consumer data synchronization program to avoid the problem of sequential storage caused by concurrent consumption;

[0076] For example, if the business system is a customer information management system and the customer number is the customer identifier, the queues are split according to the customer number. Usually the customer number is modulo processed according to the number of queues: custNo%2; there are customer numbers 1 to 10, and the result of entering queues 0 and 1 is:

[0077] 1. Customer numbers 2, 4, 6, 8, and 10 are assigned to queue 0;

[0078] 2. Customer numbers 1, 3, 5, 7, and 9 are assigned to queue 1;

[0079] 1.2 The data synchronization program performs serial consumption processing with a single thread corresponding to a single queue. The number of queues is calculated according to the consumption speed and the expected data migration completion time. The specific formula is as follows:

[0080] The variable avg represents the average time from when the original business system database is changed to when it is finally included in the new business system data, in milliseconds;

[0081] Total data sum, indicating the total number of business data records in the original business system database;

[0082] Data increment deta indicates the average number of incremental data changes that occur per second in the original business system;

[0083] Data migration time t, which indicates the time it takes to complete the migration of the old business system database data, in milliseconds;

[0084] The number of queues count indicates the number of Kafka queues;

[0085] Within the migration time t, the total number of data changes is calculated as all = sum + deta * t; therefore t = all / count * avg; thus the final count is calculated as t / (avg * (cum + deta * t))

[0086] 1.3 During the data synchronization process, the following data is processed:

[0087] If the log is insert data, it is directly added to the corresponding business table according to the business primary key; if the log is update data, and there is a record in the new business table, it is directly updated according to the business primary key;

[0088] For log data that is updated and no records exist in the new business table, this situation can be avoided through the design of the previous two steps.

[0089] Described step b is:

[0090] 2.1 Make an API call to the original business system through the gateway, and insert the built-in spy object through the interceptor before the specific business rules of the API are executed; at the same time, considering that the API method involves nested methods, in order to reduce the complexity of object transfer, ThreadLocal technology is used here to store thread variables. The main protocol fields of the indirect object are as follows:

[0091] API number, used to represent API identity; business number, used to distinguish business identity; use Map to store peripheral API numbers and access input parameters; use Map to record other more business numbers; request input parameters; the API response message;

[0092] 2.2 The original system business involves the change of the original database data through the execution of business rule code; it involves the call of the peripheral system API;

[0093] 2.3 After the call to the peripheral system API is completed, the response message of the peripheral API is stored in redis, and at the same time stored in the Map peripheral API through the ThreadLocal variable;

[0094] 2.4 After the original business system API is executed, the spy object is packaged with the protocol and then stored in MQ;

[0095] 2.5 When the new business system consumes MQ, it finds the corresponding API business method according to the API number of the indirect object and executes it;

[0096] 2.6 The logical execution of the new business system API involves the change of the new database data; the call of the peripheral system API is obtained through the redis cache, rather than directly calling and executing from the peripheral system;

[0097] 2.7 After the logic execution of the new business system API is completed, the consistency of the two API messages of the new business system and the original business system is compared. If there is any inconsistency, the cause is investigated and repaired.

[0098] In the step b, the key to data verification is to read the new and old databases at the same time and compare the new and old data by means of a program. Since the new and old business systems are in a state of accompanying each other, there are dynamic data changes. Therefore, data verification is performed according to the change time period to compare and verify the consistency of the new and old business data. Taking the day as the unit of comparison and verification, the step b is:

[0099] Compare and verify the data before the day to complete the comparison and verification of data migration; for data changes that occur on the day, perform data verification and comparison on the next day;

[0100] Data verification and comparison are always postponed for one day. This way, through long-term observation, the full amount of data can be compared and verified to ensure the consistency of the implementation of the new and old business rule codes.

[0101] The steps for comparing the consistency of the new and old data are as follows:

[0102] 3.1 Define the comparison data range according to the specified data update time;

[0103] 3.2 For the circled data, read the old data in pages according to the specified PageSize size, and store the PageSize data in the blocking queue;

[0104] 3.3 Establish a thread pool of specified PoolSize size, and each thread in the thread pool compares the old and new data on the PageSize old data set;

[0105] 3.4 A single thread traverses and compares the old PageSize data. The core of the comparison is to serialize the old data into json. At the same time, the migration data of the old data is queried from the new database and serialized into json. The new and old json serializations are compared for string-level consistency. Regarding the determination of PageSize and PoolSize, the consistency of the new and old data is compared with the program running time. If the expected program running time is T, the process of determining the size of PageSize and PoolSize is as follows:

[0106] Y is the time required for a single thread to process a consistency comparison between new and old data;

[0107] S is the total number of old data records that need to be compared for consistency within the specified time range;

[0108] The data set range of the PageSize is determined according to the order of magnitude of 1000, 10000, and 100000;

[0109] The PoolSize calculation formula is SUM / (T)*PageSize*Y.

[0110] The step d is specifically as follows:

[0111] Through the gateway, new API traffic is directed to the new business system. When the old business system has completed the existing API traffic and no new traffic flows in, in this step, the production system's dependence on the old system is completely transferred to the new system.

[0112] The present invention also relates to a post-reconstruction switching system based on integrating security and service senselessness, comprising:

[0113] Data migration module 1 is used to migrate data from the old business system database to the new business system database, using the original business system as the original point of business data;

[0114] Dual system running module 2 is used to insert spy objects through the gateway interceptor, record API calls and database changes, the new system consumes messages in the message queue and executes the corresponding API business methods, and compares the consistency of API response messages between the new and old systems;

[0115] Initial data verification module 3 is used to compare the data consistency between the new system and the original business system during the running-in period to ensure the consistency of business rule code implementation;

[0116] The new and old system switching module 4 is used to direct the API traffic to the new business system through the gateway to complete the switching of the new and old systems;

[0117] The secondary data verification module 5 is used to repeat the initial data verification steps after the system is switched to ensure the consistency of the data during the switching process.

[0118] The modular structure of the present invention provides a safe, reliable and service-insensitive system migration and switching solution. The overall advantages of the system include:

[0119] Modular design: The system consists of different modules, each module is responsible for a specific task, which makes the system more flexible, easy to manage and maintain.

[0120] Automation of data migration: Data migration module 1 automates the data migration process from the old system to the new system, reducing manual intervention and the possibility of errors.

[0121] Business continuity guarantee: Dual system running module 2 ensures that business can proceed uninterruptedly while the new and old systems run in parallel, thereby minimizing the impact on users.

[0122] Real-time monitoring and response: Through the placement of gateway interceptors and spy objects, the system can monitor API calls and database changes in real time, and detect and resolve problems in a timely manner.

[0123] Consistency verification: The initial data verification module 3 and the secondary data verification module 5 provide double protection to ensure the consistency of data during the migration and switching process.

[0124] Seamless switching: The new and old system switching module 4 diverts API traffic through the gateway, achieving seamless switching between the new and old systems and improving user experience.

[0125] Risk management: The system design allows multiple verifications during the switching process, effectively reducing the risk of system switching.

[0126] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A system post-reconstruction switching method integrating security and service awareness, characterized in that: The following steps are involved: a. Data migration: the original business system is used as the origin of business data, and the data from the old business system database is migrated to the new business system database; b. The two systems run together. Spy objects are placed through the gateway interceptor to record API calls and database changes. The new system consumes messages in the message queue and executes corresponding API business methods. The consistency of API response messages between the new and old systems is compared. c. Initial data verification: during the running-in period between the new system and the original business system, the data consistency between the new and old databases is compared to ensure the consistency of the business rule code; d. Switch between the old and new systems, and divert API traffic to the new business system through the gateway to complete the switch between the old and new systems; e. Verify the data again. After switching the system, repeat the initial data verification steps to ensure data consistency during the switching process.

2. The method for switching after system reconstruction integrating security and business senselessness according to claim 1 is characterized in that: The step a combines the binlog file of the database and the open source component canal to monitor and extract old data changes; at the same time, the extracted data is stored in the kafka queue and consumed by the data synchronization program, specifically: 1.1Kafka queues are split according to business rules to ensure that the same business data is processed by a consumer data synchronization program to avoid the problem of sequential storage caused by concurrent consumption; 1.2 The data synchronization program performs serial consumption processing with a single thread corresponding to a single queue. The number of queues is calculated according to the consumption speed and the expected data migration completion time. The specific formula is as follows: The variable avg represents the average time from when the original business system database is changed to when it is finally included in the new business system data, in milliseconds; Total data sum, indicating the total number of business data records in the original business system database; Data increment deta indicates the average number of incremental data changes that occur per second in the original business system; Data migration time t, which indicates the time it takes to complete the migration of the old business system database data, in milliseconds; The number of queues count indicates the number of Kafka queues; Within the migration time t, the total number of data changes is calculated as all = sum + deta * t; therefore t = all / count * avg; thus the final count is calculated as t / (avg * (cum + deta * t)) 1.3 During the data synchronization process, the following data is processed: If the log is insert data, it is directly added to the corresponding business table according to the business primary key; if the log is update data, and there is a record in the new business table, it is directly updated according to the business primary key; For log data that is updated and no records exist in the new business table, this situation can be avoided through the design of the previous two steps.

3. The method for switching after system reconstruction integrating security and business senselessness according to claim 1 or 2, characterized in that said step b is: 2.1 Make an API call to the original business system through the gateway, and insert the built-in spy object through the interceptor before the specific business rules of the API are executed; at the same time, considering that the API method involves nested methods, in order to reduce the complexity of object transfer, ThreadLocal technology is fully used here to store thread variables. The main protocol fields of the indirect object are as follows: API number, used to represent API identity; business number, used to distinguish business identity; use Map to store peripheral API numbers and access input parameters; use Map to record other more business numbers; request input parameters; the API response message; 2.2 The original system business involves the change of the original database data through the execution of business rule code; it involves the call of the peripheral system API; 2.3 After the call to the peripheral system API is completed, the response message of the peripheral API is stored in redis, and at the same time stored in the Map peripheral API through the ThreadLocal variable; 2.4 After the original business system API is executed, the spy object is packaged with the protocol and then stored in MQ; 2.5 When the new business system consumes MQ, it finds the corresponding API business method according to the API number of the indirect object and executes it; 2.6 The logical execution of the new business system API involves the change of the new database data; the call of the peripheral system API is obtained through the redis cache, rather than directly calling and executing from the peripheral system; 2.7 After the logic execution of the new business system API is completed, the consistency of the two API messages of the new business system and the original business system is compared. If there is any inconsistency, the cause is investigated and repaired.

4. The method for switching after system reconstruction integrating security and business senselessness according to claim 1 or 2, characterized in that said step b is: Compare and verify the data before the day to complete the comparison and verification of data migration; for data changes that occur on the day, perform data verification and comparison on the next day; Data verification and comparison are always postponed for one day. This way, through long-term observation, the full amount of data can be compared and verified to ensure the consistency of the implementation of the new and old business rule codes. The steps for comparing the consistency of the new and old data are as follows: 3.1 Define the comparison data range according to the specified data update time; 3.2 For the circled data, read the old data in pages according to the specified PageSize size, and store the PageSize data in the blocking queue; 3.3 Establish a thread pool of specified PoolSize size, and each thread in the thread pool compares the old and new data on the PageSize old data set; 3.4 A single thread traverses and compares the old PageSize data. The core of the comparison is to serialize the old data into json. At the same time, the migration data of the old data is queried from the new database and serialized into json. The new and old json serializations are compared for string-level consistency. Among them, regarding the determination of PageSize and PoolSize, the consistency of new and old data is compared with the program running time; if the expected program running time is T, then the process of determining the size of PageSize and PoolSize is as follows: Y is the time required for a single thread to process a consistency comparison between new and old data; S is the total number of old data records that need to be compared for consistency within the specified time range; The data set range of the PageSize is determined according to the order of magnitude of 1000, 10000, and 100000; The PoolSize calculation formula is SUM / (T)*PageSize*Y.

5. The method for switching after system reconstruction integrating security and business senselessness according to claim 4 is characterized in that: The step d is specifically as follows: through the gateway, the new API traffic is directed to the new business system; when the old business system completes the existing API traffic and no new traffic flows in, in this step, the production system's dependence on the old system is completely transferred to the new system.

6. A system switching system after system reconstruction based on any one of claims 1 to 5 that integrates security and service awareness, characterized in that: include: The data migration module is used to migrate the data from the old business system database to the new business system database, using the original business system as the original point of business data; The dual-system running module is used to insert spy objects through the gateway interceptor, record API calls and database changes, and the new system consumes messages in the message queue and executes the corresponding API business methods, and compares the consistency of the API response messages of the new and old systems; The initial data verification module is used to compare the data consistency between the new system and the original business system during the running-in period to ensure the consistency of business rule code implementation; The new and old system switching module is used to direct API traffic to the new business system through the gateway to complete the switch between the new and old systems; The second data verification module is used to repeat the initial data verification steps after switching the system to ensure the consistency of data during the switching process.

Images