Data synchronization method and system, storage medium and electronic equipment

Abstract

The invention discloses a data synchronization method and system, a storage medium and electronic equipment, and relates to the field of data processing. The method comprises the steps that incremental data generated when full data of an old database is written into a new database is obtained, the new database is in a running state, and the old database is in a to-be-run state; writing the incremental data into the new data and the old database, and judging the consistency of the data in the new database and the old database; and when the consistency of the data reaches a preset standard, gradually replacing the use states of the old database and the new database. By implementing the technical scheme provided by the invention, the data consistency can be ensured under the condition of not interrupting business operation in the database migration process.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a data synchronization method, system, storage medium and electronic device. Background Art

[0002] In modern information systems, database migration is a common and necessary operation. With the changes in business needs and the advancement of technology, enterprises often need to migrate data from one database system to another. However, traditional database migration methods often face severe challenges.

[0003] Common database migration methods usually require suspending business operations and resuming services after completing data migration. This method will cause service interruptions and have a significant impact on the normal operation of the enterprise. Especially for those businesses that need to run 24 / 7, any service interruption may cause huge economic losses and degrade user experience.

[0004] Therefore, during the database migration process, how to ensure data consistency without interrupting business operations has become an urgent problem to be solved. Summary of the invention

[0005] The present application provides a data synchronization method, system, storage medium and electronic device, which can ensure data consistency without interrupting business operations during database migration.

[0006] In a first aspect of the present application, the present application provides a data synchronization method, comprising: Acquire incremental data generated when writing full data of an old database into a new database, wherein the new database is in a running state and the old database is in a waiting state; Writing the incremental data to the new database and the old database, and determining the consistency of the data in the new database and the old database; When the consistency of the data reaches a preset standard, the use status of the old database and the new database are gradually replaced.

[0007] In a second aspect of the present application, a data synchronization system is provided, comprising: An incremental data acquisition module, used to acquire incremental data generated when writing the full amount of data of the old database into the new database, wherein the new database is in a running state and the old database is in a waiting state; An incremental data synchronization module, used to write the incremental data into the new database and the old database, and determine the consistency of the data in the new database and the old database; The old and new database alternation module is used to gradually replace the use status of the old database and the new database when the consistency of the data reaches a preset standard.

[0008] In a third aspect of the present application, a computer storage medium is provided, wherein the computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the above method steps.

[0009] In a fourth aspect of the present application, an electronic device is provided, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the above-mentioned method steps.

[0010] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: By obtaining the incremental data generated when the full data of the old database is written into the new database, and writing these incremental data into both the old and new databases at the same time, real-time data synchronization is achieved during the data migration process. This method not only ensures that the new database can obtain the latest data in a timely manner, but also ensures the data integrity of the old database. By judging the consistency of data in the old and new databases, the quality of data synchronization can be effectively monitored, and potential data inconsistency problems can be discovered and resolved in a timely manner. When the data consistency reaches the preset standard, the usage status of the old and new databases is gradually replaced to achieve smooth database switching. This gradual switching strategy reduces the impact on business operations and effectively reduces the risks in the data migration process. Compared with the existing technology, it successfully solves the problem of how to ensure data consistency during database migration without interrupting business operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a flowchart of a data synchronization method provided in an embodiment of the present application; Figure 2 It is a structural diagram of a data synchronization system provided in an embodiment of the present application; Figure 3 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application.

[0012] Description of reference numerals: 300, electronic device; 301, processor; 302, communication bus; 303, user interface; 304, network interface; 305, memory. DETAILED DESCRIPTION

[0013] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0014] In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.

[0015] In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0016] Please refer to Figure 1 , Figure 1 This is a flow chart of a data synchronization method provided by an embodiment of the present application. The method can be implemented by a computer program, can be implemented by a single-chip microcomputer, or can be run on a data synchronization system. The computer program can be integrated into an application or run as an independent tool application. Specifically, the data synchronization method can include the following steps: S101: Acquire incremental data generated when writing full data of an old database into a new database, wherein the new database is in a running state and the old database is in a waiting state.

[0017] Among them, the old database refers to the database system that is currently in use and needs to be migrated or replaced. The old database can be understood as the main data source for storing and managing existing business data. It is usually a mature and stable database system that may face performance bottlenecks, scalability limitations, or outdated technology. In the embodiment of the present application, the old database can be a relational database such as MySQL, Oracle, or SQL Server, or a non-relational database such as MongoDB, etc. The specific type depends on the actual business scenario and migration requirements.

[0018] The old database is used to continue to support existing business operations in the initial stage of the data migration process to ensure that business continuity is not affected. It is the source of data migration and contains the full amount of historical data that needs to be transferred to the new database as well as the incremental data generated during the migration process. During the migration process, the old database plays the role of a data provider and also serves as a benchmark for data consistency verification. As the migration process progresses, the role of the old database will gradually change from the primary data source to a backup or historical data storage, and may eventually be completely replaced or retired.

[0019] Correspondingly, the new database refers to a target database system used to replace or upgrade the old database. The new database can be understood as a data management system that is more advanced, more efficient, or more suitable for current business needs, usually with better performance, stronger scalability, or richer functional characteristics. In an embodiment of the present application, the new database can be a database system of the same type as the old database but with a newer version, or it can be a completely different type of database system, such as migrating from a relational database to a non-relational database, or from one relational database to another relational database.

[0020] Furthermore, the new database is used to receive and store the full data and incremental data migrated from the old database, and gradually take over business operations during the migration process. It is the target end of data migration and needs to be able to accurately and completely store and manage all migrated data, and ensure data consistency and integrity. In the early stages of migration, the new database mainly serves as a data receiver. As the migration process progresses, it will gradually take on the task of reading and writing data, and eventually completely replace the old database as the main data storage and management system. The design and configuration of the new database needs to fully consider the needs of future business development to ensure that it can better support business operations and development after the migration is completed.

[0021] Exemplarily, the old and new databases may be database systems of the same type but different versions, or they may be completely different types of database systems. For example, the old database may be a traditional relational database, while the new database is a distributed database system that is more suitable for processing large-scale data. It is precisely because of these differences that may exist between the old and new databases that appropriate tools and methods need to be used when synchronizing data.

[0022] Taking MySQL database migration as an example, you can choose to use Canal for data synchronization. Canal is an open source database synchronization tool from Alibaba. It can parse MySQL binlog logs, convert data changes into a specific data format, and then transmit them to the target system. Canal works by simulating the interaction protocol of MySQL slave, disguising itself as a MySQL slave, and sending a dump protocol to the MySQL master. After receiving the dump request, the MySQL master starts pushing the binary log to Canal, and Canal parses the binary log object.

[0023] If you are migrating from SQL Server, you can synchronize data by listening to CDC (Change Data Capture). CDC is a built-in data synchronization mechanism in SQL Server that can capture insert, update, and delete operations on database tables and store these changes in a format that can be used for extraction, facilitating subsequent data synchronization operations.

[0024] In actual applications, the specific synchronization tool or method to be selected needs to be determined according to the type of source database. For example, for Oracle database, Oracle GoldenGate or Logminer can be used; for PostgreSQL, tools such as logical decoding or pglogical can be used. Each database has its own specific log mechanism and synchronization tool. Choosing the right tool can ensure the efficiency and accuracy of data synchronization.

[0025] When implementing it, you first need to select appropriate data synchronization tools, such as Canal (for MySQL) or CDC (for SQL Server), which can capture data changes in the old database in real time. After the full data migration begins, these tools will continue to monitor the logs of the old database (such as MySQL's binlog or SQL Server's transaction log) and convert the captured incremental data into a processable format. This incremental data includes all data operations that occur during the full data migration process, such as inserts, updates, and deletes.

[0026] By capturing and processing this incremental data in real time, you can ensure that the data in the new database is synchronized with the old database after the full data migration is completed. The advantage of the above method is that it allows the business system to operate normally during the data migration process, because the old database can still handle normal business operations, and the data changes generated by these operations will be captured in real time and synchronized to the new database. In addition, by continuously synchronizing incremental data, the amount of data that needs to be synchronized after the full data migration is completed can be significantly reduced, thereby shortening the downtime when the system is switched.

[0027] Based on the above embodiment, as an optional embodiment, in S101: the step of writing the incremental data into the new data and the old database may further include the following steps: The incremental data is written to the old database, and after the incremental data is written to the old database, the incremental data is written to the new database.

[0028] Specifically, when writing incremental data to the old and new databases, it is important to adopt the strategy of "writing to the old database first, then writing to the new database". The core of this strategy is to ensure data consistency and business continuity, while providing an effective response mechanism for possible abnormal situations.

[0029] Specifically, the system first writes the incremental data to the old database. This step is crucial because the old database is still the main data source for the current business system and its data needs to be kept real-time and complete. For example, in an e-commerce platform, newly generated order information needs to be written to the old database first to ensure that core business processes such as order processing and inventory management can run normally. Only after confirming that the incremental data has been successfully written to the old database will the system write the same incremental data to the new database. This sequential operation method provides a layer of protection for the data synchronization process. Even if an exception occurs when writing data to the new database, it will not affect the running business system.

[0030] Furthermore, during the process of writing to the new database, data format conversion or adaptation may be required to meet the structural requirements of the new database. For example, if the old database uses MySQL and the new database uses MongoDB, the relational data structure needs to be converted to a document data structure during the writing process. This strategy of "writing to the old database first, then writing to the new database" not only ensures data consistency, but also provides a flexible rollback mechanism. If an error occurs during the writing process to the new database, the system can immediately terminate the operation without affecting the data integrity in the old database.

[0031] In addition, another advantage of the above method is that it allows real-time data verification during the data synchronization process. By comparing the data written to the old database and the new database, any inconsistencies can be discovered and corrected in a timely manner, thereby ensuring the accuracy of the entire data migration process. In addition, this writing order also provides a basis for subsequent grayscale switching, because it ensures that the data in the new database is always an accurate copy of the old database, so that when gradually switching business traffic to the new database, the risk of data inconsistency can be minimized.

[0032] S102: writing the incremental data into the new database and the old database, and determining the consistency of the data in the new database and the old database.

[0033] Data consistency refers to the data stored in the new and old databases remaining the same or equivalent in terms of content, structure, and relationships. Data consistency can be understood as the business-wise matching of data in the new database with the corresponding data in the old database at any given point in time, including but not limited to data values, formats, associations, and integrity constraints.

[0034] Specifically, during the data migration process, it is necessary to ensure that the data of the new and old databases remain synchronized, so the accuracy and reliability of the data migration can be verified through consistency checks. The process can also specifically include the following steps: S201: Determine the consistency of the total amount of data tables in the new database and the old database.

[0035] During the data migration process, determining the consistency of the total number of data tables in the new database and the old database is a key preliminary verification step. Specifically, the system will compare the total number of records for each corresponding table in the new and old databases.

[0036] For example, for an e-commerce platform, the system may compare the total number of records in key business tables such as user tables, order tables, and product tables. This comparison is usually achieved by executing a simple SQL query. For large databases, in order to improve efficiency, approximate counting methods can be used, such as using database statistics or sampling techniques. When performing the comparison, the system sets an acceptable error range, which is usually a very small percentage, such as 0.1%. If the difference in the number of records in a table in the old and new databases exceeds this threshold, the system will trigger an alarm. For example, if the user table in the old database has 1,000,000 records and the new database has only 998,000, this is obviously beyond the acceptable error range, indicating that there is data loss or synchronization failure. Once the total amount is found to be inconsistent, the system will immediately record the inconsistent table name and the specific number of differences, and push this information to the data migration team.

[0037] The above-mentioned rapid feedback mechanism enables the team to promptly discover and resolve large-scale data synchronization issues, such as bulk data loss caused by network interruptions, or record duplication caused by database index errors. Another important function of the total consistency check is to provide guidance for subsequent detailed data verification. If the total amount of a table is significantly different, then this table will be prioritized for a more fine-grained data consistency check.

[0038] In addition, by performing a total consistency check regularly, the system can monitor the progress and quality of the entire data migration process. For example, a total consistency check can be performed every hour to generate a trend report to help the team understand whether the overall status of data synchronization is stable.

[0039] S202: Determine the consistency of the core data of the new database and the old database.

[0040] Among them, core data refers to key information that is crucial to business operations and decision-making. These data are usually high-value, highly sensitive, and frequently accessed. In the embodiments of this application, core data can be understood as a collection of data that has a direct impact on system functions and business processes, including but not limited to key business data such as user basic information, transaction records, account balances, and commodity inventory.

[0041] Exemplarily, the core data may include at least one of a program sampled randomly, data sampled at a fixed time, and a key field.

[0042] Among them, the program obtained by random sampling refers to an automated data selection mechanism for extracting representative samples from large-scale data sets. In the embodiment of the present application, this program can be understood as an intelligent data screening system, which can randomly extract a certain proportion or a fixed number of data records from the old database according to preset rules and parameters, and perform consistency checks as an important part of the core data. The core of this program lies in its randomness and configurability, which ensures that the probability of each record being selected is equal, while allowing the sampling strategy to be adjusted according to actual needs.

[0043] For example, you can set it to randomly extract 1% of the records from the user table each time, or select the 1,000 latest transaction data from the transaction table every hour. The design of the program takes into account the characteristics of different data tables, and different sampling methods can be used for different types of data, such as stratified sampling or systematic sampling. In addition, the program also has the ability to execute periodically, automatically sampling at preset time intervals, and recording the results of each sampling in detail, including sampling time, range, quantity and other information, to provide a basis for subsequent analysis.

[0044] Among them, the data obtained by sampling at a fixed time refers to the data set obtained by systematically sampling the database at a pre-set specific time point. In the embodiment of the present application, this method can be understood as a timed snapshot mechanism, which can perform comprehensive or large-scale data collection on the entire database or specific key tables at a fixed time every day. This sampling method is different from random sampling, and it focuses more on capturing the complete state of the data at a specific time point.

[0045] For example, the system may fully sample the user account balance table at dawn every day, or fully record all new orders in the past 24 hours. The importance of this fixed-time sampling is that it can provide a continuous view of data changes over time, which helps to discover periodic data change patterns or anomalies. The sampling process is usually performed by automated scripts to ensure the accuracy and consistency of the sampling. The collected data will be temporarily stored or marked for comparison with the corresponding data in the new database.

[0046] Among them, key fields refer to specific fields in the database that are important to business logic, system functions or data integrity. In the embodiments of the present application, key fields can be understood as data attributes that directly affect core business processes, decision-making or system operation. These fields usually contain the most critical and sensitive information, which is essential to maintaining the normal operation of the system and the continuity of business.

[0047] For example, in an order system of an e-commerce platform, order number, user ID, product SKU, order status, payment amount, etc. can all be considered key fields. In a user management system, username, password hash value, account status, registration time, etc. may be defined as key fields. The selection of these fields is usually based on their importance in the business process, update frequency, and impact on the overall function of the system. The definition of key fields is not static, but can be dynamically adjusted according to business needs and system evolution.

[0048] Based on the above embodiment, as an optional implementation, before S102: writing the incremental data to the new database and the old database, the following steps may be further included: stopping writing the global variables in the old database to the new database.

[0049] During the data migration process, stopping writing global variables from the old database to the new database is a key transition step, which marks the change of data synchronization strategy from full replication to incremental update. This step is crucial to ensure data consistency and smooth migration process.

[0050] Specifically, global variables usually include system-level configuration parameters, statistical counters, or global status information, which are copied to the new database during the initial full data replication phase. However, as the migration process progresses, continuously replicating these global variables may cause unnecessary data conflicts or performance overhead. Therefore, before starting to process incremental data, the system needs to pause the synchronization of global variables. This operation is usually achieved by configuring the synchronization tool to specify specific global variable tables or system tables to be ignored.

[0051] At the same time, you may need to modify the application configuration to ensure that it no longer attempts to update these global variables in the new database. This step requires precise timing control and is usually performed during business off-peak hours to minimize the impact on system operations. After stopping global variable synchronization, the system immediately switches to incremental data processing mode.

[0052] Before the incremental data is written to the new data and the old database, stop writing the global variables in the old database to the new database. This allows the system to focus on incremental updates of business data, improving synchronization efficiency. Secondly, it reduces the risk of data inconsistency, because global variables usually have specific update logic, which may be different from the synchronization mechanism of ordinary business data. By stopping its synchronization, potential conflicts are avoided.

[0053] S103: When the consistency of the data reaches a preset standard, gradually replace the use status of the old database and the new database.

[0054] After the data consistency reaches the preset standard, the system begins to gradually replace the database usage status. This step can adopt a grayscale release strategy to achieve a smooth transition from the old database to the new database by gradually adjusting the allocation of database read and write operations.

[0055] Based on the above embodiment, as an optional embodiment, in S103: the step of gradually replacing the use status of the old database and the new database may further include the following steps: S301: Read the first data of the first ratio in the new database, stop reading the first data in the old database, and increase the first ratio in sequence.

[0056] The first ratio refers to the degree or ratio of the read operation transferred from the old database to the new database during the database migration process. In the embodiment of the present application, the first ratio can be understood as a dynamically adjusted parameter for controlling and quantifying the allocation ratio of read requests from the old database to the new database. Specifically, the first ratio can be expressed as a value or percentage between 0 and 1, where 0 means that all read operations are still being performed in the old database, and 1 means that all read operations have been completely migrated to the new database.

[0057] The first ratio is used to achieve a gradual migration of read operations, allowing the system to transfer database load from the old system to the new system in a controllable manner. By adjusting the first ratio, the speed and scope of the migration can be precisely controlled, thereby minimizing the impact of the migration process on normal business operations while ensuring system stability and performance. For example, when the first ratio is set to 0.3, it means that 30% of read requests will be routed to the new database, while 70% of requests are still handled by the old database.

[0058] During the database migration process, adopting a strategy of gradually switching read operations is key to ensuring system stability and data consistency. The system continuously monitors the performance indicators of the new database, such as response time, query efficiency, and data consistency. If these indicators remain within the expected range, the system gradually increases the first ratio. This growth process may be linear, such as increasing by 10% per day, or it may be exponential, depending on the stability of the system and business needs. After each increase in the first ratio, the system will observe for a period of time to ensure that the increased load does not cause performance problems or data inconsistencies in the new database.

[0059] S302: when the first ratio is 1, stop writing the new data of the second ratio into the old database, write the new data of the second ratio into the new database, and increase the second ratio to 1 in sequence.

[0060] During the database migration process, when the read operation is completely switched to the new database (that is, the first ratio reaches 1), the system starts to perform the migration of the write operation. The second ratio refers to the ratio of data written to the new database. It is a dynamically adjusted parameter used to control the allocation of write requests from the old database to the new database.

[0061] During the specific implementation, the system first sets an initial second ratio, such as 20%. This means that 20% of new data write requests will be routed to the new database, and this part of the data will be stopped from being written to the old database. The second ratio can be defined based on a variety of factors, such as traffic percentage, data type, or business scenario. For example, the system may choose to migrate low-priority log data to the new database first, or allocate write requests according to the timestamp of the data. This flexible definition allows the system to design the optimal migration strategy based on specific business needs and system architecture.

[0062] After the initial setup, the system continuously monitors the performance metrics of the new database, including write latency, transaction speed, and data consistency. If these metrics remain within acceptable ranges, the system gradually increases the second scale. The growth process may be linear or exponential, depending on the stability of the system and business needs. After each increase in the second scale, the system will conduct an observation period to ensure that the increased write load does not cause performance issues or data inconsistencies in the new database.

[0063] Based on the above embodiment, as an optional embodiment, the first ratio and the second ratio are one of traffic percentage, user group ratio and user registration ratio.

[0064] The traffic percentage refers to a metric for allocating and controlling database access based on the total system request volume during database migration. In the embodiment of the present application, the traffic percentage can be understood as a dynamically adjusted parameter for the proportion of data requests transferred from the old database to the new database. Specifically, the traffic percentage is expressed as a value between 0 and 100, where 0 means that all data requests are still processed by the old database, and 100 means that all requests have been completely migrated to the new database.

[0065] Traffic percentages are used to achieve gradual migration of database access requests, enabling the system to transfer database load from the old system to the new system in a controlled manner. By adjusting the traffic percentage, the system can precisely control the speed and scope of the migration, thereby ensuring system stability and performance while reducing the impact of the migration process on normal business operations.

[0066] Among them, the user group ratio refers to a metric for allocating and controlling database access based on a specific user group during the database migration process. In the embodiment of the present application, the user group ratio can be understood as a dynamically adjusted parameter, which is used to gradually transfer the proportion of requests from a specific user group from the old database to the new database. Specifically, the user group ratio is expressed as a value between 0 and 100, where 0 means that all data requests of the user group are still processed by the old database, and 100 means that all requests of the group have been completely migrated to the new database.

[0067] User group ratios are used to implement gradual database migration based on user characteristics, allowing the system to transfer database load from the old system to the new system in a more refined and targeted manner. By adjusting the user group ratios, the system can formulate migration strategies based on the characteristics and business importance of different user groups, thereby reducing the impact of the migration process on key business operations while ensuring system stability and performance. For example, the system can divide user groups based on user geographic location, activity, membership level and other characteristics, and gradually increase the migration ratio of each group.

[0068] Among them, the user registration ratio refers to a metric for allocating and controlling database access based on the user's registration time during the database migration process. In the embodiment of the present application, the user registration ratio can be understood as a dynamically adjusted parameter, which is used to gradually transfer the proportion of user requests in different registration time periods from the old database to the new database. Specifically, the user registration ratio is expressed as a value between 0 and 100, where 0 means that user data requests for all registration time periods are still processed by the old database, and 100 means that user requests for all registration time periods have been completely migrated to the new database.

[0069] The user registration ratio is used to implement gradual database migration based on user registration time, enabling the system to transfer database load from the old system to the new system in a time series manner. By adjusting the user registration ratio, the system can formulate a migration strategy based on the user's registration time, thereby reducing the impact of the migration process on long-term users and core businesses while ensuring system stability and performance. For example, the system can first migrate the data of recently registered users, and then gradually expand to the user group that registered earlier.

[0070] Based on the above embodiment, as an optional embodiment, the embodiment of the present application also sets relevant switching switches, including: write-only old library switch, write-only new library switch, synchronous double write switch, read-only old library switch and read-only new library switch.

[0071] Among them, the switch refers to a series of configurable parameters used to control data read and write behavior and migration process during database migration. In the embodiment of the present application, the switch can be understood as a set of Boolean configuration items or flags, which are used to dynamically adjust the system's access mode and data flow to the new and old databases. These switches are mainly used to achieve refined control, flexible switching and rapid response to abnormal situations during database migration.

[0072] Furthermore, the read-only old database switch and the write-only old database switch are key configuration items for finely controlling the database migration process in the embodiment of the present application. The read-only old database switch refers to a flag that controls whether the system only reads data from the old database, while the write-only old database switch is used to control whether the system only writes data to the old database. In the embodiment of the present application, these two switches can be understood as Boolean configuration parameters, which are used to dynamically adjust the system's read and write behavior of the old database.

[0073] When the read-only old database switch is activated, all data read requests will be routed to the old database, ensuring that the system obtains information from a reliable source during the initial migration or data verification phase. Similarly, when the write-only old database switch is activated, all data write operations will be performed on the old database, which is particularly important at the beginning of the migration or when a temporary rollback is required, and can maintain the normal operation of the business system and the integrity of the data.

[0074] Correspondingly, the read-only new database switch and the write-only new database switch are important configuration items for finely controlling the database migration process in the embodiment of the present application. The read-only new database switch refers to a flag that controls whether the system only reads data from the new database, while the write-only new database switch is used to control whether the system only writes data to the new database. In the embodiment of the present application, these two switches can be understood as Boolean configuration parameters, which are used to dynamically adjust the system's read and write behavior for the new database.

[0075] When the read-only new database switch is activated, all data read requests are routed to the new database. This switch is usually used in the later stages of data migration, when the data in the new database has been consistent with the old database and the system is ready to start shifting the read load to the new database. Activating this switch can help the system gradually adapt to the performance characteristics of the new database, while verifying that the new database can correctly handle all read requests.

[0076] When the "Write only to new database" switch is activated, all data write operations will be performed on the new database. This switch is usually used when data migration is nearly complete, indicating that the system is ready to transfer all write operations to the new database. Activating this switch marks the final stage of data migration, and the system begins to rely entirely on the new database for data storage and updates.

[0077] Furthermore, the synchronous dual write switch refers to a configuration item used to control whether the system writes data to both the old and new databases during the database migration process. In the embodiment of the present application, the synchronous dual write switch can be understood as a Boolean flag or configuration parameter, which is used to dynamically adjust the data writing behavior of the system so that it can write the same data to the new database while writing to the old database.

[0078] When the synchronous dual write switch is activated, the system will adopt the strategy of "writing to the old database first, then writing to the new database". Specifically, when there is a data write request, the system will first write the data to the old database, and after ensuring that the write is successful, the same data will be written to the new database. This mechanism ensures that the old and new databases can maintain data consistency during the migration process, and also provides an effective response strategy for possible abnormal situations.

[0079] The synchronous dual-write switch is mainly used in the transition phase of database migration. In this phase, the system needs to ensure that the new database can correctly receive and store all new data, while also keeping the data in the old database updated so that it can be quickly rolled back when needed. By activating the synchronous dual-write switch, the data write load can be gradually transferred to the new database without interrupting existing business.

[0080] The design of this switch also facilitates data consistency verification. Since the old and new databases receive the same write operations at the same time, the data of the two databases can be compared regularly to ensure that no data loss or inconsistency occurs during the migration process. If any differences are found, they can be corrected in time to ensure the integrity and accuracy of the data.

[0081] For example, at the beginning of the migration process, the system first activates the read-only old database switch and the write-only old database switch to ensure that all read and write operations are performed on the old MySQL database, and starts the initial replication of the full data. When the full data replication is completed, the team turns off the write-only old database switch and activates the synchronous dual-write switch. At this point, all write operations will occur on MySQL and PostgreSQL at the same time, while read operations still obtain data from MySQL. This stage allows the new PostgreSQL database to gradually catch up and keep pace with MySQL without affecting existing business.

[0082] After confirming that the data in PostgreSQL is consistent with MySQL, the team began to gradually switch read operations. They gradually turned off the read-only switch for the old database and activated the read-only switch for the new database. This process can be gradual, for example, switching non-critical read operations (such as product category browsing) to PostgreSQL first, while keeping critical operations (such as order query) on MySQL. By continuously monitoring system performance and data consistency, the team can gradually increase the read load of PostgreSQL.

[0083] After the read operations were completely switched to PostgreSQL and the system had been running stably for a while, the team began to prepare to switch to write operations. They first turned off the synchronous dual write switch, and then activated the write-only new database switch. This process also adopted a gradual strategy, first switching non-critical write operations such as log writing to PostgreSQL, and then gradually switching core business write operations such as order creation and inventory updates. Finally, when all read and write operations were successfully switched to PostgreSQL, the read-only new database switch and the write-only new database switch were both activated, and the other switches were turned off.

[0084] Throughout the migration process, if problems are found at any stage (such as performance degradation, data inconsistency, etc.), the team can roll back the operation by quickly adjusting these switches. For example, if a problem is found during the write operation switching phase, the switch to write only to the new database can be immediately turned off, and the synchronous dual-write switch or the switch to write only to the old database can be reactivated to quickly roll the system back to a stable state. This flexibility provides the system with a powerful risk management tool, ensuring that it can respond quickly when encountering unexpected situations and minimize the impact on the business.

[0085] Reference Figure 2 , the present application also provides a data synchronization system, including: An incremental data acquisition module, used to acquire incremental data generated when writing the full amount of data of the old database into the new database, wherein the new database is in a running state and the old database is in a waiting state; An incremental data synchronization module, used to write the incremental data into the new database and the old database, and determine the consistency of the data in the new database and the old database; The old and new database alternation module is used to gradually replace the use status of the old database and the new database when the consistency of the data reaches a preset standard.

[0086] Based on the above embodiment, as an optional embodiment, the incremental data synchronization module is further used to write the incremental data into the old database; after the incremental data is written into the old database, the incremental data is written into the new database.

[0087] Based on the above embodiment, as an optional embodiment, the incremental data synchronization module is also used to determine the consistency of the total amount of data tables of the new database and the old database; and / or determine the consistency of the core data of the new database and the old database.

[0088] Based on the above embodiment, as an optional embodiment, the incremental data synchronization module is further used to stop writing the global variables in the old database into the new database.

[0089] On the basis of the above embodiment, as an optional embodiment, the new and old database alternation module is also used to read the first data of the first proportion in the new database, stop reading the first data in the old database, and increase the first proportion in sequence; when the first proportion is 1, stop writing the new data of the second proportion to the old database, write the new data of the second proportion to the new database, and increase the second proportion to 1 in sequence.

[0090] It should be noted that: when the device provided in the above embodiment realizes its function, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0091] The embodiment of the present application also provides a computer storage medium, which can store multiple instructions. The instructions are suitable for being loaded by a processor and executing the data synchronization method of the above embodiment. The specific execution process can refer to the specific description of the embodiment shown, which will not be repeated here.

[0092] The present application also discloses an electronic device. Figure 3 , Figure 3 The electronic device 300 may include: at least one processor 301 , at least one network interface 304 , a user interface 303 , a memory 305 , and at least one communication bus 302 .

[0093] The communication bus 302 is used to realize the connection and communication between these components.

[0094] The user interface 303 may include a display interface and a camera interface. Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.

[0095] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0096] Among them, the processor 301 may include one or more processing cores. The processor 301 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 305, and calling data stored in the memory 305. Optionally, the processor 301 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 301 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface diagrams and applications, etc.; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 301, and it can be implemented separately through a chip.

[0097] Among them, the memory 305 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 305 includes a non-transitory computer-readable storage medium. The memory 305 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 305 may optionally also be at least one storage device located away from the aforementioned processor 301. Refer to Figure 3 , the memory 305 as a computer storage medium may include an operating system, a network communication module, a user interface module and an application program of a data synchronization method.

[0098] exist Figure 3In the electronic device 300 shown, the user interface 303 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 301 can be used to call an application program storing a data synchronization method in the memory 305, and when executed by one or more processors 301, the electronic device 300 executes one or more of the methods described in the above embodiments. It should be noted that for the aforementioned method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0099] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0100] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0101] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0102] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0103] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, mobile hard drives, magnetic disks or optical disks.

[0104] The above is only an exemplary embodiment of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification and the truth of practice, those skilled in the art will easily think of other embodiments of the present disclosure.

[0105] This application is intended to cover any variation, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the art not described in the present disclosure. The description and examples are to be regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A data synchronization method, characterized in that: include: Acquire incremental data generated when writing full data of an old database into a new database, wherein the new database is in a running state and the old database is in a waiting state; Writing the incremental data to the new database and the old database, and determining the consistency of the data in the new database and the old database; When the consistency of the data reaches a preset standard, the use status of the old database and the new database are gradually replaced.

2. The data synchronization method according to claim 1, characterized in that: Writing the incremental data into the new data and the old database includes: Writing the incremental data into the old database; After the incremental data is written into the old database, the incremental data is written into the new database.

3. The data synchronization method according to claim 1, characterized in that: The determining the consistency of data in the new database and the old database includes: Determining the consistency of the total amount of data tables of the new database and the old database; and / or, The consistency of the core data of the new database and the old database is determined.

4. The data synchronization method according to claim 3, characterized in that: The core data includes at least one of a program obtained by random sampling, data obtained by fixed-time sampling, and a key field.

5. The data synchronization method according to claim 1, characterized in that: Before writing the incremental data into the new data and the old database, the method further includes: Stop writing global variables in the old database to the new database.

6. The data synchronization method according to claim 1, characterized in that: The gradually replacing the usage status of the old database and the new database includes: Read the first data of the first proportion in the new database, stop reading the first data in the old database, and increase the first proportion in sequence; When the first ratio is 1, stop writing the new data of the second ratio into the old database, write the new data of the second ratio into the new database, and increase the second ratio to 1 in sequence.

7. The data synchronization method according to claim 6, characterized in that: The first ratio and the second ratio are one of a traffic percentage, a user group percentage, and a user registration percentage.

8. A data synchronization system, characterized in that: include: An incremental data acquisition module, used to acquire incremental data generated when writing the full amount of data of the old database into the new database, wherein the new database is in a running state and the old database is in a waiting state; An incremental data synchronization module, used to write the incremental data into the new database and the old database, and determine the consistency of the data in the new database and the old database; The old and new database alternation module is used to gradually replace the use status of the old database and the new database when the consistency of the data reaches a preset standard.

9. An electronic device, characterized in that: It includes a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes the method as described in any one of claims 1-7.

10. A computer storage medium, characterized in that: The computer storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 7 is executed.

Images