Dynamic data snapshot efficient construction method based on dual-mechanism collaboration

Abstract

The invention relates to the technical field of data snapshots, and discloses a dynamic data snapshot efficient construction method based on dual-mechanism collaboration, which comprises the following steps: creating master-slave database instances, configuring slave database instances into slave nodes of the master database instances, and establishing a master-slave replication relationship; the master database instance comprises a source data table, and the slave database instance comprises a snapshot data table corresponding to the source data table; at a preset snapshot time point, pausing the master-slave replication process of the slave database instance, so that the snapshot data table stops synchronizing the data of the source data table, and a snapshot data table at the preset snapshot time point is obtained; renaming the snapshot data table to obtain a snapshot supply table; in the slave database instance, creating a backup data table with the same data structure as the snapshot supply table; copying total data of the snapshot supply table into a backup data table in a physical file copying mode to serve as a new snapshot data table, and reconstructing a master-slave synchronous link; and recovering the master-slave replication process.

Description

Technical Field

[0001] This specification relates to the field of data snapshot technology, and in particular to an efficient method for constructing dynamic data snapshots based on a dual-mechanism collaborative approach. Background Technology

[0002] In the daily operation and maintenance and data analysis of large-scale business systems, there is a common need to obtain a snapshot of the full data at a precise point in time to support subsequent data verification, report generation, and external data exchange. Taking database systems in the financial and e-commerce fields as examples, the data scale of their core business tables often reaches tens of millions or even hundreds of millions.

[0003] Existing technical solutions have several shortcomings: First, the industry commonly uses SQL-based replication methods, i.e., full data replication via SELECT...INSERT statements or MySQL's dump export tool. This method requires rereading, serializing, and writing all data from the source table to a new table. When processing tens or even hundreds of millions of data points, this often takes several hours, and the generation time increases linearly with the data volume, failing to meet the business's timeliness requirements for snapshot data. Second, the above logical replication process requires the data to be static to ensure snapshot consistency, thus causing prolonged blocking of real-time transactions in the source table, severely impacting the continuity of online business. Third, while existing database snapshot solutions have introduced master-slave replication mechanisms as the basic data synchronization architecture and utilize the transient consistency of slave data to obtain snapshots, their persistent backup of snapshot data still largely relies on logical export or application-layer data replication technologies, resulting in significant bottlenecks in backup efficiency and making it difficult to achieve rapid generation and efficient storage management of historical snapshots.

[0004] Therefore, there is an urgent need for an efficient method for constructing dynamic data snapshots based on dual-mechanism collaboration to solve the above problems. Summary of the Invention

[0005] In view of this, the present invention aims to propose an efficient method for constructing dynamic data snapshots based on dual-mechanism collaboration, in order to solve the problems of long generation time, low processing efficiency and long-term blocking of real-time transactions in the source database caused by the use of full logical replication in existing snapshot generation methods, and to achieve instantaneous generation of snapshot data and efficient backup of historical snapshots.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a method for efficiently constructing dynamic data snapshots based on dual-mechanism collaboration, the method comprising: S1. Create a master database instance and a slave database instance, and configure the slave database instance as a slave node of the master database instance to establish a master-slave replication relationship; wherein, the master database instance includes a source data table a, and the slave database instance includes a snapshot data table a1 corresponding to the source data table a.

[0007] S2. At the preset snapshot time point, pause the master-slave replication process of the database instance, so that the snapshot data table a1 stops synchronizing the data of the source data table a, and obtain the snapshot data table a1 at the preset snapshot time point.

[0008] S3. Rename the snapshot data table a1 to a2 to obtain the snapshot supply table a2; the snapshot supply table a2 is used to provide the downstream system with full data up to the preset snapshot time point.

[0009] S4. In the database instance, create a new backup data table a1 with the same data structure as the snapshot supply table a2.

[0010] S5. By copying physical files, copy all the data of the snapshot supply table a2 to the backup data table a1 as a new snapshot data table, and rebuild the master-slave synchronization link between the backup data table a1 and the source data table a.

[0011] S6. Resume the master-slave replication process of the database instance, so that the backup data table a1 continues to synchronize the subsequent changes of the source data table a from the synchronization pause time at the preset snapshot time point.

[0012] The beneficial effects of this solution are as follows: In existing technologies, the generation of large-scale data snapshots generally suffers from problems such as excessive time consumption due to the use of full logical replication, linearly increasing generation efficiency with data volume, and prolonged blocking of real-time transactions in the source database, thus restricting business continuity and the timeliness of data supply. This solution, through a collaborative mechanism of master-slave replication and physical file replication, transforms snapshot generation from processing full data to processing short-term incremental data, achieving instantaneous snapshot generation and efficient backup of historical snapshots. This effectively improves the efficiency and stability of snapshot generation and ensures zero-interference operation of online services.

[0013] Furthermore, step S1 also includes the following steps: S110. Enable the binary log function on the primary database and configure the server ID of the primary database.

[0014] S120. Execute the master database management command to create a master-slave synchronization user and grant the master-slave synchronization user replication permissions, and obtain the master database connection parameters.

[0015] S130. Based on the master database connection parameters, perform data synchronization initialization on the slave database to complete the establishment of the master-slave replication relationship.

[0016] Beneficial effects: By standardizing the configuration of the master-slave replication environment, including enabling binary logs, setting a unique server identifier, creating a dedicated synchronization user and performing data initialization, a stable, secure and traceable data synchronization link can be established between the master and slave databases. This provides a reliable data consistency foundation for subsequent instantaneous snapshot generation based on the slave database, avoids data differences or synchronization failures caused by improper replication configuration, and improves the maintainability of the entire snapshot building process.

[0017] Furthermore, step S130 also includes the following steps: S131. Record the filename and log position of the current binary log of the main database.

[0018] S132. Configure master database connection parameters on the slave database to connect to the master database.

[0019] S133. Copy all data from the master database up to the log position point to the slave database to complete the initialization operation of the slave database.

[0020] Beneficial effects: By precisely recording the binary log position of the master database and using this as a benchmark to replicate a complete data snapshot of the master database to the slave database, it is possible to ensure that the slave database maintains strict data consistency with the master database after initialization, and subsequent incremental synchronization can seamlessly continue from this precise position. This mechanism establishes a reliable data benchmark for the entire snapshot scheme, avoiding snapshot data distortion caused by initial data inconsistencies or synchronization breakpoints.

[0021] Furthermore, the pause operation of the master-slave replication process is accomplished by sending a stop command to the slave database instance to suspend the execution of its I / O thread and SQL thread; the start operation of the master-slave replication process is accomplished by sending a start command to the slave database instance to restart the execution of the I / O thread and SQL thread.

[0022] Beneficial effects: By controlling the I / O and SQL threads of the slave database separately, the data synchronization process can be precisely and quickly paused and resumed. This ensures that the data in the slave database is momentarily frozen while minimizing the impact on the business continuity of the master database. This dual-thread collaborative control mechanism enables the snapshot generation process to complete the freezing and thawing of slave database data within seconds, providing a key technical guarantee for achieving efficient and non-blocking snapshot construction.

[0023] Furthermore, the pause and start operations of the master-slave replication process are remotely controlled by connecting to the server where the slave database instance is located via the SSH protocol.

[0024] Beneficial effects: Remotely controlling the pausing and starting of master-slave replication via the SSH protocol enhances the centralized management and deployment flexibility of the entire snapshot process, while reducing the risks and costs associated with manual intervention.

[0025] Furthermore, a replication status query command is executed in the slave database instance to obtain its current replication status information; based on the replication status information, the data synchronization status between the slave database instance and the master database instance is confirmed.

[0026] Beneficial effects: By actively querying and confirming the master-slave replication status, abnormal situations such as data delays and synchronization errors can be detected in a timely manner before critical operations are performed, ensuring that snapshot generation is based on a consistent and reliable data foundation, thereby improving the accuracy of snapshot data and the robustness of the entire process.

[0027] Furthermore, step S5 also includes the following steps: S510. In the slave database instance, perform a tablespace discard operation on the backup data table a1 and delete its corresponding physical data file a1.ibd.

[0028] S520. Perform a table export locking operation on the snapshot supply table a2 to generate a temporary metadata file a2.cfg corresponding to the snapshot supply table a2; and place the snapshot supply table a2 in a read-only locked state; the temporary metadata file a2.cfg is a configuration file used to verify whether the data structure is consistent during the tablespace import process.

[0029] S530. In the data directory of the database instance, copy the temporary metadata file a2.cfg and rename it to a1.cfg; rename the physical data file a2.ibd of the snapshot supply table a2 to a1.ibd.

[0030] S540. Perform an unlock operation on the snapshot supply table a2, release its read-only lock state and delete the temporary metadata file a2.cfg.

[0031] S550. Perform a tablespace import operation on the backup data table a1, loading the physical data file a1.ibd and its associated temporary metadata file a1.cfg as a new tablespace for the backup data table a1.

[0032] Beneficial effects: Through standard tablespace export, file copy and import processes, data can be instantly cloned directly in physical file form at the database level, completely avoiding the full data parsing and reconstruction process required by traditional logical copying. This reduces the copying time of hundreds of millions of data tables from several hours to several minutes. At the same time, the metadata verification mechanism ensures the integrity and structural consistency of the copied data, greatly improving the efficiency and reliability of historical snapshot backups.

[0033] Furthermore, the snapshot supply table a2 and the backup data table a1 are located in the same database of the same slave database instance.

[0034] Beneficial effects: By placing the snapshot provisioning table and the backup data table in the same database instance, physical file copying can be performed directly within the same file system, completely avoiding network overhead and latency caused by cross-instance or cross-database transmission, greatly improving data copying speed, while simplifying file path management and permission configuration, and enhancing the overall process execution efficiency. Attached Figure Description

[0035] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting; in these embodiments, the same reference numerals denote the same structures, wherein: Figure 1 This is an exemplary flowchart of a method for efficiently constructing dynamic data snapshots based on a dual-mechanism collaborative approach; Figure 2 This is an exemplary flowchart of the method for establishing a master-slave replication relationship; Figure 3 This is an exemplary flowchart of a database initialization method; Figure 4 This is an exemplary flowchart of a physical file copying method. Detailed Implementation

[0036] To more clearly illustrate the technical solutions of the embodiments in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some examples or embodiments of this specification. For those skilled in the art, these drawings can be applied to other similar scenarios without creative effort. Unless obvious from the context or otherwise specified, the same reference numerals in the drawings represent the same structures or operations.

[0037] As indicated in this specification and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of expressly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0038] Flowcharts are used in this specification to illustrate the operations performed by the system according to embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0039] The following detailed explanation illustrates the specific implementation methods: Figure 1 This is an exemplary flowchart of a method for efficiently constructing dynamic data snapshots based on a dual-mechanism collaborative approach, such as... Figure 1 As shown, this method is executed by the snapshot building system.

[0040] A snapshot building system is used to generate data snapshots in a database and perform historical backups. In this embodiment, the database used is a MySQL database.

[0041] In this embodiment, the snapshot building system is deployed on a computing device containing at least one processor and memory, and includes a snapshot data synchronization module, a snapshot data backup module, and a snapshot management and control module.

[0042] The snapshot data synchronization module is used to perform master-slave database data synchronization.

[0043] The snapshot data backup module is used to back up snapshot data based on a physical file copying mechanism.

[0044] The snapshot management and control module is used to remotely issue operation commands and coordinate and schedule various modules.

[0045] For the specific collaborative processing methods of the above modules, please refer to the specific method flow described in steps S1 to S6 below.

[0046] A method for efficiently constructing dynamic data snapshots based on a dual-mechanism collaborative approach, comprising: Step S1: Create a master database instance and a slave database instance, and configure the slave database instance as a slave node of the master database instance to establish a master-slave replication relationship; wherein, the master database instance includes a source data table a, and the slave database instance includes a snapshot data table a1 corresponding to the source data table a.

[0047] The master database instance and the slave database instance are independent MySQL database service processes that have been deployed and are running.

[0048] In this embodiment, MySQL database is used as the target database type. It can rely on cloud service platform or local virtual environment to select the appropriate instance specifications such as the number of CPU cores, memory size and storage space, configure the virtual private cloud and subnet related network settings, formulate security group rules that allow specific IP addresses and CIDR blocks to access the database, and complete the creation of the main database instance.

[0049] In this embodiment, the same configuration process as the master database instance is adopted to deploy another MySQL instance as a slave database in the same or different availability zones, and the network and security group configurations described above are used to ensure network reachability between the master and slave instances, thereby ultimately creating the slave database instance.

[0050] Master-slave replication refers to an architectural relationship in which a master database instance actively synchronizes its data changes with one or more slave database instances. In this relationship, the master database handles and records all write operations, while the slave databases apply these changes in real-time or near real-time through replication mechanisms to maintain data consistency with the master.

[0051] A slave node is a database instance configured to receive and apply data changes from the master database instance.

[0052] In this embodiment, the slave database instance runs as a slave node. It obtains updates from the master database instance through the master-slave replication link and can pause synchronization at specific times to provide static data snapshots.

[0053] Furthermore, Figure 2 This is an exemplary flowchart of the method for establishing a master-slave replication relationship, such as... Figure 2 As shown, step S1 further includes the following steps: Step S110: Enable the binary log function on the primary database and configure the server ID of the primary database.

[0054] A binary log is a log file in a MySQL database used to record all statements or events that modify data. These statements or events include data updates and structural changes.

[0055] In this embodiment, the snapshot data synchronization module modifies the MySQL configuration file by adding or setting the value of the parameter log-bin to mysql-bin in the [mysqld] configuration section to specify the filename prefix of the binary log; at the same time, it adds or sets the server-id parameter in the [mysqld] configuration section to configure the server ID of the master database; then, it restarts the MySQL service to make the above configuration take effect.

[0056] The MySQL configuration file can be my.cnf or my.ini; [mysqld] is the configuration section in the MySQL configuration file.

[0057] The `server-id` parameter must be a non-zero positive integer, and this value must serve as a unique identifier for the master database and cannot be duplicated with the `server-id` parameter value of the slave database. For example, `server-id=1`.

[0058] Step S120: Execute the master database management command to create a master-slave synchronization user and grant the master-slave synchronization user replication permissions, and obtain the master database connection parameters.

[0059] In this embodiment, the main database management command is a database management command used to execute step S120.

[0060] As an example only, in this embodiment, the snapshot data synchronization module executes the SQL command "CREATE USER 'repl_user'@'%' IDENTIFIED BY 'password'" on the main database to create a user named repl_user.

[0061] Master-slave synchronization user replication permissions refer to the permissions that allow a user to connect to the master database and request binary log events for data synchronization.

[0062] In this embodiment, after creating a user, the snapshot data synchronization module needs to further execute the SQL command "GRANT REPLICATION SLAVE ON *.* TO 'repl_user'@'%'" on the main database to grant global replication slave privileges to the user named repl_user.

[0063] Master connection parameters are a set of necessary information used by slave databases to connect to the master database for data synchronization. For example, master connection parameters may include the host address, port number, synchronization username, synchronization user password, binary log file name and position, etc.

[0064] In this embodiment, the master database connection parameters can be directly queried and obtained through master database management commands.

[0065] Step S130: Based on the master database connection parameters, perform data synchronization initialization on the slave database to complete the establishment of the master-slave replication relationship.

[0066] Furthermore, Figure 3 This is an exemplary flowchart of a database initialization method, such as... Figure 3 As shown, step S130 further includes the following steps: Step S131: Record the filename and log position of the current binary log in the main database.

[0067] Step S132: Configure the master database connection parameters on the slave database to connect to the master database.

[0068] Step S133: Copy all data from the primary database up to the log position point to the secondary database to complete the initialization operation of the secondary database.

[0069] In this embodiment, by recording the precise binary log position of the master database, configuring the master database connection parameters, and performing full data synchronization based on that position, the slave database data can be completely consistent with the master database at a specified point in time, laying an accurate and reliable data foundation for subsequent incremental synchronization and instantaneous snapshot generation.

[0070] In this embodiment, by enabling binary logs in the master database, configuring a unique server ID, creating a dedicated synchronization account, and completing the full data synchronization initialization of the slave database based on connection parameters, a stable, secure, and consistent basic replication environment between the master and slave databases can be built, providing a reliable and traceable data synchronization link guarantee for subsequent instantaneous snapshot generation based on the slave database.

[0071] Step S2: At the preset snapshot time point, pause the master-slave replication process from the database instance, so that the snapshot data table a1 stops synchronizing the data of the source data table a, and obtain the snapshot data table a1 at the preset snapshot time point.

[0072] A preset snapshot time point is a pre-defined time point used to trigger the snapshot generation operation. For example, the end of each business day or the start time of a specific data verification cycle.

[0073] Furthermore, the pause operation of the master-slave replication process is accomplished by sending a stop command to the slave database instance to suspend the execution of its I / O thread and SQL thread; the start operation of the master-slave replication process is accomplished by sending a start command to the slave database instance to restart the execution of the I / O thread and SQL thread.

[0074] The stop instruction is used to interrupt the execution of the data synchronization thread from the database instance; the start instruction is used to resume the execution of the data synchronization thread from the database instance.

[0075] In this embodiment, the master-slave replication process can be paused by sending the SQL command "STOP SLAVE IO_THREAD" to the slave database instance to pause its I / O thread and "STOP SLAVE SQL_THREAD" to pause its SQL thread.

[0076] In this embodiment, the master-slave replication process can be resumed by sending the SQL command "START SLAVE IO_THREAD" to the slave database instance to start its I / O thread and sending "START SLAVE SQL_THREAD" to start its SQL thread.

[0077] By sending stop and start commands to the slave database instance respectively to precisely control the pause and restart of its I / O thread and SQL thread, millisecond-level precise start and stop control of the master-slave replication process can be achieved, ensuring that snapshot data can be "frozen" at a precise point in time, while minimizing the blocking time of the master database's real-time business.

[0078] Furthermore, a replication status query command is executed from the slave database instance to obtain its current replication status information; based on this replication status information, the data synchronization status between the slave database instance and the master database instance is confirmed.

[0079] The command to query the replication status can be "SHOW SLAVE STATUS\G".

[0080] In this embodiment, by actively querying and verifying the replication status from the database before snapshot generation, it is possible to proactively perceive and confirm data synchronization delays, errors, and consistency, thereby ensuring that the snapshot building process begins on a healthy, synchronized, and stable data foundation, effectively improving the accuracy of snapshot data and the reliability of the entire solution.

[0081] Furthermore, the pause and start operations of the master-slave replication process are remotely controlled by the snapshot management control module via the SSH protocol connecting to the server where the slave database instance resides.

[0082] In this embodiment, the snapshot management control module can establish an encrypted network connection between the application server and the server where the slave database instance is located by running an SSH client on the application server; then, through the SSH session, it can execute the corresponding MySQL client commands on the slave server, thereby realizing remote control of the pause and start operations of the master-slave replication process.

[0083] In this embodiment, the snapshot management control module remotely executes the pause and start operations of master-slave replication based on the SSH protocol, which can realize cross-network, automated centralized management and control of database replication status, effectively reducing the operational risks and costs of manual intervention.

[0084] Step S3: Rename the snapshot data table a1 to a2 to obtain the snapshot supply table a2; the snapshot supply table a2 is used to provide the downstream system with full data up to the preset snapshot time point.

[0085] In this embodiment, this configuration is used to logically isolate the snapshot supply table a2 from the original snapshot data table a1, which is to be restored from dynamic synchronization. Since the end time of queries to snapshot data table a1 by downstream systems is unpredictable, if snapshot data table a1 is used directly for queries, the data in snapshot data table a1 will continue to be updated after master-slave synchronization is restored. This will cause downstream query results to change during the snapshot usage period, affecting data stability and consistency. By renaming snapshot data table a1 to a2, downstream systems can always query based on a static table a2, thereby ensuring data stability and repeatability within the snapshot usage period.

[0086] Step S4: Create a backup data table a1 from the database instance, with the same data structure as the snapshot provision table a2.

[0087] Step S5: The snapshot data backup module copies the full data of the snapshot supply table a2 to the backup data table a1 through physical file copying, creating a new snapshot data table, and rebuilds the master-slave synchronization link between the backup data table a1 and the source data table a.

[0088] Physical file copying refers to the method of directly copying the underlying physical data files and associated metadata configuration files of the database to achieve rapid data cloning.

[0089] In this embodiment, the underlying physical data file is an .ibd file; the metadata configuration file is a .cfg file.

[0090] In this embodiment, MySQL 5.6 or a higher version is required to use this mechanism to achieve fast cross-tablespace replication. Previously, similar functionality could only be achieved through less efficient methods such as logical export or full table copy.

[0091] Furthermore, such as Figure 4 As shown, step S5 further includes: Step S510: In the database instance, perform a tablespace discard operation on the backup data table a1 and delete its corresponding physical data file a1.ibd.

[0092] In this embodiment, by sending the SQL command "ALTER TABLE a1 DISCARDTABLESPACE" to the database instance, a tablespace discard operation is performed on the backup data table a1. This command will delete its corresponding physical data file a1.ibd, making table a1 an empty table that only retains the structure definition.

[0093] S520. Perform a table export locking operation on snapshot supply table a2 to generate a temporary metadata file a2.cfg corresponding to snapshot supply table a2; and put snapshot supply table a2 into a read-only locked state; the temporary metadata file a2.cfg is a configuration file used to verify whether the data structure is consistent during the tablespace import process.

[0094] In this embodiment, the snapshot data backup module sends the SQL command "FLUSHTABLE a2 FOR EXPORT" to the database instance to perform a table export lock operation on the snapshot provider table a2. By executing this command, a temporary metadata file a2.cfg is generated, and the snapshot provider table a2 is placed in a read-only locked state. This operation ensures that the data state of the source table is absolutely static and the structural information is accurately recorded during file copying before the physical copying of the .ibd data file.

[0095] S530. Copy the temporary metadata file a2.cfg from the data directory of the database instance and rename it to a1.cfg; rename the physical data file a2.ibd of the snapshot provision table a2 to a1.ibd.

[0096] S540. Perform an unlock operation on snapshot supply table a2, release its read-only lock state and delete the temporary metadata file a2.cfg.

[0097] In this embodiment, the snapshot provisioning table a2 is unlocked by sending the SQL command "UNLOCK TABLES" to the database instance. This command will release the read-only lock on the snapshot provisioning table a2, restoring normal read and write access, and automatically delete the temporary metadata file a2.cfg.

[0098] S550. Perform a tablespace import operation on the backup data table a1, loading the physical data file a1.ibd and its associated temporary metadata file a1.cfg as a new tablespace for the backup data table a1.

[0099] In this embodiment, by sending the SQL command "ALTER TABLE a1 IMPORTTABLESPACE" to the database instance to perform a tablespace import operation on the backup data table a1, the backup data table a1 instantly has the same data content and structure as the snapshot supply table a2, thereby realizing instantaneous data recovery based on physical files.

[0100] Furthermore, the snapshot provision table a2 and the backup data table a1 reside in the same database within the same slave database instance.

[0101] By placing the snapshot provision table a2 and the backup data table a1 in the same database of the same database instance, direct copying of physical files within the same file system can be achieved, completely avoiding the network transmission, permission verification, and path mapping overhead that may be caused by cross-instance or cross-database operations. This minimizes data copying time, simplifies the operation process, and improves the atomicity and reliability of the overall execution.

[0102] Step S6: Resume the master-slave replication process from the database instance, so that the backup data table a1 continues to synchronize subsequent changes to the source data table a from the synchronization pause time at the preset snapshot time point.

[0103] In this embodiment, steps S1-S6, through a collaborative mechanism of master-slave replication and physical file replication, transform snapshot generation from processing full data to processing short-term incremental data, thereby achieving instantaneous snapshot generation and efficient backup of historical snapshots. This effectively improves the efficiency and stability of snapshot generation and ensures zero-interference operation of online services.

[0104] The basic concepts have been described above. Obviously, for those skilled in the art, the detailed disclosure above is merely illustrative and does not constitute a limitation of this specification. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this specification. Such modifications, improvements, and corrections are suggested in this specification and therefore remain within the spirit and scope of the exemplary embodiments described herein.

[0105] Furthermore, unless expressly stated in the claims, the order of processing elements and sequences, the use of numbers and letters, or other names described in this specification are not intended to limit the order of the processes and methods described herein. Although various examples have been discussed in the foregoing disclosure of some embodiments of the invention that are currently considered useful, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments; rather, the claims are intended to cover all modifications and equivalent combinations that conform to the spirit and scope of the embodiments described herein. For example, while the system components described above can be implemented using hardware devices, they can also be implemented solely using software solutions, such as installing the described system on existing servers or mobile devices.

[0106] Similarly, it should be noted that, in order to simplify the description disclosed herein and thus aid in the understanding of one or more embodiments of the invention, the foregoing description of embodiments in this specification may sometimes combine multiple features into a single embodiment, drawing, or description thereof. However, this method of disclosure does not imply that the subject matter of this specification requires more features than those mentioned in the claims. In fact, the embodiments contain fewer features than all the features of a single embodiment disclosed above.

[0107] In some embodiments, numbers describing the quantity of components and attributes are used. It should be understood that such numbers used in the description of embodiments are modified in some examples with the terms "approximately," "approximately," or "generally." Unless otherwise stated, "approximately," "approximately," or "generally" indicates that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may be changed depending on the characteristics required by individual embodiments. In some embodiments, numerical parameters should take into account specified significant digits and employ a general method of digit reservation. Although the numerical ranges and parameters used to confirm their breadth of range in some embodiments of this specification are approximate values, in specific embodiments, such values ​​are set as precisely as feasible.

[0108] For each patent, patent application, patent application publication, and other material such as articles, books, specifications, publications, and documents referenced in this specification, the entire contents of which are incorporated herein by reference. This excludes historical application documents that are inconsistent with or conflict with the content of this specification, as well as documents that limit the broadest scope of the claims in this specification (currently or subsequently appended to this specification). It should be noted that in the event of any inconsistency or conflict between the descriptions, definitions, and / or terminology used in the supplementary materials to this specification and the content of this specification, the descriptions, definitions, and / or terminology used in this specification shall prevail.

[0109] Finally, it should be understood that the embodiments described in this specification are merely illustrative of the principles of the embodiments described herein. Other variations may also fall within the scope of this specification. Therefore, alternative configurations of the embodiments described herein are intended to be illustrative rather than limiting, and should be considered consistent with the teachings of this specification. Accordingly, the embodiments described herein are not limited to those explicitly introduced and described herein.

Claims

1. A dynamic data snapshot efficient construction method based on dual-mechanism cooperation, characterized in that, The method comprises: S1, creating a master database instance and a slave database instance, and configuring the slave database instance as a slave node of the master database instance to establish a master-slave replication relationship; wherein the master database instance comprises a source data table a, and the slave database instance comprises a snapshot data table a1 corresponding to the source data table a; S2, at a preset snapshot time point, pausing the master-slave replication process of the slave database instance, so that the snapshot data table a1 stops synchronizing data of the source data table a, to obtain a snapshot data table a1 of the preset snapshot time point; S3, renaming the snapshot data table a1 as a2 to obtain a snapshot supply table a2; the snapshot supply table a2 is used to provide full data of the preset snapshot time point to a downstream system; S4, in the slave database instance, a backup data table a1 with the same data structure as the snapshot supply table a2 is newly created; S5, full data of the snapshot supply table a2 is copied to the backup data table a1 by a physical file copying mode, as a new snapshot data table, and a master-slave synchronization link between the backup data table a1 and the source data table a is reestablished; S6, the master-slave replication process of the slave database instance is resumed, so that the backup data table a1 continues to synchronize subsequent changed data of the source data table a from the synchronization pause time point of the preset snapshot time point.

2. The method of claim 1, wherein, The step S1 further comprises the following steps: S110, enabling a binary log function on the master database, and configuring a server ID of the master database; S120, executing a master database management command to create a master-slave synchronization user and grant the master-slave synchronization user a replication permission, to obtain master database connection parameters; S130, based on the master database connection parameters, performing data synchronization initialization on the slave database to complete establishment of the master-slave replication relationship.

3. The method of claim 2, wherein, The step S130 further comprises the following steps: S131, recording a file name and a log position point of a current binary log of the master database; S132, configuring the master database connection parameters on the slave database to connect the master database; S133, copying full data of the master database up to the log position point to the slave database to complete initialization operation of the slave database.

4. The method of claim 1, wherein, Further comprising: The pause operation of the master-slave replication process is completed by sending a stop instruction to the slave database instance to pause execution of an I / O thread and an SQL thread; The start operation of the master-slave replication process is completed by sending a start instruction to the slave database instance to restart execution of the I / O thread and the SQL thread.

5. The method of claim 4, wherein, The pause and start operations of the master-slave replication process are realized by remotely controlling a server where the slave database instance is located through an SSH protocol.

6. The method of claim 4, wherein, Further comprising: executing a replication state query instruction in the slave database instance to obtain current replication state information thereof; based on the replication state information, confirming a data synchronization state between the slave database instance and the master database instance.

7. The method of claim 1, wherein, The step S5 further comprises the following steps: S510, performing a table space discard operation on the backup data table a1 in the slave database instance to delete its corresponding physical data file a1.ibd; S520, performing a table export locking operation on the snapshot provisioning table a2 to generate a temporary metadata file a2.cfg corresponding to the snapshot provisioning table a2, and putting the snapshot provisioning table a2 into a read-only locking state; the temporary metadata file a2.cfg is a configuration file used to verify consistency of data structures in a table space import process; S530, copying and renaming the temporary metadata file a2.cfg as a1.cfg under the data directory of the slave database instance, and renaming the physical data file a2.ibd of the snapshot provisioning table a2 as a1.ibd; S540, performing an unlocking operation on the snapshot provisioning table a2 to release its read-only locking state and delete the temporary metadata file a2.cfg; S550, performing a table space import operation on the backup data table a1 to load the physical data file a1.ibd and its associated temporary metadata file a1.cfg as a new table space of the backup data table a1.

8. The method of claim 1, wherein, The snapshot provisioning table a2 and the backup data table a1 are located in a same database of a same slave database instance.

Images