Data synchronization method and system, electronic equipment and computer readable storage medium
By merging commands with identical keys from the same database and sending them to the target database, the problems of high network bandwidth and large latency during data synchronization are solved, thus improving synchronization efficiency and stability.
Patent Information
- Application Number
- CN202411016728.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2026-02-03
AI Technical Summary
During data synchronization, existing technologies suffer from high network bandwidth consumption, significant latency, and high resource consumption, especially during long-distance transmission and large key operations, which have a significant impact on the performance of the target database.
By parsing the source database log files, identifying commands that are in the same database and have the same key, merging these commands to form a second command set, and sending it to the target database for data synchronization, network interaction is reduced.
It significantly reduced network bandwidth usage, decreased command transmission latency, improved data synchronization efficiency and stability, and solved the problems of high latency and high resource consumption.
Smart Images
Figure CN121456048A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data transmission technology, and in particular to data synchronization methods, systems, electronic devices, and computer-readable storage media. Background Technology
[0002] Data synchronization refers to maintaining consistency between data from different data sources or storage locations. It can include real-time synchronization, periodic synchronization, or on-demand synchronization. In databases, data synchronization typically refers to maintaining data consistency between different databases, including master-slave synchronization, multi-master synchronization, and cross-datacenter synchronization. Data synchronization is crucial in distributed systems, cloud computing environments, and multi-datacenter deployments, ensuring data integrity, availability, and consistency. However, due to factors such as network bandwidth and transmission distance between the source and target databases, data synchronization latency can be high. Summary of the Invention
[0003] This application provides data synchronization methods, systems, electronic devices, and computer-readable storage media to solve one or more of the above-mentioned technical problems.
[0004] In a first aspect, embodiments of this application provide a data synchronization method, including:
[0005] After parsing the source database log file into commands supported by the current data synchronization framework, commands belonging to the same database and having the same key are identified to obtain a first set of commands, wherein the commands are used to perform modification operations on the source data in the source database;
[0006] The commands in the first command set are merged to obtain the second command set;
[0007] The second set of commands is sent to the target database to synchronize the target data in the target database, wherein the source data corresponds to the target data.
[0008] Secondly, embodiments of this application provide a data synchronization system, including:
[0009] A command parser is used to parse the source database log file into commands supported by the current data synchronization framework, wherein the commands are used to perform modification operations on the source data in the source database;
[0010] A command merger is used to identify commands that belong to the same database and have the same key, obtain a first command set, and merge the commands in the first command set to obtain a second command set;
[0011] A command writer is used to send the second set of commands to a target database to synchronize target data in the target database, wherein the source data corresponds to the target data.
[0012] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method described in any of the above-mentioned embodiments.
[0013] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method described in any of the above-mentioned embodiments.
[0014] Fifthly, embodiments of this application provide a computer program product, including computer instructions, which, when executed by a processor, implement the method described above.
[0015] Compared with related technologies, this application has the following advantages:
[0016] According to the embodiments of this application, after parsing the source database log file into commands supported by the current data synchronization framework, commands belonging to the same database and having the same key are identified to obtain a first command set. These commands are used to perform modification operations on source data in the source database. The commands in the first command set are then merged to obtain a second command set. This second command set is then sent to the target database to synchronize target data in the target database, where the source data corresponds to the target data. In other words, this embodiment merges commands belonging to the same database and having the same key before sending them to the target database for execution. This reduces network interaction with the target database, reduces bandwidth usage, and thus improves data synchronization efficiency and stability. It solves the technical problem in related technologies where commands from the source database are sent sequentially to the target database for execution, resulting in significant data synchronization delays and high resource consumption.
[0017] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description
[0018] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.
[0019] Figure 1 This illustration shows an application scenario diagram of the data synchronization method provided in the embodiments of this application;
[0020] Figure 2 A flowchart of the data synchronization method provided in the embodiments of this application is shown;
[0021] Figure 3 A structural block diagram of the data synchronization device provided in the embodiments of this application is shown;
[0022] Figure 4 A structural block diagram of the data synchronization system provided in the embodiments of this application is shown;
[0023] Figure 5 The diagram illustrates the structural block diagram of a data synchronization system for deep learning model optimization scenarios provided in this application embodiment, and...
[0024] Figure 6 A block diagram of an electronic device used to implement embodiments of this application is shown. Detailed Implementation
[0025] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.
[0026] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.
[0027] Data Transmission Service (DTS) enables real-time synchronization between key-value (KV) databases (KV databases are key-value stores that store data in simple key-value pairs, where each key is unique and corresponds to a value. These databases are typically used in scenarios requiring fast data storage and retrieval, such as caching, session storage, and configuration information. Common KV databases include Redis, Memcached, LevelDB, and RocksDB). Figure 1During the process (as shown), the following technical problems typically exist: 1. If DTS executes commands in the target database according to the command order of the source database, it will result in high network bandwidth consumption. When the source database experiences frequent insertions, updates, and deletions of large keys, insufficient network bandwidth can lead to latency. 2. When there is long-distance transmission between the DTS process and the target database, which refers to data transmission over a significant physical distance—potentially involving data transmission between different cities, countries, or even continents—long-distance transmission in computer networks typically requires consideration of network latency, bandwidth limitations, and security. Therefore, appropriate data transmission protocols and technologies are needed to ensure data security and reliability. In data synchronization, long-distance transmission is a crucial consideration because it can impact data transmission speed, stability, and cost, leading to high command transmission response time (RT), low overall synchronization performance, and latency. 3. Frequent operations on large keys (keys with values occupying a large amount of space) significantly impact the performance of the target database. When users want to use DTS to downgrade the target database, insufficient performance of the target database can cause DTS latency.
[0028] In view of this, embodiments of this application provide a data synchronization method to solve all or part of the above-mentioned technical problems. Application scenarios of embodiments of this application include, but are not limited to: hot key updates (i.e., high-concurrency modification of certain field values in the same row of data in a database within a short period of time; for example, flash sales of goods, continuous modification and updating of inventory data, etc.), long-link transmission, etc. Figure 2 As shown, the above data synchronization methods include:
[0029] S202, after parsing the source database log file into commands supported by the current data synchronization framework, determine the commands that belong to the same database and have the same key, and obtain the first set of commands, where the command is used to perform modification operations on the source data in the source database.
[0030] It should be noted that the source databases mentioned above, and the corresponding target databases in subsequent steps, include, but are not limited to, key-value databases such as Redis, Amazon DynamoDB, etcd, and TiKV. Multiple target databases may be included, and data synchronization operations need to be performed on all of them during data synchronization. Furthermore, the source and target databases are generally homogeneous. If heterogeneous databases exist, data processing is required before data transmission to enable data synchronization in the target databases.
[0031] It should also be noted that, in this embodiment, the source database log file can be parsed into commands supported by the current data synchronization framework by parsing and serializing the log file into a data format that the current data synchronization framework can process after obtaining the source database log file. The aforementioned log file is used to record all transactions and the modifications made to the database by each transaction.
[0032] In one possible implementation, the above commands can be represented as (database db where the command resides, specific command), for example, command 1 (db0, sets K1 to V1), command 2 (db0, sets K1 to V2), etc. The first set of commands mentioned above can generally include multiple commands. These commands include, but are not limited to, string commands, hash table commands, list commands, etc.
[0033] S204, merge the commands in the first command set to obtain the second command set.
[0034] It should be noted that the aforementioned second command set may include one command or multiple commands. However, the total number of commands in the second command set is generally less than the total number of commands in the aforementioned first command set. Optionally, the commands in the first command set can be merged in ways including, but not limited to: Method 1, deleting some commands from the first command set; Method 2, rewriting the commands in the first command set. The principles for merging in Method 1 and rewriting in Method 2 can be: disregarding the intermediate state of data writing and only requiring eventual consistency.
[0035] Additionally, it should be noted that in this embodiment, if the commands in the second command set are not supported by the target database, the system can fall back to the first command set for sequential sending.
[0036] Optionally, in the embodiments of this application, the above command merging process can be optimized using a deep learning model (large model). Specifically, it can include the following: (1) Predictive optimization: The large model can pre-merge or reorganize database commands by predicting user query behavior or application patterns. For example, the model can analyze past query patterns, predict possible high-frequency queries in the short term, and perform query optimization in advance. (2) Automated rule learning: In traditional database management, query merging requires database administrators to manually set it based on experience. The large model technology can automatically learn and propose the best merging rule configuration by analyzing historical data, thereby achieving automated database maintenance and optimization. (3) Anomaly detection and self-adjustment: The large model can monitor abnormal behavior of database operations and automatically adjust the merging strategy to adapt to special situations or stress tests. For example, it can automatically adjust the query execution order and resource allocation during peak traffic periods. (4) Simulation and prediction of execution paths: Machine learning models are used to simulate different command merging strategies, predict their execution efficiency, and select the optimal strategy, similar to using simulated search in complex decision-making processes.
[0037] In addition, the above command merging process can be started automatically when the network conditions are poor, or it can be started by default without affecting the data quality.
[0038] S206, the second set of commands is sent to the target database to synchronize the target data in the target database, wherein the source data corresponds to the target data.
[0039] Optionally, in this embodiment, the timing of sending the second command set to the target database includes, but is not limited to, updating the target database during off-peak hours or at night to avoid impacting normal service operations. Of course, the sending timing can also differ for different databases. Furthermore, in this embodiment, synchronizing the target data in the target database can involve executing the commands in the second command set on the target data corresponding to the source data, so that the data in the source database and the target database eventually remain consistent.
[0040] Through steps S202-S206 above, after parsing the source database log file into commands supported by the current data synchronization framework, commands belonging to the same database and sharing the same key are identified, resulting in a first command set. These commands are used to perform modification operations on source data in the source database. The commands in the first command set are then merged to obtain a second command set. This second command set is then sent to the target database to synchronize target data, where the source data corresponds to the target data. In other words, this embodiment merges commands belonging to the same database and sharing the same key before sending them to the target database for execution. This reduces network interaction with the target database, reduces bandwidth usage, and thus improves data synchronization efficiency and stability. It solves the technical problem in related technologies where commands from the source database are sent sequentially to the target database for execution, resulting in significant data synchronization delays and high resource consumption.
[0041] In one possible implementation, merging the commands in the first command set to obtain the second command set may include: S11, merging the commands in the first command set into a set whose total number of commands is less than the total number of commands in the first command set, thus obtaining the second command set. For example, assuming the commands in the first command set include: command 1, command 2, and command 3, then the first command set can be merged into a set containing only command 3. Alternatively, the first command set can be merged into a set including command 1 and command 2. Optionally, merging the commands in the first command set to obtain the second command set may also include: S12, rewriting the commands in the first command set into a set of command strings, thus obtaining the second command set, wherein the command string is a command form that concatenates multiple commands into one command. For example, assuming the commands in the first command set include: command 1, command 2, and command 3, then the commands in the first command set can be concatenated into a final command, which includes command 1, command 2, and command 3. Alternatively, assuming the first command set includes commands 1, 2, and 3, commands 1 and 2 are concatenated into a single command A. The second command set then includes command A and command 3. Through the above steps S11 and S12, network bandwidth usage is significantly reduced, and command transmission latency is decreased.
[0042] One possible implementation of S11 above could be: S111, delete the second command preceding the first command in the first command set, wherein the final command determined by the first command and the second command is the first command; S112, determine the first command as a command in the second command set, thus obtaining the second command set. Examples are given below for each of the second command sets including one or more commands. For example, if the second command set includes one command, consider the following: Suppose the commands in the first command set are string commands, including: (db0, set k1 v1), (db0, set k1 v2), (db0, set k1 v3), (db0, set k1 v4), these four commands can be merged into one command (db0, set k1 v4). For example, suppose the commands in the first command set are string commands, including: (db0, set k1v1), (db0, set k1 v2), (db0, set k1 v3), (db0, set k1 v4), (db0, del k1), these 5 commands can be merged into 1 command (db0, del k1). Regarding the second command set including multiple commands, for example: Suppose the commands in the first command set are string commands, including: (db0, set k1 v1), (db0, set k2 v2), (db1, set k1 v1), (db1, set k2 v2), (db0, set k1 v2), (db0, set k1 v3), (db0, set k1 v4). This includes the operations of k1 and k2 in db0 and the operations of k1 and k2 in db1. The merged command is (db0, set k1v4), (db0, set k2 v2), (db1, set k1 v1), (db1, set k2 v2). Furthermore, for a list command within the first command set, consider the following example: Assume the commands in the first command set are list commands, including: (db0, lpush k1 v1), (db0, lpush k1 v2), and (db0, lpop k1), which can be merged into (db0, lpush k1 v1). It should be noted that "set" corresponds to "set", "lpush" corresponds to "insert one or more values at the head of the list", and "lpop" corresponds to "remove and retrieve the first element of the list".
[0043] One possible implementation of S12 above could be: S121, rewriting the commands in the first command set into one or more commands using a concatenation function to obtain a command string set; S122, setting the command string set as the second command set. For example, the commands in the first command set include: (db0, hset k1 f1 v1), (db0, hset k1 f2v2), (db0, hset k3 f3 v3), (db1, hset k1 f1 v1), which can be merged into (db0, hset k1 f1 v1 f2v2), (db0, hset k3 f3 v3), (db1, hset k1 f1 v1). For example, the commands in this first command set include: (db0, lpush k1 v1), (db0, lpush k1 v2), (db0, lpush k3 v3), (db1, lpush k1 v1), which can be merged into (db0, lpush k1 v1 v2), (db0, lpush k3 v3), (db1, lpush k1 v1). It should be noted that the above "hmset" corresponds to "simultaneously setting multiple field-value pairs into the hash table key".
[0044] In one possible implementation, after obtaining the source database log file, the process may further include: S21, obtaining the target data format supported by the current data synchronization framework; and S22, parsing the log file into commands with the target data format. Steps S21 and S22 resolve the technical problem in related technologies where unreadable logs prevent further data synchronization.
[0045] S104 above may include: S31, caching commands belonging to the same database and having the same key into the same hash slot; S32, determining the first command set through the commands in the same hash slot. It should be noted that the hash slot can be used to manage the relationship between data and nodes, essentially placing slots on nodes and data in the slots. Efficient element retrieval is achieved by establishing a mapping between keys and values. Specifically, in the application embodiment, the database of the command and the command key in the log file can be used as unique identifiers (ids), cached into hash slots respectively, and merged and rewritten with commands in the same hash slot.
[0046] In summary, this application's embodiments adopt the principle of eventual data consistency. By merging and rewriting keys under the same database, commands from a batch of source databases are converted into one or more commands supported by the target database. This reduces network interaction with the target database, reduces bandwidth usage, improves performance, and reduces the impact on the target database. Through actual testing and application, the above merging method has achieved significant technical effects in real-time KV database synchronization: by merging hot keys, it significantly reduces network bandwidth usage, reduces command transmission latency, improves real-time synchronization performance and efficiency, effectively enhances long-distance data synchronization performance, and reduces synchronization latency and resource consumption. The bandwidth used by a single-task DTS has decreased from 120MBPS to 40MBPS, achieving a 3-fold performance improvement, while bandwidth usage is reduced to 1 / 3 of the original. This effectively solves the performance bottleneck problem in real-time synchronization, improves synchronization efficiency and data transmission stability, and has high practical value and economic benefits in KV database data transmission synchronization systems.
[0047] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.
[0048] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.
[0049] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide a data synchronization device, such as... Figure 3 The diagram shown is a structural block diagram of a data synchronization device according to an embodiment of this application. The device may include:
[0050] The first determining module 32 is used to determine, after parsing the source database log file into commands supported by the current data synchronization framework, commands belonging to the same database and with the same key, to obtain a first command set. This command is used to perform modification operations on source data in the source database. It should be noted that the aforementioned source database and the corresponding target database in subsequent steps include, but are not limited to, key-value databases such as Redis, Amazon DynamoDB, etcd, and TiKV. Multiple target databases may be included, and data synchronization operations need to be performed on these multiple target databases during data synchronization. Furthermore, generally, the source database and the target database are homogeneous databases. If heterogeneous databases exist, data processing is required before data transmission to enable data synchronization in the target database. It should also be noted that, in this embodiment, parsing the source database log file into commands supported by the current data synchronization framework can be achieved by parsing and serializing the log file into commands in a data format that the current data synchronization framework can process after obtaining the source database log file. The aforementioned log file is used to record all transactions and the modifications made to the database by each transaction. In one possible implementation, the above commands can be represented as (database db where the command resides, specific command), for example, it can represent command 1 (db0, setting K1 to V1), command 2 (db0, setting K1 to V2), etc. The above first set of commands can generally include multiple commands. The above commands include, but are not limited to: string commands, hash table commands, list commands, etc.
[0051] The merging module 34 is used to merge the commands in the first command set to obtain a second command set. It should be noted that the second command set may include one command or multiple commands. However, the total number of commands in the second command set is generally less than the total number of commands in the first command set. Optionally, the methods for merging the commands in the first command set include, but are not limited to: Method 1, deleting some commands in the first command set; Method 2, rewriting the commands in the first command set. The principles for merging in Method 1 and rewriting in Method 2 can be: disregarding the intermediate state of data writing, only requiring eventual consistency. Furthermore, it should be noted that in this embodiment, if the commands in the second command set are not supported by the target database, it is possible to fall back to the first command set for sequential sending.
[0052] Optionally, in this embodiment, the above-mentioned merging module 34 can be optimized using a deep learning model (large model). Specifically, it can include the following: (1) Predictive optimization: The large model can pre-merge or reorganize database commands by predicting user query behavior or application patterns. For example, the model can analyze past query patterns, predict possible high-frequency queries in the short term, and perform query optimization in advance. (2) Automated rule learning: In traditional database management, query merging requires database administrators to manually set it based on experience. The large model technology can automatically learn and propose the best merging rule configuration by analyzing historical data, thereby achieving automated database maintenance and optimization. (3) Anomaly detection and self-adjustment: The large model can monitor abnormal behavior of database operations and automatically adjust the merging strategy to adapt to special situations or stress tests. For example, it can automatically adjust the query execution order and resource allocation during peak traffic periods. (4) Simulation and prediction of execution paths: Machine learning models are used to simulate different command merging strategies, predict their execution efficiency, and select the optimal strategy, similar to using simulated search in complex decision-making processes.
[0053] In addition, the aforementioned merging module 34 can be automatically activated when network conditions are poor, or it can be activated by default without affecting data quality.
[0054] Synchronization module 36 is used to send the second command set to the target database to synchronize target data in the target database, wherein the source data corresponds to the target data. Optionally, in this embodiment, the timing of sending the second command set to the target database includes, but is not limited to, updating the target database during off-peak hours or at night to avoid affecting normal service operations. Of course, the sending timing can also be different for different databases. In addition, in this embodiment, synchronizing the target data in the target database can be achieved by executing the commands in the second command set on the target data corresponding to the source data, so that the data in the source database and the target database eventually remain consistent.
[0055] pass Figure 3 The device shown merges commands belonging to the same database and with the same key before sending them to the target database for execution. This reduces network interaction with the target database, reduces bandwidth usage, and thus improves data synchronization efficiency and stability. It solves the technical problems in related technologies where commands from the source database are sent sequentially to the target database for execution, resulting in large data synchronization delays and high resource consumption.
[0056] In one possible implementation, the merging module 34 includes: a merging unit, configured to merge the commands in the first command set into a set whose total number of commands is less than the total number of commands in the first command set, to obtain the second command set; or, a rewriting unit, configured to rewrite the commands in the first command set into a set of command strings, to obtain the second command set, wherein the command string is a command form that concatenates multiple commands into one command.
[0057] The aforementioned merging unit includes: a deletion subunit, used to delete a second command preceding the first command in the first command set, wherein the final command determined by the first command and the second command is the first command; and a determination subunit, used to determine the first command as a command in the second command set, thereby obtaining the second command set.
[0058] The above rewriting unit includes: a rewriting subunit, used to rewrite the commands in the first command set into one or more commands through a concatenation function to obtain a command string set; and a setting subunit, used to set the command string set as the second command set.
[0059] The first determining module 32 includes: a caching unit for storing commands that belong to the same database and have the same key; and a determining unit for determining the first set of commands through the commands in the same hash slot.
[0060] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.
[0061] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide a data synchronization system. For example... Figure 4 The diagram shown is a structural block diagram of a data synchronization system according to an embodiment of this application. The system may include:
[0062] Command parser 42 is used to parse the source database log file into commands supported by the current data synchronization framework, wherein the commands are used to perform modification operations on the source data in the source database;
[0063] Command merger 44 is used to identify commands belonging to the same database and with the same key, obtain a first command set, and merge the commands in the first command set to obtain a second command set. It should be noted that the source database and the corresponding target database in subsequent steps include, but are not limited to, key-value databases such as Redis, Amazon DynamoDB, etcd, and TiKV. Multiple target databases may be included, and data synchronization operations need to be performed on these multiple target databases during data synchronization. Furthermore, generally, the source database and the target database are homogeneous databases. If heterogeneous databases exist, data processing is required before data transmission to enable data synchronization in the target database. It should also be noted that in this embodiment, the source database log file can be parsed and serialized into commands in a data format that the current data synchronization framework can process after obtaining the source database log file. The log file records all transactions and the modifications made to the database by each transaction. In one possible implementation, the above command can be represented as (database db where the command resides, specific command), or as command 1 (db0, setting K1 to V1), command 2 (db0, setting K1 to V2), etc. The first command set generally includes multiple commands. These commands include, but are not limited to, string commands, hash table commands, list commands, etc. It should be noted that the second command set can include one command or multiple commands. However, the total number of commands in the second command set is generally less than the total number of commands in the first command set. Optionally, the methods for merging commands in the first command set include, but are not limited to, method one: deleting some commands from the first command set; and method two: rewriting the commands in the first command set. The principles for merging in method one and rewriting in method two can be: not caring about the intermediate state of data writing, only requiring eventual consistency. Furthermore, it should be noted that in this embodiment, if the commands in the second command set are not supported by the target database, it is possible to fall back to the first command set for sequential sending.
[0064] Optionally, in this embodiment, the command merger 44 can be optimized using a deep learning model (large model) 52. Specifically, this may include the following: (1) Predictive optimization: The large model can pre-merge or reorganize database commands by predicting user query behavior or application patterns. For example, the model can analyze past query patterns, predict possible high-frequency queries in the short term, and perform query optimization in advance. (2) Automated rule learning: In traditional database management, query merging requires database administrators to manually set it based on experience. The large model technology can automatically learn and propose the best merging rule configuration by analyzing historical data, thereby achieving automated database maintenance and optimization. (3) Anomaly detection and self-adjustment: The large model can monitor abnormal behavior of database operations and automatically adjust the merging strategy to adapt to special situations or stress tests, such as automatically adjusting the query execution order and resource allocation during peak traffic periods. (4) Simulation and prediction of execution paths: Using machine learning models to simulate different command merging strategies, predict their execution efficiency, and select the optimal strategy, similar to using simulated search in complex decision-making processes. Figure 5 As shown.
[0065] In addition, the aforementioned command merger 44 can be automatically enabled when network conditions are poor, or it can be enabled by default without affecting data quality.
[0066] Command writer 46 is used to send the second set of commands to the target database to synchronize target data in the target database, wherein the source data corresponds to the target data. Optionally, in this embodiment, the timing of sending the second set of commands to the target database includes, but is not limited to, updating the target database during off-peak hours or at night to avoid affecting normal service operations. Of course, the sending timing can also be different for different databases. In addition, in this embodiment, synchronizing the target data in the target database can be achieved by executing the commands in the second set of commands on the target data corresponding to the source data, so that the data in the source database and the target database eventually remain consistent.
[0067] pass Figure 4 The system shown merges commands belonging to the same database and with the same key before sending them to the target database for execution. This reduces network interaction with the target database, reduces bandwidth usage, and thus improves data synchronization efficiency and stability. It solves the technical problems in related technologies where commands from the source database are sent to the target database sequentially for execution, resulting in large data synchronization delays and high resource consumption.
[0068] In one possible implementation, the command merger 44 described above may also include at least one of the following: a string command merger, a hash table command merger, or a list command merger.
[0069] The functions of each module in each system of the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.
[0070] Figure 6 This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 6 As shown, the electronic device includes a memory 601 and a processor 602. The memory 601 stores a computer program that can run on the processor 602. When the processor 602 executes the computer program, it implements the methods described in the above embodiments. The number of memories 601 and processors 602 can be one or more.
[0071] The electronic device also includes:
[0072] The communication interface 603 is used to communicate with external devices and perform data exchange and transmission.
[0073] If the memory 601, processor 602, and communication interface 603 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0074] Optionally, in a specific implementation, if the memory 601, processor 602, and communication interface 603 are integrated on a single chip, then the memory 601, processor 602, and communication interface 603 can communicate with each other through an internal interface.
[0075] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.
[0076] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.
[0077] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.
[0078] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.
[0079] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).
[0080] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.
[0081] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.
[0082] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.
[0083] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.
[0084] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).
[0085] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.
[0086] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.
[0087] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A data synchronization method, comprising: After parsing the source database log file into commands supported by the current data synchronization framework, commands belonging to the same database and having the same key are identified to obtain a first set of commands, wherein the commands are used to perform modification operations on the source data in the source database; The commands in the first command set are merged to obtain the second command set; The second set of commands is sent to the target database to synchronize the target data in the target database, wherein the source data corresponds to the target data.
2. The method according to claim 1, wherein, The commands in the first command set are merged to obtain the second command set, which includes: The commands in the first command set are merged into a set whose total number of commands is less than the total number of commands in the first command set, thus obtaining the second command set; or... The commands in the first command set are rewritten into a command string set to obtain the second command set, wherein the command string is a command form that concatenates multiple commands into one command.
3. The method according to claim 2, wherein, The commands in the first command set are merged into a set whose total number of commands is less than the total number of commands in the first command set, resulting in the second command set, which includes: Delete the second command preceding the first command in the first command set, wherein the final command determined by the first command and the second command is the first command; The first command is identified as a command in the second command set, thus obtaining the second command set.
4. The method according to claim 2, wherein, The commands in the first command set are rewritten into a set of command strings to obtain the second command set, which includes: The commands in the first command set are rewritten into one or more commands using a concatenation function to obtain a command string set; Set the set of command strings as the second set of commands.
5. The method according to claim 1, wherein, The commands that belong to the same database and process the same object are identified, resulting in a first set of commands including: Commands belonging to the same database and with the same key are cached in the same hash slot. The first set of commands is determined by the commands in the same hash slot.
6. The method according to claim 1, wherein, The source database and the target database are of the type including key-value (KV) databases.
7. The method according to any one of claims 1 to 6, wherein, The commands include at least one of the following: string commands, hash table commands, and list commands.
8. A data synchronization system, comprising: A command parser is used to parse the source database log file into commands supported by the current data synchronization framework, wherein the commands are used to perform modification operations on the source data in the source database; A command merger is used to identify commands that belong to the same database and have the same key, obtain a first command set, and merge the commands in the first command set to obtain a second command set; A command writer is used to send the second set of commands to a target database to synchronize target data in the target database, wherein the source data corresponds to the target data.
9. The system according to claim 8, wherein, The command merger includes at least one of the following: a string command merger, a hash table command merger, and a list command merger.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1-7.
11. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1-7.
12. A computer program product comprising computer instructions, wherein the computer instructions, when executed by a processor, implement the method of any one of claims 1-7.