Data synchronization method, device, and medium

Abstract

The present invention relates to the field of computers, and provides a data synchronization method, a device, and a medium. The method comprises: using a synchronization system to acquire data information from different data sources to be synchronized (100); using the synchronization system to identify data information to obtain common attribute information of said different data sources (200); by means of the synchronization system and on the basis of the common attribute information, performing data synchronization between said different data sources, and detecting a data synchronization state corresponding to any data source to be synchronized (300); and when the data synchronization state is synchronization failure, giving an alert and performing data synchronization again by means of the synchronization system (400).

Description

A data synchronization method, apparatus and medium

[0001] This application claims priority to Chinese Patent Application No. 202410482917.3, filed on April 22, 2024, entitled "A Data Synchronization Method, Apparatus and Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computers, and in particular to a data synchronization method, apparatus and medium. Background Technology

[0003] As the number of customers grows and business becomes more complex, a significant demand for data processing arises. For example, in the big data field, it's common to encounter situations where a single dataset needs to be synchronized to various heterogeneous data sources. In such scenarios, the following approaches are generally used: 1) Synchronization using built-in data tools; 2) Synchronization using open-source middleware; 3) Implementing a custom synchronization system to address specific synchronization and business issues. Generally, the first two methods are sufficient for small datasets or limited scenarios. However, for large datasets with multiple data sources, only the third method can be used. However, this approach incurs substantial development and maintenance costs. Summary of the Invention

[0004] The purpose of this application is to provide a data synchronization method, apparatus, and medium that can improve data synchronization efficiency and provide timely alarms.

[0005] To achieve the above objectives, this application provides the following solution:

[0006] In a first aspect, this application provides a data synchronization method, comprising: acquiring data information from different data sources to be synchronized through a synchronization system; identifying the data information using the synchronization system to obtain common attribute information of the different data sources to be synchronized; realizing data synchronization between the different data sources to be synchronized through the synchronization system based on the common attribute information, while detecting the data synchronization status corresponding to any of the data sources to be synchronized; when the data synchronization status is synchronization failure, the synchronization system issues an alarm and re-synchronizes the data.

[0007] In a second aspect, the present invention provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the data synchronization method.

[0008] Thirdly, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data synchronization method. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 is a flowchart illustrating the data synchronization method according to one or more embodiments of this application.

[0011] Figure 2 is a schematic diagram of an example of this application.

[0012] Figure 3 is a diagram of the internal structure of a computer device. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] To make the objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0015] The data synchronization method provided in this application embodiment can be applied in the following application environments: A terminal communicates with a server via a network. A data storage system can store data that the server needs to process. The data storage system can be set up independently, integrated into a server, or placed in the cloud or on another server. The data storage system serves as the data source to be synchronized. The synchronization system can be set up on the server. The server obtains and identifies data information from different data sources to be synchronized, obtaining common attribute information of the different data sources. Then, based on the common attribute information, it realizes data synchronization between different data sources to be synchronized, while simultaneously detecting the data synchronization status corresponding to any data source to be synchronized. When the data synchronization status is "synchronization failed," an alarm is issued to the terminal to remind the user, and data synchronization is re-performed.

[0016] The terminals can be, but are not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle systems. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. Servers can be implemented using independent servers, server clusters composed of multiple servers, or cloud servers.

[0017] In an exemplary embodiment, as shown in FIG1, a data synchronization method is provided, including the following steps 100-400.

[0018] Step 100: Obtain data information from different data sources to be synchronized through the synchronization system.

[0019] In one embodiment, the data source to be synchronized refers to a relational database table used in business operations to store data, containing information such as orders, products, and payments. This information is not limited to detailed or summary data. That is, the data source to be synchronized is used to store order information, product information, and user payment information; the data source to be synchronized can be one of MySQL, SQL Server, Oracle, or NoSQL.

[0020] Step 200: The synchronization system is used to identify the data information to obtain common attribute information of different data sources to be synchronized; the common attribute information includes the data connection method, access method, and query method. In addition, the common attribute information may also include data permissions and synchronization tasks, etc.

[0021] Step 300: Based on the public attribute information, data synchronization between different data sources to be synchronized is achieved through the synchronization system, while the data synchronization status corresponding to any of the data sources to be synchronized is detected.

[0022] The synchronization system supports and is compatible with various data sources, such as MySQL and SQL Server. By connecting to these different data sources, data synchronization between them is achieved. For example, synchronizing data from MySQL to a SQL Server database table, MySQL is the data source to be synchronized, and the SQL Server database table is the corresponding target data source. Based on data source compatibility, it can implement data query and write functions for the data sources. These different data sources have both commonalities and differences in read and write operations. Therefore, the synchronization system must support each data source individually. Furthermore, for maintainability and adaptability considerations, abstraction is necessary; otherwise, implementing a separate read and write logic for each data source would be extremely complex and time-consuming.

[0023] In one embodiment, the public attribute information includes connection information of the data source to be synchronized, such as address, port, and database / table name. Based on this information, the synchronization system knows how to establish a connection with the data source and which specific table to synchronize. After establishing the connection and obtaining the database / table, the synchronization system will perform data querying from the source table according to the specific synchronization task information configured in the public attributes, such as the fields to be synchronized, the synchronization speed (how many records to synchronize each time), the number of retries after synchronization failure, and the timeout period for data synchronization (if this time is exceeded, the synchronization task will be abandoned). After processing by the synchronization system, the data is written to the target data source and the information of this task execution is updated. The process of querying the source table and writing to the target table is repeated until an exception is encountered or all data is synchronized, at which point a synchronization is considered complete.

[0024] Step 400: When the data synchronization status is synchronization failure, the synchronization system issues an alarm and simultaneously performs data synchronization again, or it can exit the data synchronization process.

[0025] In another example, as shown in Figure 2, the method further includes: obtaining the system startup status through the self-test of the synchronization system; determining whether the system startup status is normal startup; when the system startup status is normal startup, performing data synchronization between different data sources to be synchronized; when the system startup status is not normal startup, stopping data synchronization, and simultaneously checking the data in the data sources to be synchronized to determine the cause of the anomaly.

[0026] In another example, as shown in Figure 2, the method further includes: 1) detecting the data integrity status corresponding to any of the data sources to be synchronized. 2) When the data integrity status is incomplete, issuing an alarm through the synchronization system and simultaneously re-synchronizing the data, or exiting the data synchronization process. 3) When the data synchronization status is successful and the data integrity status is complete, the synchronization system stops data synchronization.

[0027] In summary, when handling heterogeneous data source synchronization, this application first extracts the common attributes of the data source, such as connection method, access method, and query method. Then, it develops an adaptation layer for these attributes to ensure compatibility. Finally, it drives the entire process through configuration, thereby improving data synchronization efficiency and reducing cost.

[0028] This application acquires and identifies data information from different data sources to be synchronized through a synchronization system, obtaining common attribute information of these data sources. Based on this common attribute information, the synchronization system achieves data synchronization between different data sources. When the data synchronization status of any data source fails, the synchronization system issues an alarm and simultaneously re-synchronizes the data. In other words, this application abstracts heterogeneous data sources by setting up a synchronization system to reduce adaptation costs. It achieves a single, compatible method to handle heterogeneous differences, thus avoiding the need for separate process design and customized implementation for each data source. Overall, this significantly reduces cost while enabling autonomous adaptation to new data sources, achieving a one-time, reusable solution. Furthermore, the synchronization system in this application can promptly issue warnings when data synchronization fails, alerting relevant personnel.

[0029] In another embodiment, a computer device is provided, which may be a database, and its internal structure diagram is shown in Figure 3. The computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is connected to the system bus via the I / O interfaces. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device stores transactions to be processed. The I / O interfaces of the computer device are used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a data processing method.

[0030] In another embodiment, a computer device is also provided, including a memory and a processor for a computer program stored in the memory and executable on the processor, the processor executing the computer program to implement the steps in the above-described data synchronization method embodiments.

[0031] In another embodiment, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps in the above-described data synchronization method embodiments.

[0032] In another embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described data synchronization method embodiments.

[0033] It should be noted that the object information (including but not limited to object device information, object personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the object or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0034] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0035] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0036] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data synchronization method, wherein, The data synchronization method includes: Data information is obtained from different data sources to be synchronized through the synchronization system; The synchronization system is used to identify the data information in order to obtain the common attribute information of different data sources to be synchronized; Based on the public attribute information, the synchronization system realizes data synchronization between different data sources to be synchronized, and simultaneously detects the data synchronization status of any of the data sources to be synchronized. When the data synchronization status is "synchronization failed", the synchronization system issues an alarm and simultaneously attempts to resynchronize the data.

2. The data synchronization method according to claim 1, wherein, The public attribute information includes the data connection method, access method, and query method.

3. The data synchronization method according to claim 1, wherein, The data source to be synchronized is used to store order information, product information, and user payment information.

4. The data synchronization method according to claim 1, wherein, The data source to be synchronized is one of MySQL, SQL Server, Oracle, or NoSQL.

5. The data synchronization method according to claim 1, wherein, The data synchronization method further includes: The system startup status is obtained through the self-test of the synchronization system; Determine whether the system startup status is normal startup; When the system is in normal startup state, data synchronization is performed between different data sources to be synchronized. When the system startup status is not normal startup, data synchronization is stopped, and the data in the data source to be synchronized is checked to determine the cause of the anomaly.

6. The data synchronization method according to claim 1, wherein, The data synchronization method further includes: Detect the data integrity status corresponding to any of the aforementioned data sources to be synchronized; When the data integrity status is incomplete, an alarm is issued through the synchronization system and data synchronization is re-performed. When the data synchronization status is "synchronization successful" and the data integrity status is "data complete", the synchronization system stops data synchronization.

7. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the data synchronization method according to any one of claims 1-6.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein, When executed by a processor, the computer program implements the steps of the data synchronization method according to any one of claims 1-6.

Images