Data synchronization method, apparatus, medium, and electronic device

Abstract

Embodiments of the present application disclose a data synchronization method, device, medium and electronic equipment. The method comprises: obtaining synchronization configuration information of a to-be-synchronized database, and determining target synchronization data in the to-be-synchronized database according to a data synchronization range in the synchronization configuration information; determining synchronization associated data associated with the target synchronization data according to a foreign key selection operation acting on the target synchronization data; generating a data synchronization task according to the target synchronization data and the synchronization associated data; and publishing the data synchronization task to distributed database nodes for each distributed database node to preempt and execute the data synchronization task. In the data synchronization process, the embodiments of the present application synchronize the association relationship between data in the to-be-synchronized database, thereby ensuring the data consistency of the synchronization data.

Description

Technical Field

[0001] This application relates to the field of databases, and more particularly to a data synchronization method, apparatus, medium, and electronic device. Background Technology

[0002] In the era of big data, the application value of data is being increasingly recognized, making databases for organizing, storing, and managing data a key focus of industry research. Databases include relational databases and non-relational databases. Relational databases like MySQL are typically used to process structured data, while non-relational databases like Elasticsearch are more often used for processing and storing large volumes of irregular data. To improve data retrieval efficiency, data from relational databases can be synchronized to non-relational databases.

[0003] Currently, batch synchronization of data from relational databases to non-relational databases mainly relies on the data synchronization interface provided by the non-relational databases. This synchronization method cannot preserve the relationships between data in relational databases, making it difficult to guarantee data consistency. Summary of the Invention

[0004] This application provides a data synchronization method, apparatus, medium, and electronic device. By synchronizing the relationships between data in the database to be synchronized during the data synchronization process, the data consistency of the synchronized data is guaranteed.

[0005] In a first aspect, embodiments of this application provide a data synchronization method, executed by a data synchronization system, the method comprising:

[0006] Obtain the synchronization configuration information of the database to be synchronized, and determine the target synchronization data in the database to be synchronized based on the data synchronization range in the synchronization configuration information;

[0007] Based on the foreign key selection operation applied to the target synchronization data, determine the synchronization association data associated with the target synchronization data;

[0008] A data synchronization task is generated based on the target synchronization data and the synchronization association data;

[0009] The data synchronization task is published to the distributed database nodes, allowing each of the distributed database nodes to preempt and execute the data synchronization task.

[0010] Secondly, embodiments of this application provide a data synchronization device configured in a data synchronization system, the device comprising:

[0011] The target synchronization data determination module is used to obtain the synchronization configuration information of the database to be synchronized, and determine the target synchronization data in the database to be synchronized according to the data synchronization range in the synchronization configuration information.

[0012] The synchronization association data determination module is used to determine the synchronization association data associated with the target synchronization data based on the foreign key selection operation performed on the target synchronization data;

[0013] The data synchronization task generation module is used to generate a data synchronization task based on the target synchronization data and the synchronization-related data.

[0014] The data synchronization task publishing module is used to publish the data synchronization task to the distributed database nodes, so that each of the distributed database nodes can compete for and execute the data synchronization task.

[0015] Thirdly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data synchronization method as described in embodiments of this application.

[0016] Fourthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data synchronization method as described in embodiments of this application.

[0017] The technical solution provided in this application embodiment obtains the synchronization configuration information of the database to be synchronized, and determines the target synchronization data in the database to be synchronized according to the data synchronization range in the synchronization configuration information; determines the synchronization association data associated with the target synchronization data according to the foreign key selection operation applied to the target synchronization data; generates a data synchronization task based on the target synchronization data and the synchronization association data; and publishes the data synchronization task to the distributed database nodes for each distributed database node to compete for and execute the data synchronization task. This application embodiment provides users with a personalized data synchronization solution. Users can select the target synchronization data by determining the data synchronization range according to actual business needs, and select the synchronization association data by selecting the foreign key of the target synchronization data, making data synchronization more flexible. In the data synchronization process, this application embodiment synchronizes the synchronization association data associated with the target synchronization data, ensuring the data consistency and integrity of the synchronized data. Attached Figure Description

[0018] Figure 1 This is a flowchart of a data synchronization method provided in Embodiment 1 of this application;

[0019] Figure 2 This is a flowchart of another data synchronization method provided in Embodiment 2 of this application;

[0020] Figure 3 This is a flowchart of yet another data synchronization method provided in Embodiment 3 of this application;

[0021] Figure 4 This is a schematic diagram of the structure of a data synchronization device provided in Embodiment 4 of this application;

[0022] Figure 5 This is a schematic diagram of the structure of an electronic device provided in Embodiment Six of this application. Detailed Implementation

[0023] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present application, not the entire structure.

[0024] Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of these steps can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the steps can be rearranged. The process can be terminated when its operation is complete, but may also have additional steps not included in the figures. The process can correspond to a method, function, procedure, subroutine, subroutine, etc.

[0025] Example 1

[0026] Figure 1 This is a flowchart of a data synchronization method provided in Embodiment 1 of this application. This embodiment is applicable to synchronizing data from a relational database to a non-relational database such as Elasticsearch. The method can be executed by the data synchronization device provided in this embodiment, which can be implemented in software and / or hardware and integrated into an electronic device running this system.

[0027] like Figure 1 As shown, the data synchronization method includes:

[0028] S110. Obtain the synchronization configuration information of the database to be synchronized, and determine the target synchronization data in the database to be synchronized according to the data synchronization range in the synchronization configuration information.

[0029] In this context, the database to be synchronized refers to the database storing data to be synchronized. In one optional embodiment, the database to be synchronized is a relational database. It is known that relational databases use a structured approach, storing data in tables. Tables in a relational database can have relationships. The database to be synchronized can be determined based on actual business needs. Specifically, users can input the database identifier of the database to be synchronized into the data synchronization system through the data synchronization interaction page. The data synchronization system determines the database to be synchronized from the candidate databases based on the database identifier. The data synchronization system obtains the basic information of the data tables stored in the database to be synchronized based on the database identifier. The database identifier can be at least one of the following: database IP address, database port number, or database table name. The basic information of the data tables stored in the database to be synchronized can be at least one of the following: table field type, field index, primary key, and foreign key.

[0030] The synchronization configuration information is determined based on actual business needs. This information can be entered by the user into the data synchronization system through the data synchronization interaction page. The synchronization configuration information includes the data synchronization range. The data synchronization range is used to determine the data to be synchronized in the database to be synchronized. Optionally, the data synchronization range is determined based on the data synchronization mode. Specifically, the user can select the data synchronization mode through the data synchronization interaction page of the data synchronization system. Data synchronization modes include full synchronization mode and incremental synchronization model. The data synchronization range corresponding to the full synchronization mode is all data in the database to be synchronized; the data synchronization range corresponding to the incremental synchronization model is data belonging to a specific time period in the database to be synchronized. The incremental synchronization model is suitable when the amount of data to be synchronized is too large, and it requires determining the date field to which the data to be synchronized belongs. Views or tables in the database to be synchronized are the smallest unit of data synchronization.

[0031] The data synchronization system determines the target synchronization data in the database to be synchronized based on the data synchronization range in the synchronization configuration information. The target synchronization data refers to the data in the database to be synchronized that needs to be synchronized.

[0032] S120. Based on the foreign key selection operation applied to the target synchronization data, determine the synchronization association data associated with the target synchronization data.

[0033] As we know, a row in a database table is called a record, and a record has several attributes. The set of attributes that uniquely identifies a record is called the primary key of the table. A foreign key in a table is the primary key of another table and is used to establish relationships with other tables. Defining primary and foreign keys in a database table helps maintain the integrity of the relational database and ensures data consistency.

[0034] For example, if the data table is a student table, it includes attributes such as student ID, name, gender, and class. The student ID is the primary key of the student table, uniquely identifying each record. If the data table is a course table, it includes attributes such as course number, course name, and credits. The course number is the primary key of the course table, uniquely identifying each record.

[0035] When the data table is a grade table, it includes attributes such as student ID, course number, and grade. The student ID is no longer considered the primary key, nor is the course number; neither the student ID nor the course number can uniquely identify a record. The attribute group consisting of the student ID and course number forms the primary key of the grade table, while the student ID and course number are foreign keys.

[0036] Once the target synchronization data is determined, the data synchronization system displays the foreign keys of the target synchronization data to the user through the data synchronization interaction page, allowing the user to select foreign keys based on actual business needs. For example, the foreign key selection operation can be a user clicking on the corresponding control of the target synchronization data's foreign key on the synchronization interaction page, or a user typing in the foreign key. The type of operation for selecting foreign keys in the target synchronization data is not limited here and will be determined based on the specific circumstances.

[0037] The foreign key in the target synchronization data is used to identify other data in the database to be synchronized that is associated with the target synchronization data. Users can select the data to be synchronized from the other data associated with the target synchronization data by selecting the foreign key in the target synchronization data.

[0038] The data synchronization system determines the synchronization-related data in the database to be synchronized based on the foreign key selection operation performed on the target synchronization data. The synchronization-related data is the data in the database to be synchronized that is associated with the target synchronization data. The primary key of the synchronization-related data is consistent with the foreign key selected in the foreign key selection operation performed on the target synchronization data.

[0039] S130. Generate a data synchronization task based on the target synchronization data and the synchronization association data.

[0040] The target synchronization data and the synchronization-related data together constitute the data to be synchronized. The data synchronization system generates a data synchronization task based on the target synchronization data and the synchronization-related data.

[0041] S140. Publish the data synchronization task to the distributed database nodes, so that each of the distributed database nodes can compete for and execute the data synchronization task.

[0042] Data synchronization tasks are executed by distributed database nodes. A data synchronization task refers to synchronizing data to be synchronized to the database belonging to the distributed database node. In an optional embodiment, the distributed database node is an Elasticsearch node. The database belonging to the Elasticsearch node is the Elasticsearch database (ES library). ES is a NoSQL database that provides a distributed, multi-user, full-text search engine. It allows for easy searching and analysis of large amounts of data using a RESTful interface, offering powerful functionality and ease of use, and is one of the mainstream software in the global search field.

[0043] The data synchronization system publishes data synchronization tasks to each distributed database node. Distributed database nodes compete for these tasks, and the nodes that successfully compete for the tasks execute them, synchronizing the target data and related data to the Elasticsearch database. This application utilizes a distributed approach for data synchronization, significantly improving its efficiency.

[0044] The technical solution provided in this application embodiment obtains the synchronization configuration information of the database to be synchronized, and determines the target synchronization data in the database to be synchronized according to the data synchronization range in the synchronization configuration information; determines the synchronization association data associated with the target synchronization data according to the foreign key selection operation applied to the target synchronization data; generates a data synchronization task based on the target synchronization data and the synchronization association data; and publishes the data synchronization task to the distributed database nodes for each distributed database node to compete for and execute the data synchronization task. This application embodiment provides users with a personalized data synchronization solution. Users can select the target synchronization data by determining the data synchronization range according to actual business needs, and select the synchronization association data by selecting the foreign key of the target synchronization data, making data synchronization more flexible. In the data synchronization process, this application embodiment synchronizes the synchronization association data associated with the target synchronization data, ensuring the data consistency and integrity of the synchronized data.

[0045] In an optional embodiment, a data synchronization task is generated based on the target synchronization data and the synchronization-related data, including: counting the data volume of the target synchronization data and the synchronization-related data as the total data volume to be synchronized; determining the number of data synchronization tasks based on the total data volume to be synchronized and the single synchronization data volume in the synchronization configuration information; and determining the content of the data synchronization task based on the association relationship between the target synchronization data and the synchronization-related data.

[0046] The data synchronization system separately counts the data volume of the target synchronization data and the synchronization-related data, and calculates the sum of the data volumes of the target synchronization data and the synchronization-related data as the total data volume to be synchronized. Based on the single synchronization data volume and the total data volume to be synchronized in the synchronization configuration information, the data synchronization system splits the data to be synchronized into at least two data synchronization tasks. Specifically, based on the total data volume to be synchronized and the single synchronization data volume, the number of synchronizations required to complete the synchronization of all data to be synchronized can be determined. The data volume corresponding to the set number of synchronizations is then split into at least two data synchronization tasks. Specifically, the data volume corresponding to the set number of synchronizations is considered as the data volume to be synchronized in one data synchronization task. The set number of synchronizations is determined based on actual business needs and circumstances, and is not limited here. For example, the number of synchronizations can be determined based on the number of distributed database nodes and the performance of the distributed database nodes.

[0047] To preserve the relationship between the target synchronized data and the associated synchronized data during data synchronization, this relationship needs to be recorded in the data synchronization task. This ensures that the distributed database nodes retain the relationship during the data synchronization process. Specifically, when the distributed database node is an Elasticsearch node, the target synchronized data and the associated synchronized data are saved as a parent-child relationship document, and the table fields are adjusted to the types required by the Elasticsearch database.

[0048] The above technical solution splits the data to be synchronized into at least two data synchronization tasks. This improves data synchronization efficiency and avoids database server crashes or abnormal restarts caused by excessive data volume. Determining the content of the data synchronization tasks based on the relationship between the target data and related data allows distributed database nodes to maintain this relationship during the execution of the data synchronization tasks.

[0049] Example 2

[0050] Figure 2 This is a flowchart of another data synchronization method provided in Embodiment 2 of this application. This embodiment further optimizes the above embodiment. Specifically, before publishing the data synchronization task to the distributed database node, the method further includes: determining the task configuration parameters of the data synchronization task according to the execution setting operation applied to the data synchronization task; wherein, the task configuration parameters include: task execution time and data synchronization interval; generating task configuration information according to the task configuration parameters and the single synchronization data volume; associating the task configuration information with the data synchronization task to instruct the distributed database node to execute the data synchronization task according to the task configuration information.

[0051] like Figure 2 As shown, the data synchronization method includes:

[0052] S210. Obtain the synchronization configuration information of the database to be synchronized, and determine the target synchronization data in the database to be synchronized according to the data synchronization range in the synchronization configuration information.

[0053] S220. Based on the foreign key selection operation applied to the target synchronization data, determine the synchronization association data associated with the target synchronization data.

[0054] S230. Generate a data synchronization task based on the target synchronization data and the synchronization association data.

[0055] S240. Determine the task configuration parameters of the data synchronization task according to the execution setting operation applied to the data synchronization task.

[0056] The data synchronization system determines the task configuration parameters of the data synchronization task based on the execution settings applied to the data synchronization task.

[0057] The task configuration parameters determine the execution method of the data synchronization task. Specific configuration parameters may include the task execution time and the data synchronization interval. The task execution time refers to the start time of the data synchronization task; the data synchronization interval refers to the time interval between two data synchronization operations performed by the distributed database nodes. Optionally, the data synchronization task can be a scheduled synchronization task, and the task execution time can be determined based on the cron expression input to the data synchronization system.

[0058] The execution setting operation applied to the data synchronization task can, for example, be a user clicking the control corresponding to the execution method item of the data synchronization task on the synchronization interaction page, or typing the execution method item of the data synchronization task. The type of operation applied to the execution setting operation of the data synchronization task is not limited here and will be determined according to the actual situation. The execution setting operation of the data synchronization task is actually the operation by which the user configures the execution method parameters for the data synchronization task.

[0059] S250. Generate task configuration information based on the task configuration parameters and the amount of data synchronized in a single instance.

[0060] The single-synchronization data volume is used to limit the amount of data synchronized by a distributed database node in a single operation. The single-synchronization data volume can be determined based on the storage unit of the database to be synchronized. If the database uses pages as its basic storage unit, the single-synchronization data volume can be the number of pages synchronized in a single operation. The data synchronization system generates task configuration information based on the task configuration parameters and the single-synchronization data volume. The task configuration information corresponds to the data synchronization task, and each data synchronization task has its own corresponding task configuration information. Data synchronization tasks corresponding to the same data to be synchronized can have the same task configuration information.

[0061] S260. Associate the task configuration information with the data synchronization task.

[0062] Associating task configuration information with data synchronization tasks allows distributed database nodes, after preempting a data synchronization task, to obtain the associated task configuration information and execute the task accordingly. Specifically, during task execution, a distributed database node can query and synchronize data of the same size as the target database for a single synchronization operation. After completing one synchronization cycle, it waits for the data synchronization interval before querying and synchronizing data of the same size again, until the data synchronization task is complete.

[0063] S270. Publish the data synchronization task to the distributed database nodes, so that each of the distributed database nodes can compete for and execute the data synchronization task.

[0064] The technical solution provided in this application determines the task configuration parameters of a data synchronization task based on the execution setting operation applied to the data synchronization task. These task configuration parameters include: task execution time and data synchronization interval. Task configuration information is generated based on the task configuration parameters and the amount of data synchronized in a single operation. The task configuration information is then associated with the data synchronization task to instruct the distributed database nodes to execute the data synchronization task according to the task configuration information. This application improves the flexibility of data synchronization by setting the amount of data synchronized in a single operation and the data synchronization interval. It controls the flow of data during the process, ensuring data synchronization efficiency without affecting the performance of the database server in providing external business services. It avoids excessive synchronization pressure on the database server due to large amounts of synchronized data and high-concurrency queries and writes, which could impact normal business operations.

[0065] In an optional embodiment, after publishing the data synchronization task to the distributed database nodes, the method further includes: obtaining the task execution status of each of the distributed database nodes, determining the synchronization failure task based on the task execution status, and republishing the synchronization failure task to the distributed database nodes until the synchronization failure task is successfully executed.

[0066] The data synchronization system obtains the task execution status of each distributed database node. Optionally, the data synchronization system can obtain the task execution status of each distributed database node through its task execution logs. These task execution logs are generated by the distributed database nodes and record key information such as the start and end times of the data synchronization tasks, the number of synchronized records, the synchronization time, whether the task was successful, and other synchronization log data.

[0067] The task execution status of distributed database nodes can be pushed to the data synchronization system by the distributed database nodes, or the data synchronization system can pull the task execution status from the distributed data nodes at set time intervals. The method by which the data synchronization system obtains the task execution status of the distributed database nodes is not limited here; it should be determined based on the specific circumstances.

[0068] If the task execution status obtained by the data synchronization system includes instances of failed synchronization tasks, the system collects these failed tasks and republishes them until they are successfully executed. Optionally, after identifying failed synchronization tasks, the system displays a notification message indicating the failure on the data synchronization interaction page.

[0069] The above technical solution can promptly detect data synchronization anomalies by monitoring the task execution status of each distributed database node and republishing failed synchronization tasks. This avoids situations where data synchronization tasks cannot be completed due to distributed database node downtime or network fluctuations, thus ensuring data integrity.

[0070] Example 3

[0071] Figure 3 This is a flowchart of another data synchronization method provided in Embodiment 3 of this application. This embodiment further optimizes the above embodiments. Specifically, after publishing the data synchronization task to the distributed database nodes, the method further includes: receiving task execution requests sent by the distributed database nodes; determining target database nodes for executing the data synchronization tasks among the distributed database nodes according to the request initiation time of each task execution request and the number of data synchronization tasks; allocating the data synchronization tasks to each target database node, and establishing an association between the target database nodes and the data synchronization tasks.

[0072] like Figure 3 As shown, the data synchronization method includes:

[0073] S310. Obtain the synchronization configuration information of the database to be synchronized, and determine the target synchronization data in the database to be synchronized according to the data synchronization range in the synchronization configuration information.

[0074] S320. Based on the foreign key selection operation applied to the target synchronization data, determine the synchronization association data associated with the target synchronization data.

[0075] S330. Generate a data synchronization task based on the target synchronization data and the synchronization association data.

[0076] S340. Publish the data synchronization task to the distributed database nodes.

[0077] The data synchronization system publishes data synchronization tasks to the distributed database nodes. Upon receiving the message about the data synchronization task, the distributed database nodes preempt the data synchronization task.

[0078] S350: Receive the task execution request sent by the distributed database node.

[0079] The task execution request is generated by a distributed database node, and each task execution request corresponds to a specific distributed database node. The task execution request is used to request a data synchronization task. In response to receiving a message announcing a data synchronization task, the distributed database node generates a task execution request and sends it to the data synchronization system. The data synchronization system then receives the task execution request from the distributed database node.

[0080] S360. Based on the request initiation time of each task execution request and the number of data synchronization tasks, determine the target database node for executing the data synchronization tasks among the distributed database nodes.

[0081] The request initiation time refers to the time when a distributed database node generates a task execution request. There is a one-to-one correspondence between the request initiation time and the task execution request; each task execution request has a corresponding initiation time. The request initiation time can reflect the resource conditions of the distributed database nodes to some extent. It is known that after a data synchronization task is published, distributed data nodes with good network conditions and abundant resources will be the first to generate and send task execution requests to the data synchronization system.

[0082] The data synchronization system sorts task execution requests based on their initiation time. Specifically, it sorts the requests in chronological order, placing those initiated earlier at the top.

[0083] Select a range of task execution requests from the sorted task execution requests as target task execution requests. Identify the distributed database node corresponding to the target task execution request as the target database node, which will then execute the data synchronization task. The range can be determined based on the number of data synchronization tasks. Optionally, the range can be the same as the number of data synchronization tasks.

[0084] S370. Assign the data synchronization task to each of the target database nodes and establish the association between the target database nodes and the data synchronization task.

[0085] The data synchronization system assigns data synchronization tasks to target database nodes and associates these tasks with the corresponding distributed database nodes based on the task assignments. It also records the association between the target database nodes and the data synchronization tasks, ensuring that the data synchronization tasks are traceable and facilitating their management.

[0086] Optionally, after the data synchronization system assigns a data synchronization task to the target database node, the target database distributed node acquires a read-write lock on the data synchronization task to prevent the data synchronization task from being executed repeatedly. The read-write lock is a special type of spin lock; when the read-write lock is in a write-locked state, all threads attempting to acquire the lock will be blocked until the lock is unlocked.

[0087] This application embodiment determines the target database node for executing data synchronization tasks among distributed database nodes based on the request initiation time of each task execution request and the number of data synchronization tasks. Data synchronization tasks are then assigned to each target database node, establishing a relationship between the target database node and the data synchronization tasks. This ensures that the data synchronization tasks are traceable, facilitates their management, and prevents them from being executed repeatedly.

[0088] Example 4

[0089] Figure 4 This is a schematic diagram of a data synchronization device provided in Embodiment 4 of this application. This embodiment is applicable to synchronizing data from a relational database to a non-relational database such as Elasticsearch. The device can be implemented by software and / or hardware and can be integrated into electronic devices such as smart terminals.

[0090] like Figure 4As shown, the device may include: a target synchronization data determination module 410, a synchronization association data determination module 420, a data synchronization task generation module 430, and a data synchronization task publishing module 440.

[0091] The target synchronization data determination module 410 is used to obtain the synchronization configuration information of the database to be synchronized, and determine the target synchronization data in the database to be synchronized according to the data synchronization range in the synchronization configuration information.

[0092] Synchronous association data determination module 420 is used to determine the synchronous association data associated with the target synchronous data based on the foreign key selection operation applied to the target synchronous data;

[0093] The data synchronization task generation module 430 is used to generate a data synchronization task based on the target synchronization data and the synchronization-related data.

[0094] The data synchronization task publishing module 440 is used to publish the data synchronization task to the distributed database nodes, so that each of the distributed database nodes can compete for and execute the data synchronization task.

[0095] The technical solution provided in this application embodiment obtains the synchronization configuration information of the database to be synchronized, and determines the target synchronization data in the database to be synchronized according to the data synchronization range in the synchronization configuration information; determines the synchronization association data associated with the target synchronization data according to the foreign key selection operation applied to the target synchronization data; generates a data synchronization task based on the target synchronization data and the synchronization association data; and publishes the data synchronization task to the distributed database nodes for each distributed database node to compete for and execute the data synchronization task. This application embodiment provides users with a personalized data synchronization solution. Users can select the target synchronization data by determining the data synchronization range according to actual business needs, and select the synchronization association data by selecting the foreign key of the target synchronization data, making data synchronization more flexible. In the data synchronization process, this application embodiment synchronizes the synchronization association data associated with the target synchronization data, ensuring the data consistency and integrity of the synchronized data.

[0096] Optionally, the data synchronization task generation module 430 includes: a total data volume determination submodule, used to count the data volume of the target synchronization data and the synchronization-related data as the total data volume to be synchronized; a data synchronization task quantity determination submodule, used to determine the quantity of the data synchronization tasks based on the total data volume to be synchronized and the single synchronization data volume in the synchronization configuration information; and a data synchronization task content determination submodule, used to determine the content of the data synchronization tasks based on the association relationship between the target synchronization data and the synchronization-related data.

[0097] Optionally, the apparatus further includes: a task configuration parameter determination module, configured to determine the task configuration parameters of the data synchronization task based on the execution setting operation applied to the data synchronization task before publishing the data synchronization task to the distributed database node; wherein the task configuration parameters include: task execution time and data synchronization interval; a task configuration information generation module, configured to generate task configuration information based on the task configuration parameters and the single synchronization data volume; and a task configuration information association module, configured to associate the task configuration information with the data synchronization task to instruct the distributed database node to execute the data synchronization task according to the task configuration information.

[0098] Optionally, the apparatus further includes: a synchronization failure task determination module, configured to obtain the task execution status of each of the distributed database nodes after the data synchronization task is published to the distributed database nodes, and determine the synchronization failure task based on the task execution status; and a synchronization failure task publishing module, configured to republish the synchronization failure task to the distributed database nodes until the synchronization failure task is successfully executed.

[0099] Optionally, the apparatus further includes: a task execution request receiving module, configured to receive a task execution request sent by the distributed database node after the data synchronization task is published to the distributed database node; a target database node determining module, configured to determine a target database node for executing the data synchronization task among the distributed database nodes based on the request initiation time of each task execution request and the number of data synchronization tasks; and an association relationship establishing module, configured to allocate the data synchronization task to each target database node and establish an association relationship between the target database node and the data synchronization task.

[0100] Optionally, the distributed database node is an Elasticsearch node, and the database to be synchronized is a relational database.

[0101] The data synchronization device provided in this embodiment of the invention can execute a data synchronization method provided in any embodiment of the invention, and has the corresponding performance modules and beneficial effects for executing a data synchronization method.

[0102] Example 5

[0103] Embodiment 5 of this application also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform a data synchronization method executed by a data synchronization system. The method includes:

[0104] Obtain the synchronization configuration information of the database to be synchronized, and determine the target synchronization data in the database to be synchronized based on the data synchronization range in the synchronization configuration information;

[0105] Based on the foreign key selection operation applied to the target synchronization data, determine the synchronization association data associated with the target synchronization data;

[0106] A data synchronization task is generated based on the target synchronization data and the synchronization association data;

[0107] The data synchronization task is published to the distributed database nodes, allowing each of the distributed database nodes to preempt and execute the data synchronization task.

[0108] Storage medium refers to any type of memory electronic device or storage electronic device. The term "storage medium" is intended to include: mounting media, such as CD-ROMs, floppy disks, or magnetic tape devices; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (e.g., hard disks or optical storage); registers or other similar types of memory elements. Storage medium may also include other types of memory or combinations thereof. Furthermore, storage medium may reside in a computer system in which a program is executed, or it may reside in a different second computer system connected to the computer system via a network (such as the Internet). The second computer system can provide program instructions to the computer for execution. The term "storage medium" can include two or more storage media that may reside in different unknowns (e.g., in different computer systems connected via a network). Storage medium may store program instructions (e.g., specifically implemented as a computer program) that can be executed by one or more processors.

[0109] Of course, the computer-executable instructions provided in the embodiments of this application are not limited to the data synchronization operations described above, but can also perform related operations in the data synchronization method provided in any embodiment of this application.

[0110] Example 6

[0111] This application provides an electronic device in embodiment six, which may integrate the data synchronization device provided in this application embodiment. The electronic device may be configured within a system or may be a device that performs some or all of the functions within the system. Figure 5 This is a schematic diagram of the structure of an electronic device provided in Embodiment Six of this application. Figure 5As shown, this embodiment provides an electronic device 500, which includes: one or more processors 520; and a storage device 510 for storing one or more programs. When the one or more programs are executed by the one or more processors 520, the one or more processors 520 implement the data synchronization method provided in this embodiment, the method including:

[0112] Obtain the synchronization configuration information of the database to be synchronized, and determine the target synchronization data in the database to be synchronized based on the data synchronization range in the synchronization configuration information;

[0113] Based on the foreign key selection operation applied to the target synchronization data, determine the synchronization association data associated with the target synchronization data;

[0114] A data synchronization task is generated based on the target synchronization data and the synchronization association data;

[0115] The data synchronization task is published to the distributed database nodes, allowing each of the distributed database nodes to preempt and execute the data synchronization task.

[0116] Of course, those skilled in the art will understand that the processor 520 also implements the technical solutions of the data synchronization method provided in any embodiment of this application.

[0117] Figure 5 The electronic device 500 shown is merely an example and should not impose any limitations on the performance and scope of use of the embodiments of this application.

[0118] like Figure 5 As shown, the electronic device 500 includes a processor 520, a storage device 510, an input device 530, and an output device 540; the number of processors 520 in the electronic device can be one or more. Figure 5 Taking a processor 520 as an example; the processor 520, storage device 510, input device 530, and output device 540 in the electronic device can be connected via a bus or other means. Figure 5 Taking the connection via bus 550 as an example.

[0119] The storage device 510, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and module units, such as the program instructions corresponding to the data synchronization method in the embodiments of this application.

[0120] Storage device 510 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for performance; the data storage area may store data created based on terminal usage. Furthermore, storage device 510 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, storage device 510 may further include memory remotely located relative to processor 520, and these remote memories can be connected via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0121] Input device 530 can be used to receive input digital, character, or voice information, and to generate key signal inputs related to user settings and performance control of the electronic device. Output device 540 may include electronic devices such as a display screen and a speaker.

[0122] The data synchronization device, medium, and electronic device provided in the above embodiments can execute the data synchronization method provided in any embodiment of this application, and have the corresponding performance modules and beneficial effects for executing the method. Technical details not described in detail in the above embodiments can be found in the data synchronization method provided in any embodiment of this application.

[0123] Note that the above are merely preferred embodiments and the technical principles employed in this application. Those skilled in the art will understand that this application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the appended claims.

Claims

1. A data synchronization method, characterized by, The method is executed by a data synchronization system, and comprises: obtaining synchronization configuration information of a database to be synchronized, and determining target synchronization data in the database to be synchronized according to a data synchronization range in the synchronization configuration information; determining synchronization associated data associated with the target synchronization data according to a foreign key selection operation acting on the target synchronization data; the foreign key selection operation is a click operation of a user on a foreign key corresponding control of the target synchronization data in a synchronization interaction page or is a keying operation of a foreign key of the target synchronization data; generating a data synchronization task according to the target synchronization data and the synchronization associated data; publishing the data synchronization task to distributed database nodes for each of the distributed database nodes to preempt and execute the data synchronization task; wherein, in response to receiving a message of data synchronization task publishing, each of the distributed database nodes generates a task execution request and sends the task execution request to the data synchronization system; the data synchronization system determines a target database node for executing the data synchronization task in the distributed database nodes according to a request initiation time of each of the task execution requests and a quantity of the data synchronization tasks, and executes the data synchronization task by the target database node; the request initiation time is used to reflect resource conditions of the distributed database nodes; wherein, generating the data synchronization task according to the target synchronization data and the synchronization associated data comprises: counting data quantities of the target synchronization data and the synchronization associated data as a total data quantity to be synchronized; determining a quantity of the data synchronization tasks based on the total data quantity to be synchronized and a single synchronization data quantity in the synchronization configuration information; and determining content of the data synchronization tasks based on an association relationship of the target synchronization data and the synchronization associated data.

2. The method of claim 1, wherein, Before publishing the data synchronization task to the distributed database nodes, the method further comprises: determining a task configuration parameter of the data synchronization task according to an execution setting operation acting on the data synchronization task; wherein, the task configuration parameter comprises a task execution time and a data synchronization interval; generating task configuration information according to the task configuration parameter and the single synchronization data quantity; associating the task configuration information to the data synchronization task to instruct the distributed database nodes to execute the data synchronization task according to the task configuration information.

3. The method of claim 1, wherein, After publishing the data synchronization task to the distributed database nodes, the method further comprises: obtaining a task execution situation of each of the distributed database nodes, and determining a synchronization failure task according to the task execution situation; re-publishing the synchronization failure task to the distributed database nodes until the synchronization failure task is executed successfully.

4. The method of claim 1, wherein, After publishing the data synchronization task to the distributed database nodes, the method further comprises: allocating the data synchronization task to each of the target database nodes, and establishing an association relationship between the target database nodes and the data synchronization task.

5. The method of claim 1, wherein, The distributed database nodes are Elasticsearch nodes, and the database to be synchronized is a relational database.

6. A data synchronization apparatus, characterized by comprising: The device is configured in a data synchronization system, and comprises: a target synchronization data determination module configured to acquire synchronization configuration information of a database to be synchronized, and determine target synchronization data in the database to be synchronized according to a data synchronization range in the synchronization configuration information; a synchronization associated data determination module configured to determine synchronization associated data associated with the target synchronization data according to a foreign key selection operation acting on the target synchronization data; the foreign key selection operation is a click operation of a user on a foreign key corresponding control of the target synchronization data in a synchronization interaction page or a key input operation of a foreign key of the target synchronization data; a data synchronization task generation module configured to generate a data synchronization task according to the target synchronization data and the synchronization associated data; a data synchronization task publishing module configured to publish the data synchronization task to distributed database nodes, so that each of the distributed database nodes occupies and executes the data synchronization task; wherein, in response to receiving a message of data synchronization task publishing, each of the distributed database nodes generates a task execution request and sends the task execution request to the data synchronization system; the data synchronization system determines a target database node for executing the data synchronization task in the distributed database nodes according to a request initiation time of each of the task execution requests and a number of the data synchronization tasks, and executes the data synchronization task by the target database node; the request initiation time is used to reflect resource conditions of the distributed database nodes; wherein, the data synchronization task generation module comprises: a total data amount to be synchronized determination submodule configured to count data amounts of the target synchronization data and the synchronization associated data as a total data amount to be synchronized; a data synchronization task number determination submodule configured to determine a number of the data synchronization tasks based on the total data amount to be synchronized and a single synchronization data amount in the synchronization configuration information; and a data synchronization task content determination submodule configured to determine contents of the data synchronization tasks based on an association relationship between the target synchronization data and the synchronization associated data.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that The program is executed by the processor to implement the data synchronization method of any one of claims 1-5.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the data synchronization method of any one of claims 1-5.

Images