Data synchronization method and device, equipment, storage medium and program product

Abstract

The invention provides a data synchronization method and device, electronic equipment, a computer readable storage medium and a computer program product, and the method comprises the steps: determining N data change records of to-be-synchronized data, N being a positive integer greater than 1; based on the N data change records, detecting whether a primary key of the to-be-synchronized data is changed or not to obtain N detection results; based on the N detection results, grouping the N data change records to obtain M groups of data change records, M being a positive integer greater than 1, and M being less than N; and synchronizing the to-be-synchronized data in each group of data change records by adopting a parallel processing mode. Therefore, the data synchronization efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data synchronization method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] In the case of data synchronization, most related technologies use serial processing to synchronize data based on the time sequence of data changes, thereby avoiding primary key conflicts during data synchronization. Alternatively, in order to improve data synchronization efficiency, data synchronization is performed in parallel based on the differences in primary keys of the changed data.

[0003] However, although the serial processing method can avoid primary key conflicts during data synchronization, it will result in low data synchronization efficiency. As for the parallel processing method, although it improves data synchronization efficiency, the data change process may also involve changes in primary keys. If only the differences in the primary keys of the changed data are considered for parallel processing, primary key conflicts may occur, that is, errors may be reported during the data synchronization process, which will also lead to low data synchronization efficiency. Summary of the invention

[0004] Embodiments of the present application provide a data synchronization method, device, electronic device, and computer-readable storage medium, which can improve data synchronization efficiency.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present invention provides a data synchronization method, which includes:

[0007] Determine N data change records of the data to be synchronized, where N is a positive integer greater than 1;

[0008] Based on the N data change records, detecting whether the primary key of the data to be synchronized has been changed, and obtaining N detection results;

[0009] Based on the N detection results, the N data change records are grouped to obtain M groups of data change records, where M is a positive integer greater than 1 and M is less than N;

[0010] The data to be synchronized in each set of data change records is synchronized in parallel processing.

[0011] The present application provides a data synchronization device, the device comprising:

[0012] A determination module, used to determine N data change records of data to be synchronized, where N is a positive integer greater than 1;

[0013] A detection module, used to detect whether the primary key of the data to be synchronized is changed based on the N data change records, and obtain N detection results;

[0014] A grouping module, used for grouping the N data change records based on the N detection results to obtain M groups of data change records, where M is a positive integer greater than 1 and M is less than N;

[0015] The synchronization module is used to synchronize the data to be synchronized in each set of data change records in a parallel processing manner.

[0016] An embodiment of the present application provides an electronic device, including:

[0017] A memory for storing executable instructions;

[0018] The processor is used to implement the data synchronization method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0019] An embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will execute the data synchronization method provided by the embodiment of the present application.

[0020] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the data synchronization method provided in the embodiment of the present application.

[0021] The embodiment of the present application provides a computer program product, which includes computer executable instructions, and the computer executable instructions are stored in a computer readable storage medium. The processor of the electronic device reads the computer executable instructions from the computer readable storage medium, and the processor executes the computer executable instructions, so that the electronic device executes the data synchronization method provided in the embodiment of the present application.

[0022] The embodiments of the present application have the following beneficial effects:

[0023] First, based on the data change records, whether the primary key of the data to be synchronized has changed is detected, and then based on the detection results, the data change records are grouped, and finally the data to be synchronized in each group of data change records obtained by grouping is synchronized by parallel processing. In this way, the data change records are grouped based on whether the primary key has changed, and then each group of data change records is synchronized by parallel processing. Compared with the serial processing method in the related technology, the data synchronization efficiency is improved, and the primary key conflict problem that may occur when the parallel processing method is used in the related technology is also solved, which not only improves the success rate of data synchronization, but also further improves the efficiency of data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a schematic diagram of the architecture of a data synchronization system 100 provided in an embodiment of the present application;

[0025] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0026] Figure 3 It is a flowchart of a data synchronization method provided in an embodiment of the present application;

[0027] Figure 4 It is a schematic diagram of a data change record provided in an embodiment of the present application;

[0028] Figure 5 It is a flow chart of grouping multiple data change records to obtain multiple groups of data change records provided by an embodiment of the present application;

[0029] Figure 6 is a schematic diagram of M groups of data change records provided in an embodiment of the present application;

[0030] Figure 7 It is a flowchart of a method for optimizing data playback provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0032] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0033] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0035] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0036] 1) Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Artificial intelligence technology is an interdisciplinary subject that covers a wide range of fields, including both hardware-level and software-level technologies. Basic AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies.

[0037] 2) Client, also known as the user end, refers to the program corresponding to the server that provides local services to users. Except for some applications that can only run locally, it is generally installed on an ordinary client and needs to cooperate with the server to run, that is, it requires corresponding servers and service programs in the network to provide corresponding services. In this way, specific communication connections need to be established on the client and server sides to ensure the normal operation of the application.

[0038] 3) Primary Key is a column or a group of columns used to uniquely identify each row of records in a database table. The function of the primary key is to ensure that each record in the table can be uniquely identified, avoid data duplication and ensure data integrity.

[0039] 4) Database schema, a structural description of the database, including objects such as tables, views, indexes, stored procedures, and their attributes and relationships.

[0040] 5) Database table: A data storage entity in a database, consisting of columns and rows, used to store specific types of data and containing actual data records.

[0041] 6) Table metadata, which is metadata about the table, including table structure (column name, data type, etc.), constraints (primary key, foreign key, etc.), index and other information.

[0042] The embodiment of the present application provides a data synchronization method, which first determines N data change records of the data to be synchronized, and based on the N data change records, detects whether the primary key of the data to be synchronized has changed, and obtains N detection results; further, the N data change records are grouped based on the N detection results to obtain M groups of data change records, and finally the data to be synchronized in each group of data change records is synchronized by parallel processing. In this way, the data change records are grouped based on whether the primary key has changed, and then each group of data change records is synchronized by parallel processing. Compared with the serial processing method in the related technology, the data synchronization efficiency is improved, and the primary key conflict problem that may occur when the parallel processing method is used in the related technology is also solved, which not only improves the success rate of data synchronization, but also further improves the efficiency of data synchronization.

[0043] The data synchronization method provided in the embodiment of the present application can be executed by an electronic device, and the electronic device can be a terminal or a server, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Deliver Network), and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop, a desktop computer, a set-top box, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, and a mobile device (for example, a mobile phone, a portable music player, a personal digital assistant, a dedicated messaging device, a portable gaming device, an intelligent speaker and a smart watch), etc., but is not limited thereto. The terminal device and the server can be directly or indirectly connected by wired or wireless communication, which is not limited in the embodiment of the present application.

[0044] See also Figure 1 , Figure 1 It is an architectural diagram of a data synchronization system 100 provided in an embodiment of the present application. To realize an application scenario of data synchronization, a terminal (terminal 400 is shown as an example) is connected to a server 200 via a network 300. The network 300 may be a wide area network or a local area network, or a combination of the two. The terminal 400 is used for users to use a client 401, which is displayed on a display interface (display interface 401-1 is shown as an example). The terminal 400 and the server 200 are connected to each other via a wired or wireless network.

[0045] The terminal 400 is used to send N data change records of the data to be synchronized to the server 200;

[0046] The server 200 is used to determine N data change records of the data to be synchronized, where N is a positive integer greater than 1; based on the N data change records, detect whether the primary key of the data to be synchronized is changed to obtain N detection results; based on the N detection results, group the N data change records to obtain M groups of data change records, where M is a positive integer greater than 1 and M is less than N; and synchronize the data to be synchronized in each group of data change records by using a parallel processing method.

[0047] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. In practical applications, the electronic device can be Figure 1 The server 200 or terminal 400 shown, see Figure 2 , Figure 2 The electronic device shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 2 Various buses are labeled as bus system 440 .

[0048] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0049] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0050] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0051] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0052] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0053] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0054] A network communication module 452, used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0055] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., display screen, speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripherals and displaying content and information);

[0056] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.

[0057] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 The data synchronization device 455 stored in the memory 450 is shown, which can be software in the form of a program and a plug-in, etc., and includes the following software modules: a determination module 4551, a detection module 4552, a grouping module 4553 and a synchronization module 4554. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0058] In other embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the data synchronization device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data synchronization method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic components.

[0059] In some embodiments, the terminal or server can implement the data synchronization method provided in the embodiments of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as an instant messaging APP, a web browser APP; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0060] Based on the above description of the data synchronization system and electronic device provided by the embodiment of the present application, the data synchronization method provided by the embodiment of the present application is described below. In actual implementation, the data synchronization method provided by the embodiment of the present application can be implemented by the terminal or the server alone, or by the terminal and the server in collaboration, so that Figure 1 The server 200 in the embodiment of the present application alone executes the data synchronization method provided by the embodiment of the present application as an example for explanation. Figure 3 , Figure 3 This is a flow chart of the data synchronization method provided by the embodiment of the present application. Figure 3 The steps shown are explained.

[0061] Step 101: The server determines N data change records of data to be synchronized, where N is a positive integer greater than 1.

[0062] In actual implementation, the server receives a data synchronization request sent by the terminal for the data to be synchronized that has data changes in the database, parses the data synchronization request, and thereby determines the log record of the database used to record the data to be synchronized that has data changes in the database. Here, the database may be one or more, and the log record of the database may be pre-stored locally in the terminal, or may be obtained by the terminal from the outside world (such as the Internet). This embodiment of the application does not limit this.

[0063] It should be noted that the data to be synchronized in the N data change records of the data to be synchronized refers to the data to be synchronized that has data changes recorded in the N data change records. There may be more than one data to be synchronized, and multiple data to be synchronized may share N data change records, or one data to be synchronized may have N data change records. This is not limited in the embodiments of the present application.

[0064] In actual implementation, the process of determining N data change records of the data to be synchronized may be to extract N data change records from the log records of the database, specifically, traverse the log records of the database to obtain metadata corresponding to the log records; and analyze the metadata to obtain N data change records.

[0065] It should be noted that, in the process of traversing the log records of the database and obtaining the metadata corresponding to the log records, the log records of the database are first traversed to obtain the target information corresponding to the log records, wherein the target information is used to indicate the objects affected by the log records; based on the target information corresponding to the log records, the metadata of the target information, i.e., the metadata corresponding to the log records, is queried from the database.

[0066] The affected objects can be specific data, etc. The database usually records the operations of database structure changes, that is, data change operations, such as creating, modifying or deleting tables, columns, indexes, etc. These change records are usually stored in a special database change log table, that is, the database log record, which contains information about the operation type, change time, affected objects, etc.

[0067] The target information may be information related to the database schema and the database table, and the process of traversing the log records of the database to obtain the target information corresponding to the log records may, for example, be to extract information related to the database schema and the database table, that is, the target information, from the log records of the database, including the schema name and the table name involved. The target information is used to determine which operations have been performed and which database objects have been affected.

[0068] Then, based on the target information corresponding to the log record, the table metadata of the target information, that is, the metadata corresponding to the log record, is queried from the database. For example, based on the extracted target information such as schema and table name, the metadata of the schema and table in the database, that is, the metadata corresponding to the log record, can be obtained through the metadata table of the database management system (such as INFORMATION_SCHEMA or system table); wherein the metadata includes the structure (column name, data type), index information, constraint information (primary key, foreign key, etc.) of the table targeted by the log record, and other attributes related to the table, such as field changes in the table.

[0069] Finally, the metadata corresponding to the log record is analyzed to obtain N data change records. Figure 4 , Figure 4 is a schematic diagram of a data change record provided in an embodiment of the present application, based on Figure 4 After determining the metadata corresponding to the log record, the metadata corresponding to the log record is analyzed to obtain the following Figure 4 The 7 data change records indicated are serial numbers 1 to 7.

[0070] In this way, by traversing the log records of the database, the metadata of each log record is obtained, and then based on the metadata of each log record, multiple data change records are obtained. In this way, the comprehensiveness of the obtained data change records is guaranteed through the traversal process, and then the data change records are obtained through the metadata, which provides accuracy and efficiency in obtaining data change records.

[0071] Step 102: Based on the N data change records, whether the primary key of the data to be synchronized is changed is detected to obtain N detection results.

[0072] In actual implementation, based on N data change records, the primary keys corresponding to the data to be synchronized in the data change records are detected to obtain N detection results. The process can be to parse the data change records for each of the N data change records to obtain the original primary key of the data change record and the current primary key of the data change record; and, compare the original primary key with the current primary key to obtain the detection result of the data change record.

[0073] It should be noted that the original primary key of the data change record indicates the primary key of the data before the change, and the current primary key of the data change record indicates the primary key of the data after the change; the primary key corresponding to the data to be synchronized in the data change record may change or may not change. Therefore, the original primary key and the current primary key of each data change record will be compared to obtain the corresponding comparison results. Since there are N data change records, there are also N detection results.

[0074] For example, see Figure 4 , the original primary keys of the data change records with serial numbers 1, 2, 5, 6, and 7 are 9, 11, 2, 1, and 2 respectively, and the current primary keys of these data change records are also 9, 11, 2, 1, and 2 respectively; the original primary keys of the data change records with serial numbers 3 and 4 are 10 and 11 respectively, and the current primary keys are 9 and 10 respectively; therefore, after comparing the original primary key and the current primary key of each data change record, 7 test results are obtained.

[0075] In this way, by comparing the primary keys before and after the data change, it is detected whether the primary key of the data to be synchronized corresponding to the data change record has changed, thereby ensuring the accuracy of the process of determining whether the primary key of the data to be synchronized corresponding to the data change record has changed.

[0076] Step 103: based on the N detection results, group the N data change records to obtain M groups of data change records, where M is a positive integer greater than 1 and M is less than N.

[0077] It should be noted that the detection result of a data change record is used to indicate that the primary key of the data to be synchronized in the data change record has changed, or that the primary key of the data to be synchronized has not changed. Therefore, based on N detections, that is, based on whether the primary key of the data to be synchronized in the data change record has changed, the N data change records can be grouped to obtain M groups of data change records.

[0078] In actual implementation, for the process of grouping N data change records based on N detection results to obtain M groups of data change records, see Figure 5 , Figure 5 The present invention provides a flow chart of grouping multiple data change records to obtain multiple groups of data change records, based on Figure 5 , step 103 can be implemented by the following steps.

[0079] Step 1031, based on N detection results, divide the N data change records into a first data change record group and a second data change record group, the primary key of the data to be synchronized in the data change records in the first data change record group has changed, and the primary key of the data to be synchronized in the data change records in the second data change record group has not changed.

[0080] In actual implementation, the process of dividing N data change records into a first data change record group and a second data change record group based on N detection results may be, based on the N detection results, adding the data change record indicating that the primary key of the data to be synchronized has changed to the first data change record group; dividing the data change records to be grouped into a first category of data to be grouped and a second category of data to be grouped according to the relationship between the primary key in the data change record to be grouped and the primary key included in the first data change record group; the data change record to be grouped refers to the data change record among the N data change records that is not included in the first data change record group; adding the first category of data to be grouped to the first data change record group, and adding the second category of data to be grouped to the second data change record group.

[0081] It should be noted that the detection result indicates that the data change record indicates that the primary key of the data to be synchronized has changed, that is, the data change record indicates that the original primary key of the data to be synchronized is inconsistent with the current primary key. At this time, the data change record belongs to the first data change record group.

[0082] For example, see Figure 4 For the data change record of sequence number 3, the original primary key is 10 and the current primary key is 9. The data change record indicates that the original primary key of the data to be synchronized is inconsistent with the current primary key, that is, the primary key of the data to be synchronized has changed. The data change record of sequence number 3 belongs to the first data change record group; and for the data change record of sequence number 4, the original primary key is 11 and the current primary key is 10. The data change record indicates that the original primary key of the data to be synchronized is inconsistent with the current primary key, that is, the primary key of the data to be synchronized has changed. The data change record of sequence number 4 belongs to the first data change record group.

[0083] In actual implementation, after the first data change record group is determined, the data change records to be grouped are divided into the first category of data to be grouped and the second category of data to be grouped according to the relationship between the primary key in the data change record to be grouped and the primary key included in the first data change record group. Specifically, for each data change record to be grouped, if the primary keys included in the first data change record group are different from the primary keys in the data change records to be grouped, the data change record to be grouped is determined to be the second category of data to be grouped; if the primary key in the data change record to be grouped is the same as any primary key included in the first data change record group, the data change record to be grouped is determined to be the first category of data to be grouped.

[0084] It should be noted that the primary key included in the first data change record group refers to the primary key included in each data change record in the first data change record group, that is, the original primary key and the current primary key of each data change record. Therefore, if the primary keys included in the first data change record group are different from the primary keys in the data change records to be grouped, before determining the data change records to be grouped as the second type of data to be grouped, it is also possible to parse the first data change record group to obtain the current primary key and the original primary key of each data change record included in the first data change record group; The primary key in the data change record is compared with the current primary key and the original primary key respectively to obtain a comparison result; when the comparison result indicates that the primary key in the data change record to be grouped is the same as at least one of the current primary key and the original primary key, it is determined that the primary key in the data change record to be grouped is the same as any primary key included in the first data change record group (that is, the original primary key and the current primary key); when the comparison result indicates that the primary key in the data change record to be grouped is different from the current primary key and the original primary key, it is determined that the primary keys included in the first data change record group are all different from the primary keys in the data change records to be grouped.

[0085] Among them, for each data change record included in the first data change record group, there will be two comparison results, namely, the comparison result corresponding to the original primary key and the comparison result corresponding to the current primary key. However, since the primary key of the data change record to be grouped for comparison with the data change record has not changed, there is only one primary key. Therefore, for each data change record included in the first data change record group, when there is a comparison result in the two comparison results that indicates that the primary key of the data change record to be grouped is the same as the original primary key or the current primary key of the data change record, that is, when one of the original primary key and the current primary key of the data change record is the same as the primary key of the data change record to be grouped, then A comparison result is obtained to indicate that the primary key in the data change record to be grouped is the same as at least one of the current primary key and the original primary key; when there is no comparison result in the two comparison results that represents that the primary key of the data change record to be grouped is the same as the original primary key or the current primary key of the data change record, that is, when the original primary key and the current primary key of the data change record are not the same as the primary key of the data change record to be grouped, a comparison result is obtained to indicate that the primary key in the data change record to be grouped is different from the current primary key and the original primary key; based on this, for each data change record included in the first data change record group, although there are two comparison results, only one comparison result will be obtained.

[0086] For example, see Figure 4, firstly, the data change records in which the primary key of the data to be synchronized is changed are classified into the first data change record group, that is, the data change records of sequence numbers 3 and 4 are classified into the first data change record group, and the data change records to be grouped at this time include the data change records of sequence numbers 1, 2, 5, 6, and 7; then, for the data change record of sequence number 1, the primary key 9 of the data change record is respectively compared with the original primary key 10 and the current primary key 9 of the data change record of sequence number 3, and the primary key 9 of the data change record is respectively compared with the original primary key 11 and the current primary key 10 of the data change record of sequence number 4;

[0087] Among them, the current primary key of the data change record of sequence number 3 is the same as the primary key of the data change record of sequence number 1. Therefore, based on the data change record of sequence number 3, it is determined that among the primary keys of the data change records included in the first data change record group, there is a primary key that is the same as the primary key of the data change record to be grouped. Based on the data change record of sequence number 4, it is determined that among the primary keys of the data change records included in the first data change record group, there is no primary key that is the same as the primary key of the data change record to be grouped. In this way, the data change record group to which the data change record of sequence number 1 belongs is determined; other data change records are processed in the same way as above, and no further details are given.

[0088] It should be noted that when the original primary key and / or current primary key of the data change record in the first data change record group is the same as the primary key of the data change record to be grouped, the data change record to be grouped belongs to the first data change record group; when the original primary key and / or current primary key of the data change record does not exist in the first data change record group, and is the same as the primary key of the data change record to be grouped, the other data change records belong to the second data change record group.

[0089] Take the current primary key of the existing data change record as the same as the primary key of the data change record to be grouped as an example. If the current primary key of the existing data change record is the same as the primary key of the data change record to be grouped, due to the nature of the primary key itself, that is, the primary key is a column or a group of columns used to uniquely identify each row of records in the database table, which is used to ensure that each record in the table can be uniquely identified, avoid data duplication and ensure data integrity. Therefore, the primary key has uniqueness. Based on this, when the current primary key of the existing data change record is the same as the primary key of the data change record to be grouped, the primary key of the data change record to be grouped will be forced to change to meet the uniqueness of the primary key. In this way, the data change record to be grouped needs to be classified into the first data change record group.

[0090] It should be noted that, in the case where the original primary key of a data change record is the same as the primary key of the data change record to be grouped, it will be known in the following description that the data change records associated with the primary key will be grouped in the first data change record group, and the original primary key of the data change record is the same as the primary key of the data change record to be grouped, which means that the primary key of the data change record is associated with the primary key of the data change record to be grouped. If the data change record to be grouped belongs to the first data change record group, then in the subsequent splitting process, the data change records associated with the primary key are grouped together, so that this group of data change records is processed in sequence based on the execution order of each data change record;

[0091] If the data change record to be grouped is not assigned to the first data change record group, but is assigned to the second data change record group, then in the subsequent process, the data change record to be grouped will belong to two groups with the data change record with the same original primary key as mentioned above, and thus be processed in parallel. Since the primary key of the data change record to be grouped is the same as the original primary key of the above data change record, a primary key conflict may occur in the process of parallel processing, resulting in an error.

[0092] In this way, if there is an original primary key of a data change record that is the same as the primary key of the data change record to be grouped, the primary key of the data change record to be grouped needs to be assigned to the first data change record group, thereby avoiding the possibility of primary key conflicts and improving the success rate and efficiency of the data synchronization process.

[0093] Based on this, when the original primary key and / or current primary key of the data change record in the first data change record group is the same as the primary key of the data change record to be grouped, the data change record to be grouped needs to be assigned to the first data change record group. When the original primary key and / or current primary key of the data change record does not exist in the first data change record group, and is the same as the primary key of the data change record to be grouped, the data change record to be grouped is assigned to the second data change record group.

[0094] In this way, compared with grouping data change records based only on whether the primary key of the data change record has changed, by re-checking the primary key of the data change record, the data change records whose primary key has not been changed are grouped again. This takes into account the situation where the primary key may be forced to change in the subsequent process, avoids the situation where primary key conflicts will arise in the subsequent processing process, and improves the success rate and efficiency of the data synchronization process.

[0095] Step 1032: Determine the M groups of data change records based on the first group of data change records and the second group of data change records.

[0096] In actual implementation, the process of determining M groups of data change records based on the first data change record group and the second data change record group can be: based on whether the primary keys of the data change records included in the first data change record group are associated, the first data change record group is split to obtain a first split data change record group; wherein, there is an association between the keys of the data change records in each first split data change record group; the data change records with the same primary key in the second data change record group are merged to obtain a merged data change record group, and the merged target data change record is split to obtain a second split data change record group; based on the first split data change record group and the second split data change record group, M groups of data change records are obtained.

[0097] It should be noted that the number of the first split data change record groups can be one or more, which is not limited in the embodiments of the present application, and the primary keys of the data change records included in each first split data change record group are associated with each other.

[0098] Wherein, for the primary keys of the data change records included in the first data change record group, the primary keys here include the primary keys of the data change records that have not undergone the primary key change, and the original primary keys and the current primary keys of the data change records that have undergone the primary key change; and for the process of determining whether the primary keys are associated, for any primary key, if there are other primary keys (including other primary keys that have not been changed, the original primary keys that have undergone the primary key change, and the current primary key) that are the same as the primary key, then the two primary keys are considered to be associated; at the same time, if two primary keys are associated, and the primary key of one of the two associated primary keys is also associated with the other primary key, then the three primary keys are associated;

[0099] For example, see Figure 4Based on the above process, the data change records of sequence numbers 1, 2, 3, and 4 are divided into the first data change record group. For the data change record of sequence number 1, if the current primary key of the data change record of sequence number 3 is the same, then the primary key of the data change record of sequence number 1 is considered to be associated with the primary key of the data change record of sequence number 3; for the data change record of sequence number 2, if the original primary key of the data change record of sequence number 4 is the same, then the primary key of the data change record of sequence number 2 is considered to be associated with the primary key of the data change record of sequence number 4; for the data change record of sequence number 3, the original primary key is the same as the current primary key of the data change record of sequence number 4, and the current If the primary key of the data change record of sequence number 1 is the same as the primary key of the data change record of sequence number 1, then the primary key of the data change record of sequence number 3 is considered to be associated with the primary key of the data change record of sequence number 4, and the primary key of the data change record of sequence number 3 is also associated with the primary key of the data change record of sequence number 1; and for the data change record of sequence number 4, the original primary key is the same as the primary key of the data change record of sequence number 2, and the current one is the same as the original primary key of the data change record of sequence number 3, then the primary key of the data change record of sequence number 4 is considered to be associated with the primary key of the data change record of sequence number 2, and the primary key of the data change record of sequence number 4 is also associated with the primary key of the data change record of sequence number 3. Based on this, the primary keys of the data change records of sequence numbers 1-4 are associated with each other, so the data change records of sequence numbers 1-4 are divided into the same group, that is, the first data change record group is split to obtain a group of first split data change record groups.

[0100] Continuing with the above example, if there is still a data change record in the first data change record group, for example, the original primary key is 5 and the current primary key is 6, then the primary key of the data change record is not associated with the primary keys of the data change records with serial numbers 1-4, then the first data change record group is split to obtain 2 groups of first split data change record groups, one of which includes the data change records with serial numbers 1-4, and the other includes the data change record with the original primary key of 5 and the current primary key of 6.

[0101] In this way, the first data change record group is split, and then each group of the first split data change record group is processed in parallel in the subsequent process, thereby improving the data synchronization efficiency; at the same time, because each group of the first split data change record group will be processed in parallel in the subsequent process, the data change records associated with the primary key are divided into the same group, thereby avoiding the situation of primary key conflicts in the data synchronization process, and improving the success rate of data synchronization while ensuring the efficiency of data synchronization.

[0102] In actual implementation, the process of merging data change records with the same primary key in the second data change record group to obtain the merged data change record group can be: analyzing the data change records with the same primary key in the second data change record group to obtain the execution type of the data change record; based on the execution type of the data change record, merging the data change records with the same primary key to obtain a merged data change record; and using the merged data change record and the data change records in the second data change record group except the merged data change record as the merged data change record.

[0103] It should be noted that the execution types include creation, update, and deletion, etc.; the merge process is only performed on the data change records with the same primary key in the second data change record group, and the data change records with different primary keys cannot be merged.

[0104] For example, see Figure 4 , Figure 4 The data change records corresponding to sequence numbers 5, 6, and 7 belong to the second data change record group, and the data change records corresponding to sequence numbers 5 and 7 among these three data change records have the same primary key. Therefore, the data change records corresponding to sequence numbers 5 and 7 can be merged; that is, for the process of merging the second data change record group to obtain the merged data change records, specifically, the data change records corresponding to sequence numbers 5 and 7 are merged to obtain the merged data change record, and then the merged data change record and the data change record corresponding to sequence number 6 are used as the merged data change record.

[0105] In actual implementation, based on the execution type of the data change record, the data change records with the same primary key are merged to obtain the process of merged data change records. The process may be to traverse the data change records with the same primary key based on the execution time sequence of each data change record in the data change record with the same primary key; during the traversal process, perform the following processing on the currently traversed data change record to obtain a merged data change record: based on the execution time of the data change record, if the execution type of the data change record is deletion, delete the data change record before the data change record; based on the execution time of the data change record, if the execution type of the data change record is update, then when there is a data change record with the execution type of new creation before the data change record, delete the data change record with the execution type of new creation, and transform the execution type of the data change record with the execution type of update to new creation.

[0106] It should be noted that each data change record will carry the corresponding data change time, that is, the execution time, and the data change record before the data change record, that is, the data change record whose execution time is before the execution time of the data change record.

[0107] In actual implementation, when the execution type of the current data change record traversed is delete, the data change records with the execution type of create and update before the data change record are deleted. In this way, since the merge process is for data with the same primary key, when the execution type of the current data change record is delete, no matter the execution type of the previous data change record is create or update, the data will be deleted in the end. Therefore, these data change records are directly merged into a data change record with the execution type of delete;

[0108] When the execution type of the current data change record traversed is update, if there is a data change record with an execution type of new creation before the data change record, the data change record with the execution type of new creation will be deleted, and the execution type of the data change record with the execution type of update will be changed to new creation. In this way, since the merge processing is for data with the same primary key, when the execution type of the current data change record is update, no matter what kind of content field is to be newly created by the previous data change record with the execution type of new creation, the final content field is the content field corresponding to the data change record with the execution type of update, which has nothing to do with the content field of the data change record with the execution type of new creation. Therefore, the data change record with the execution type of new creation can be deleted; at the same time, since there is no new creation operation in front of the data change record with the execution type of update, that is, there is no old data, the update operation cannot be performed directly. Therefore, it is necessary to change the execution type of the data change record from update to new creation; based on this, these data change records are merged into a data change record with an execution operation of new creation and a content field corresponding to the content field of the previous data change record with the execution type of update.

[0109] It should be noted that if the execution type of the current data change record is new creation, no merging process will be performed.

[0110] For example, see Figure 4 , Figure 4 The data change records corresponding to sequence numbers 5 and 7 can be merged, and the execution type of the data change record corresponding to sequence number 5 is new creation, and the execution type of the data change record corresponding to sequence number 7 is update. When traversing to the data change record corresponding to sequence number 5, no processing is performed. When traversing to the data change record corresponding to sequence number 7, the data change record corresponding to sequence number 5 is deleted, and the execution type of the data change record corresponding to sequence number 7 is changed from update to new creation, thereby realizing the merging process.

[0111] In actual implementation, the process of splitting the merged target data change record to obtain the second split data change record group can be to split the merged data change record according to the principle of grouping the same primary key into one group to obtain the second split data change record group.

[0112] It should be noted that the number of the second split data change record groups can be one or more, and the embodiments of the present application do not limit this; the data change records included in each second split data change record group have the same primary key. Among them, according to the principle that the same primary key belongs to one group, the merged data change records are split to obtain the second split data change record group, that is, the data change records with the same primary key in the target data change record are grouped together to obtain the second split data change record group.

[0113] For example, see Figure 4 , continuing with the above example, the merged data change records include the merged data change records corresponding to sequence numbers 5 and 7, and the data change record corresponding to sequence number 6, and the primary key of the merged data change record is 2, and the primary key of the data change record corresponding to sequence number 6 is 1, then the merged data change records are split to obtain 2 groups of second split data change record groups, one of which is the merged data change record, and the other is the data change record corresponding to sequence number 6.

[0114] In this way, by merging data change records, the number of data change records that need to be processed is reduced, resource consumption is reduced, and data synchronization efficiency is improved.

[0115] In actual implementation, the process of obtaining M groups of data change records based on the first split data change record group and the second split data change record group can be, after obtaining the first split data change record group and the second split data change record group, summarizing the first split data change record group and the second split data change record group to obtain M groups of data change records.

[0116] For example, see Figure 6 , Figure 6 is a schematic diagram of M group data change records provided in an embodiment of the present application, based on Figure 6 , M is 3, and Figure 6 The three sets of data change records in are based on Figure 4The dashed box 601 indicates the first data change record group, including one first split data change record group, the dashed box 602 indicates the second data change record group, including two second split data change record groups, wherein 603 indicates Figure 4 The merged data change record is obtained by merging the data change records corresponding to sequence numbers 5 and 7.

[0117] Step 104: synchronize the data to be synchronized in each set of data change records in a parallel processing manner.

[0118] It should be noted that after obtaining M groups of data change records, the M groups of data change records are processed in parallel to synchronize the data to be synchronized in each group of data change records; wherein, for each group of data change records in the M groups of data change records, if the group of data change records includes multiple data change records, the multiple data change records are processed serially based on the execution time of each data change record.

[0119] In this way, the data to be synchronized in each set of data change records is synchronized in a parallel processing manner, thereby improving data synchronization efficiency.

[0120] By applying the above-mentioned embodiment of the present application, firstly, based on the data change record, whether the primary key of the data to be synchronized has changed is detected, then based on the detection result, the data change record is grouped, and finally, the data to be synchronized in each group of data change records obtained by grouping is synchronized by parallel processing. In this way, the data change records are grouped based on whether the primary key has changed, and then each group of data change records is synchronized by parallel processing. Compared with the serial processing method in the related technology, the data synchronization efficiency is improved, and the primary key conflict problem that may occur when the parallel processing method is used in the related technology is also solved, which not only improves the success rate of data synchronization, but also further improves the efficiency of data synchronization.

[0121] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0122] When synchronizing data, related technologies mostly use a serial processing method to synchronize data based on the time sequence of data changes, or use a parallel processing method to synchronize data based on the difference in primary keys of the changed data; however, the first serial processing method will result in low data synchronization efficiency, and for the second parallel processing method, since the data change process may also involve changes in primary keys, only considering the difference in primary keys of the changed data for parallel processing may cause primary key conflicts, that is, errors may be reported during the data synchronization process, which will also lead to low data synchronization efficiency.

[0123] Based on this, an embodiment of the present application provides a method for optimizing data playback, which solves the problem of synchronization suspension and inability to resume due to changes in primary keys and unique indexes while improving synchronization efficiency through parallel playback.

[0124] See also Figure 7 , Figure 7 is a flow chart of a method for optimizing data playback provided in an embodiment of the present application, based on Figure 7 , the method for optimizing data playback provided by the embodiment of the present application is implemented by steps 701 to 717. Specifically, first, a batch of database change log records are obtained. Then, the database change log records of the batch are traversed. Then, one of the database change log records is obtained, and the database schema and table of this operation are obtained. Then, the table metadata of the schema and table of the target database is obtained according to the schema and table. Then, the field changes of the record are obtained. Then, according to the table metadata and field changes, it is determined whether there is a change in the primary key of the record. Among them, if there is a primary key change, the data is stored in a separate array (the first data change record group), and the array is marked as SArray; if there is no primary key change, the primary key of the data is first obtained, and the primary key of the data whose primary key has been changed before is searched in SArray to see if it is the same as the primary key. If there is a primary key of the data whose primary key has been changed and the primary key is the same as the primary key, the data is stored in SArray. If there is no data with changed primary key whose primary key is the same as the primary key, the data is stored in a default array (the second data change record group), which is marked as DArray. Then, the data in DArray is split and merged, and the data in SArray is also split.

[0125] It should be noted that, for the process of splitting and merging data in DArray, first, the data with the same primary key is merged. Specifically, if the current data indicates a DELETE operation, the INSERT and UPDATE statements before the record are deleted. If the current data indicates an UPDATE operation, if there is an INSERT statement before, the fields of the UPDATE will overwrite the fields in the previous INSERT. If the current record is an INSERT operation, no merging process is performed. Then, after the merging process, the merged data is split based on the difference in the primary keys. For the process of splitting and merging data in SArray, specifically, the data in SArray is split directly based on the association between the primary keys. Finally, the data obtained by splitting based on SArray and DArray is processed in parallel, and the data is replayed and executed in the target database.

[0126] It should be noted that DArray and SArray are array structures. DArray is used to store log records without primary key changes, and SArray is used to store log records with primary key changes and related log records. Figure 4 , 5 As shown, the data in the examples of DArray and SArray arrays below use the serial numbers in the above table to represent the row data. As for d_ids and s_ids, d_ids is used to store tables and primary key values ​​that do not have primary key changes, and s_ids is used to store tables and primary key values ​​that have primary key changes and are associated with the primary key changes. p_batch and s_batch are formed after traversing the relevant logs. All data that can be played back in parallel is stored in p_batch, and the relevant log data with primary key and unique index changes is stored in s_batch.

[0127] like Figure 4 , 6 As shown, the execution process is as follows: traverse the list data and obtain the data with sequence number 1.

[0128] 1. Get the log with serial number 1 (data change record). If there is no primary key change, store it in DArray, record the log id, and store it in d_ids, and get DArray = [1], d_ids = [9].

[0129] 2. Get the log with serial number 2. There is no primary key change. Store it in DArray and d_ids, and get DArray = [1, 2], d_ids = [9, 11].

[0130] 3. Get the log with serial number 3 and find that the primary key has changed, and the new and old ids are 10 and 9. Then search in d_ids and find that there is data with primary key 9. Traverse the data in DArray, take out the data with primary key 9 in DArray, and store them in SArray in order, and store the data with serial number 3 in SArray. That is, the data stored in SArray is the logs with serial numbers 1 and 3; then DArray = [1, 2], d_ids = [9, 11] changes to DArray = [2], d_ids =

[11] , SArray = [1, 3], s_ids = [9, 10].

[0131] 4. Get the log with sequence number 4 and find that the primary key has changed, and the new and old ids are 11 and 10. Then search in d_ids and find that 11 exists. According to the logic of step 3, d_ids = [], DArray = [] changes to SArray = [1, 2, 3, 4], s_ids = [9, 10, 11].

[0132] 5. Get the log with serial number 5. No primary key change is found. Check whether the primary key exists in s_ids. If not, store the data in DArray. At this time, d_ids = [2], DArray = [5], SArray = [1, 2, 3, 4], s_ids = [9, 10, 11].

[0133] 6. Get the log with serial number 6. No primary key change is found. Check whether the primary key exists in s_ids. If not, store the data in DArray. At this time, d_ids = [2, 1], DArray = [5, 6], SArray = [1, 2, 3, 4], s_ids = [9, 10, 11].

[0134] 7. Get the log with serial number 7. No primary key change is found. Check whether the primary key exists in s_ids. If not, store the data in DArray. At this time, d_ids = [2, 1], DArray = [5, 6, 7], SArray = [1, 2, 3, 4], s_ids = [9, 10, 11].

[0135] 8. Traverse SArray and split it. The changes of the primary key id in s_ids are divided into one group: s_ids = [9, 10, 11]. That is, the data in SArray is a group and played back in order. s_batch = [1, 2, 3, 4].

[0136] 9. Traverse DArray and split and merge according to the table and primary key. DArray = [5, 6, 7] According to the merge rule, the data with sequence numbers 5 and 7 are finally merged. According to the split rule, 7` after the merger of [5, 7] is split from 6. Two groups of data p_batch = [7`, 6] are formed.

[0137] 10. Finally, replay each group of data in s_batch and p_batch. Each group of data in s_batch needs to be replayed in strict sequence. The grouped data in p_batch is replayed in batches.

[0138] Thus, through the technical solution of the present application, first, data is merged, the execution of playback is reduced, and the synchronization performance is improved. Second, by checking the changes of primary keys and unique indexes, the relevant data is processed to avoid data synchronization anomalies caused by parallel playback. Third, while improving the synchronization performance, data synchronization suspension is avoided.

[0139] By applying the above-mentioned embodiment of the present application, firstly, based on the data change record, whether the primary key of the data to be synchronized has changed is detected, then based on the detection result, the data change record is grouped, and finally, the data to be synchronized in each group of data change records obtained by grouping is synchronized by parallel processing. In this way, the data change records are grouped based on whether the primary key has changed, and then each group of data change records is synchronized by parallel processing. Compared with the serial processing method in the related technology, the data synchronization efficiency is improved, and the primary key conflict problem that may occur when the parallel processing method is used in the related technology is also solved, which not only improves the success rate of data synchronization, but also further improves the efficiency of data synchronization.

[0140] The following further describes an exemplary structure of the data synchronization device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the data synchronization device 455 of the memory 450 may include:

[0141] A determination module 4551 is used to determine N data change records of data to be synchronized, where N is a positive integer greater than 1;

[0142] A detection module 4552 is used to detect whether the primary key of the data to be synchronized is changed based on the N data change records, and obtain N detection results;

[0143] A grouping module 4553 is used to group the N data change records based on the N detection results to obtain M groups of data change records, where M is a positive integer greater than 1 and less than N;

[0144] The synchronization module 4554 is used to synchronize the data to be synchronized in each group of data change records in a parallel processing manner.

[0145] In some embodiments, the determination module 4551 is further used to traverse the log records of the database to obtain metadata of the log records; and analyze the metadata to obtain the N data change records.

[0146] In some embodiments, the detection module 4552 is also used to parse each data change record in the N data change records to obtain the original primary key of the data change record and the current primary key of the data change record; and, compare the original primary key with the current primary key to obtain the detection result of the data change record.

[0147] In some embodiments, the detection result of a data change record is used to indicate that the primary key of the data to be synchronized in the data change record has changed, or to indicate that the primary key of the data to be synchronized has not changed; the grouping module 4553 is also used to divide the N data change records into a first data change record group and a second data change record group based on the N detection results, wherein the primary key of the data to be synchronized in the data change records within the first data change record group has changed, and the primary key of the data to be synchronized in the data change records within the second data change record group has not changed; based on the first data change record group and the second data change record group, determine the M groups of data change records.

[0148] In some embodiments, the grouping module 4553 is also used to add the data change record indicating that the primary key of the data to be synchronized has changed to the first data change record group based on the N detection results; divide the data change records to be grouped into a first category of data to be grouped and a second category of data to be grouped according to the relationship between the primary key in the data change record to be grouped and the primary key included in the first data change record group; the data change record to be grouped refers to the data change record among the N data change records that is not included in the first data change record group; add the first category of data to be grouped to the first data change record group, and add the second category of data to be grouped to the second data change record group.

[0149] In some embodiments, the grouping module 4553 is also used to, for each data change record to be grouped, if the primary keys included in the first data change record group are different from the primary keys in the data change records to be grouped, then the data change records to be grouped are determined as the second category of data to be grouped; if the primary key in the data change record to be grouped is the same as any primary key included in the first data change record group, then the data change record to be grouped is determined as the first category of data to be grouped.

[0150] In some embodiments, the grouping module 4553 is also used to split the first data change record group based on whether the primary keys of the data change records included in the first data change record group are associated, so as to obtain a first split data change record group; wherein there is an association between the keys of the data change records in each first split data change record group; merge the data change records with the same primary key in the second data change record group to obtain a merged data change record group, and split the merged target data change records to obtain a second split data change record group; based on the first split data change record group and the second split data change record group, M groups of data change records are obtained.

[0151] In some embodiments, the grouping module 4553 is also used to analyze the data change records with the same primary key in the second data change record group to obtain the execution type of the data change record; based on the execution type of the data change record, merge the data change records with the same primary key to obtain a merged data change record; and use the merged data change record and the data change records in the second data change record group except the merged data change record as the merged data change record.

[0152] In some embodiments, the grouping module 4553 is also used to traverse the data change records with the same primary key based on the execution time sequence of each data change record in the data change records with the same primary key; during the traversal process, the following processing is performed on the currently traversed data change record to obtain a merged data change record: based on the execution time of the data change record, if the execution type of the data change record is deletion, the data change record before the data change record is deleted; based on the execution time of the data change record, if the execution type of the data change record is update, then when there is a data change record with an execution type of new creation before the data change record, the data change record with an execution type of new creation is deleted, and the execution type of the data change record with an execution type of update is converted to new creation.

[0153] In some embodiments, the grouping module 4553 is further used to split the merged data change records according to the principle that records with the same primary key belong to one group, so as to obtain a second split data change record group.

[0154] The present application embodiment provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the data synchronization method described above in the present application embodiment, for example, Figure 3 The data synchronization method is shown.

[0155] The embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the data synchronization method provided by the embodiment of the present application, for example, Figure 3 The data synchronization method is shown.

[0156] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0157] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0158] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0159] As an example, the executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0160] It should be noted that in the embodiments of the present application, it involves obtaining relevant data including log records of the database. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0161] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A data synchronization method, characterized in that: The method comprises: Determine N data change records of the data to be synchronized, where N is a positive integer greater than 1; Based on the N data change records, detecting whether the primary key of the data to be synchronized has been changed, and obtaining N detection results; Based on the N detection results, the N data change records are grouped to obtain M groups of data change records, where M is a positive integer greater than 1 and M is less than N; The data to be synchronized in each set of data change records is synchronized in parallel processing.

2. The method according to claim 1, characterized in that The determining of N data change records of the data to be synchronized includes: Traversing the log records of the database to obtain metadata of the log records; The metadata is analyzed to obtain the N data change records.

3. The method according to claim 1, characterized in that Based on the N data change records, detecting whether the primary key of the data to be synchronized is changed, and obtaining N detection results, including: For each data change record in the N data change records, the data change record is parsed to obtain an original primary key of the data change record and a current primary key of the data change record; And, the original primary key is compared with the current primary key to obtain the detection result of the data change record.

4. The method according to claim 1, characterized in that The detection result of a data change record is used to indicate that the primary key of the data to be synchronized in the data change record has changed, or indicates that the primary key of the data to be synchronized has not changed; The step of grouping the N data change records based on the N detection results to obtain M groups of data change records includes: Based on the N detection results, the N data change records are divided into a first data change record group and a second data change record group, wherein the primary key of the data to be synchronized in the data change records in the first data change record group is changed, and the primary key of the data to be synchronized in the data change records in the second data change record group is not changed; The M groups of data change records are determined based on the first data change record group and the second data change record group.

5. The method according to claim 4, characterized in that The dividing the N data change records into a first data change record group and a second data change record group based on the N detection results includes: Based on the N detection results, adding a data change record indicating that a primary key of the to-be-synchronized data has been changed to a first data change record group; According to the relationship between the primary key in the data change record to be grouped and the primary key included in the first data change record group, the data change record to be grouped is divided into a first category of data to be grouped and a second category of data to be grouped; the data change record to be grouped refers to a data change record that is not included in the first data change record group among the N data change records; The first category of data to be grouped is added to the first data change record group, and the second category of data to be grouped is added to a second data change record group.

6. The method according to claim 5, characterized in that The step of dividing the data change records to be grouped into a first category of data to be grouped and a second category of data to be grouped according to the relationship between the primary key in the data change records to be grouped and the primary key included in the first data change record group includes: For each data change record to be grouped, if the primary keys included in the first data change record group are all different from the primary keys in the data change record to be grouped, then the data change record to be grouped is determined as the second type of data to be grouped; If the primary key in the data change record to be grouped is the same as any primary key included in the first data change record group, the data change record to be grouped is determined as the first type of data to be grouped.

7. The method according to claim 4, characterized in that The determining the M groups of data change records based on the first data change record group and the second data change record group includes: Based on whether the primary keys of the data change records included in the first data change record group are associated, the first data change record group is split to obtain a first split data change record group; wherein the keys of the data change records in each first split data change record group are associated; Merging the data change records with the same primary key in the second data change record group to obtain a merged data change record group, and splitting the merged target data change record to obtain a second split data change record group; Based on the first split data change record group and the second split data change record group, M groups of data change records are obtained.

8. The method according to claim 7, characterized in that The merging of the data change records with the same primary key in the second data change record group to obtain a merged data change record group includes: Analyze the data change records with the same primary key in the second data change record group to obtain the execution type of the data change record; Based on the execution type of the data change record, the data change records with the same primary key are merged to obtain a merged data change record; The merged data change record and the data change records in the second data change record group other than the merged data change record are used as the merged data change records.

9. The method according to claim 8, characterized in that The step of merging the data change records with the same primary key based on the execution type of the data change records to obtain a merged data change record includes: Based on the execution time order of each data change record in the data change records with the same primary key, the data change records with the same primary key are traversed; During the traversal process, the following processing is performed on the currently traversed data change record to obtain the merged data change record: Based on the execution time of the data change record, if the execution type of the data change record is deletion, deleting the data change record before the data change record; Based on the execution time of the data change record, if the execution type of the data change record is update, when there is a data change record with an execution type of new creation before the data change record, the data change record with an execution type of new creation is deleted, and the execution type of the data change record with an execution type of update is changed to new creation.

10. The method according to claim 7, characterized in that The step of splitting the merged target data change record to obtain a second split data change record group includes: According to the principle that the same primary key belongs to one group, the merged data change records are split to obtain a second split data change record group.

11. A data synchronization device, Its feature is that the device comprises: A determination module, used to determine N data change records of data to be synchronized, where N is a positive integer greater than 1; A detection module, used to detect whether the primary key of the data to be synchronized is changed based on the N data change records, and obtain N detection results; A grouping module, used for grouping the N data change records based on the N detection results to obtain M groups of data change records, where M is a positive integer greater than 1 and M is less than N; The synchronization module is used to synchronize the data to be synchronized in each set of data change records in a parallel processing manner.

12. An electronic device, characterized in that: include: A memory for storing executable instructions; A processor, used to implement the data synchronization method described in any one of claims 1 to 10 when executing the executable instructions stored in the memory.

13. A computer-readable storage medium, characterized in that: Executable instructions are stored, which are used to cause the processor to execute and implement the data synchronization method described in any one of claims 1 to 10.

14. A computer program product comprising computer executable instructions, characterized in that: When the computer executable instructions are executed by a processor, the data synchronization method described in any one of claims 1 to 10 is implemented.

Images