Data synchronization method and device, electronic equipment and computer program product

Abstract

The invention discloses a data synchronization method and device, electronic equipment and a computer program product, and relates to the big data technology, and the method comprises the steps: determining a data synchronization mode according to the type of an interface for receiving to-be-synchronized data when the to-be-synchronized data is received in a local park; performing initialization processing on each piece of data in the to-be-synchronized data according to the data synchronization mode, so that each piece of data comprises a target field; after the to-be-synchronized data is successfully written in the local park, synchronizing the to-be-synchronized data to the target park according to the data synchronization mode and the target field; when the data synchronization mode is a concurrent mode, the target field comprises a timestamp field, and when the data synchronization mode is a batch mode, the target field comprises the timestamp field and a deletion field. According to the method, the consistency and accuracy of the data between the two parks can be guaranteed, and the data processing reliability is improved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data synchronization method, device, electronic device and computer program product. Background Art

[0002] In data storage systems, dual-campus architecture has become a common setting to ensure data disaster recovery. Its purpose is to quickly switch access applications to another park to maintain business continuity when one park fails due to a failure or maintenance. However, data synchronization and consistency assurance under the dual-campus architecture has become a difficult technical challenge.

[0003] Traditional data synchronization mechanisms have the problem of poor data consistency when dealing with complex and changing situations. For example, in a common data synchronization method, after Park A completes the data update and writes it to the message middleware, when synchronizing data with Park B, if the message middleware suddenly fails, Park B may only obtain part of the data, which will cause the difference in the data reading results of the two parks and destroy the data consistency; for example, if the same data is updated twice, and the order in which the update information enters the message middleware is reversed, that is, the later updated data is written to the target park first, while the earlier updated data is written later, then the data finally stored in the target park is not the latest data. In this scenario, the data consistency is poor, and when processing data in the two parks, there is a problem of low data processing reliability. Summary of the invention

[0004] The present application provides a data synchronization method, device, electronic device and computer program product, which can ensure the consistency of dual-park data and improve data processing reliability.

[0005] In a first aspect, the present application provides a data synchronization method, comprising:

[0006] When the local park receives the data to be synchronized, determining the data synchronization mode according to the type of the interface receiving the data to be synchronized;

[0007] Initialize each piece of data in the data to be synchronized according to the data synchronization method so that each piece of data includes a target field;

[0008] When the data to be synchronized is successfully written in the local park, the data to be synchronized is synchronized to the target park according to the data synchronization method and the target field; when the data synchronization method is a concurrent mode, the target field includes a timestamp field, and when the data synchronization method is a batch mode, the target field includes a timestamp field and a deletion field.

[0009] In a second aspect, the present application provides a data synchronization device, the device comprising:

[0010] A mode determination module, used to determine the data synchronization mode according to the type of interface receiving the data to be synchronized when the data to be synchronized is received in the local park;

[0011] A data processing module, used for initializing each piece of data in the data to be synchronized according to the data synchronization mode, so that each piece of data includes a target field;

[0012] A data synchronization module is used to synchronize the data to be synchronized to the target park according to the data synchronization method and the target field after the data to be synchronized is successfully written in the local park; when the data synchronization method is a concurrent mode, the target field includes a timestamp field, and when the data synchronization method is a batch mode, the target field includes a timestamp field and a deletion field.

[0013] In a third aspect, the present application further provides an electronic device, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data synchronization method described in any embodiment of the present application.

[0017] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data synchronization method described in any embodiment of the present application when executed.

[0018] In a fifth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the data synchronization method described in any embodiment of the present application.

[0019] The data synchronization solution provided in the embodiment of the present application, when receiving the data to be synchronized in the local park, first determines the adapted data synchronization method according to the interface type, and then initializes each data according to the synchronization method, so that each data contains a target field related to the synchronization method. When the data is successfully written in the local park, the synchronization method and the target field are combined to synchronize the data to the target park. In concurrent mode, the timestamp field in the target field can accurately mark the data order and version, effectively avoiding data conflicts; in batch mode, the timestamp field and the delete field work together to maintain the correctness of the data version and ensure the accurate execution of the deletion operation. Through the solution provided in this embodiment, whether it is dealing with message middleware failures or dealing with the situation where the order of data updates is reversed, the consistency and accuracy of the data between the two parks can be guaranteed, and the reliability of data processing can be improved.

[0020] It should be noted that the above computer instructions may be stored in whole or in part on a computer-readable storage medium, wherein the computer-readable storage medium may be packaged together with the processor of the data synchronization device, or may be packaged separately from the processor of the data synchronization device, which is not limited in this application.

[0021] The description of the second, third, fourth and fifth aspects of the present application can refer to the detailed description of the first aspect; and the beneficial effects of the description of the second, third, fourth and fifth aspects can refer to the beneficial effect analysis of the first aspect, which will not be repeated here.

[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description.

[0023] It is understandable that before using the technical solutions disclosed in the embodiments of this application, the type, scope of use, and usage scenarios of the personal information involved in this application should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 It is a flowchart of a data synchronization method provided by an embodiment of the present application;

[0026] Figure 2 is another flowchart of the data synchronization method provided in an embodiment of the present application;

[0027] Figure 3 This is another flowchart of the data synchronization method provided by the embodiment of the present application;

[0028] Figure 4 is a structural diagram of a data synchronization device provided in an embodiment of the present application;

[0029] Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the present application will be clearly and completely described below in conjunction with the drawings in the present embodiment. Obviously, the described embodiment is only a part of the embodiment of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only the parts related to the present application, rather than all structures, are shown in the accompanying drawings.

[0033] Figure 1A flow chart of a data synchronization method provided in an embodiment of the present application is provided. This embodiment is applicable to the case of synchronizing data in a dual-park architecture. The method can be executed by a data synchronization device, which can be implemented in the form of hardware and / or software and integrated in an electronic device that executes the method. Preferably, the electronic device in the embodiment of the present application can be a server, or a computer device, etc.

[0034] refer to Figure 1 The data synchronization method of this embodiment includes but is not limited to the following steps:

[0035] S110. When the local park receives the data to be synchronized, determine the data synchronization mode according to the type of the interface for receiving the data to be synchronized.

[0036] In a dual-park processing architecture, the local park refers to a data park at a local location. It can also be understood that when there are two parks, A and B, for an application, if Park A is specified as the local park, the application will give priority to writing to Park A when writing data, and will only write to Park B when the writing to Park A fails. After that, the data synchronization program will synchronize the data from Park A to Park B. In this example, Park B is the target park described later.

[0037] The data to be synchronized refers to the related data set that needs to be synchronized (maintaining data consistency) between different parks (such as the local park and the target park). These data may come from business operations, such as the company's sales records, customer information updates, inventory changes, etc. Because the business needs to achieve data consistency in multiple data storage areas, synchronization is required.

[0038] The interface type indicates the different access methods used by data to enter the local park. Common ones include online interfaces (corresponding to real-time interactive data input, such as real-time input of data generated by online transactions into the system) and batch import interfaces (importing a batch of data into the system in a specific time period, such as companies regularly uploading financial report data in batches). Data of different interface types often have different characteristics and need to be adapted to different data processing and synchronization methods.

[0039] The data synchronization method is a processing mode that allows data to be consistent between different parks based on the characteristics and system requirements of the data to be synchronized. This solution provides two modes: concurrent mode (suitable for data synchronization scenarios with high real-time requirements and multiple data can be processed simultaneously) and batch mode (suitable for scenarios where batch data is synchronized in a certain order). Different types of data are synchronized in different ways.

[0040] When determining the data synchronization method according to the interface type, when the local park receives the data to be synchronized, it must first identify which interface the data enters through. If it is transmitted through an online interface, such as the data generated by customer orders, payments, and other operations in the real-time online business system, the system determines that its interface type is an online interface, and then determines to use the concurrent mode as the data synchronization method; if the data is entered through a batch import interface, such as employee attendance data and sales statistics data imported by enterprises in batches on a daily, weekly, monthly, etc. basis, the system determines that it is a batch import interface and determines to use the batch mode for data synchronization. By accurately selecting the appropriate data synchronization method according to the interface type, the concurrent mode is used for real-time data transmitted through the online interface, which can fully utilize system resources to process multiple data at the same time, quickly synchronize data to the target park, avoid data accumulation and delay, and meet the rapid response requirements of real-time business; and the batch mode is used for batch imported data, which is synchronized in an orderly manner in a reasonable order, and can also efficiently complete the synchronization of a large amount of data in the batch processing scenario, which improves the overall synchronization speed of different types of data between the local park and the target park.

[0041] In a preferred implementation, determining the data synchronization mode according to the type of interface receiving the data to be synchronized can be implemented by the following steps a) to c):

[0042] a) Add a verification timestamp flag to the data to be synchronized.

[0043] A space is reserved in the data structure for placing a check timestamp flag. This flag can be a binary bit or other simple data representation form, and its purpose is to be used to carry key information related to data synchronization in the future. For example, in an enterprise's information management system, when sales order data, customer information data, etc. need to be synchronized, the system will add this check timestamp flag to each such data to be synchronized, and record its specific synchronization attribute related information.

[0044] b) Determine the flag value corresponding to the check timestamp flag according to the interface type.

[0045] When the interface type corresponding to the data is identified as an online interface, such as a real-time online trading system, each time a user makes a transaction, the transaction data is transmitted to the background through the online interface for synchronization processing. At this time, the system will set the value of the verification timestamp flag of the data to 1. This means that the system has clearly identified that the data comes from a real-time interactive online interface and needs to be processed in accordance with the corresponding specific synchronization method. If it is determined that the data enters the system through a batch import interface, such as a scenario where an enterprise regularly batch imports employee attendance data and financial report data, the system will assign the verification timestamp flag value of the corresponding data to 0. In this way, data from different source interfaces are distinguished, laying the foundation for the subsequent selection of an appropriate data synchronization method.

[0046] c) Determine the data synchronization method based on the flag bit value.

[0047] When the flag bit value is 1, it indicates that the data comes from the online interface, and the system will enable concurrent mode for data synchronization. In concurrent mode, the system may use multi-threading or multi-process technologies to process multiple such data at the same time to achieve fast synchronization between different storage nodes and application modules; when the flag bit value is 0, that is, the data comes from the batch import interface, the system uses batch mode to synchronize data. This mode will process and synchronize batches of data in a certain batch order.

[0048] The data synchronization method provided by this embodiment is determined according to the type of interface receiving the data to be synchronized, and different synchronization methods are used for data of different interface types. The data of the online interface adopts the concurrent mode, which can make full use of system resources, and process a large amount of real-time generated data in real time and quickly, avoiding data congestion and delay during the online interaction process. The batch mode is used for the data of the batch import interface, which can process large-scale data sets in an orderly manner, and complete the synchronization in a reasonable order and rhythm, which improves the synchronization efficiency of different types of data after entering the system as a whole, and reduces the time cost of data waiting for synchronization. By checking the timestamp flag, the data from different interface sources is clearly distinguished, so that the system can accurately select the appropriate synchronization method according to their respective characteristics. The solution provided by this embodiment can better adapt to complex and diverse business scenarios, while also reducing data errors, losses and other problems caused by inappropriate synchronization methods, and improving the accuracy of data synchronization.

[0049] S120 , initializing each piece of data in the data to be synchronized according to the data synchronization method, so that each piece of data includes a target field.

[0050] The initialization process is used to perform an initial preparation operation on each piece of data to be synchronized, with the purpose of making each piece of data contain the target fields required in the subsequent data synchronization process.

[0051] The target field represents a specific data field that is added or already exists in each piece of data after initialization processing in order to meet the data synchronization requirements. The target fields are different in different data synchronization modes. For example, there is a timestamp field in concurrent mode, and there are timestamp fields and deletion fields in batch mode. These fields carry relevant information that is important for data synchronization operations.

[0052] Specifically, the process of initializing each piece of data to be synchronized may be: checking whether each piece of data written contains a delete field (DeleteFlag), if not, adding a delete field to the data and initializing it to 0; and also including checking whether each piece of data written contains a timestamp field (TimeFlag), if not, adding a timestamp field, and writing the corresponding timestamp at the time when the data is received; if yes, updating the corresponding timestamp according to the time when the data is received. The currently provided process of initializing each piece of data to be synchronized does not take into account the data synchronization method, that is, as long as there is written data, the delete field and timestamp field of each piece of data are checked and updated.

[0053] Since the target fields used by different data synchronization methods are different, in order to improve the initialization efficiency, this embodiment considers initializing each piece of data in the synchronization data according to the data synchronization method, so that each piece of data includes the target field. When the synchronization method is concurrent mode, the target field includes a timestamp field, that is, for the current batch of data to be synchronized, it is only necessary to check whether each piece of data written contains a timestamp field. If not, add a timestamp field and write the corresponding timestamp of the time when the data is received; if yes, update the corresponding timestamp according to the time when the data is received. When the data synchronization method is batch mode, the target field includes a timestamp field and a delete field, and it is necessary to check and initialize the delete field for each piece of data, as well as check and update the timestamp field.

[0054] In this embodiment, initialization processing is performed to add target fields to each piece of data according to the data synchronization method, which helps to accurately determine key information such as the order and timeliness of the data during the synchronization process. Whether in concurrent mode or batch mode, data can be processed correctly based on these fields, reducing data synchronization errors and ensuring data consistency between the local park and the target park.

[0055] S130. When the data to be synchronized is successfully written in the local park, the data to be synchronized is synchronized to the target park according to the data synchronization mode and the target field.

[0056] In this embodiment, when the data to be synchronized is successfully written in the local park, a flag indicating successful writing is returned, without waiting for the target park to be successfully written before returning the information. The purpose of this is to synchronize data to the target park in an asynchronous synchronous manner to improve data throughput. Specifically, it can be explained as follows: in the dual-park system architecture, when data is sent to the local park for writing and storage, as long as the storage module in the local park successfully writes the received data, the system will feedback information about successful writing. Compared with the traditional mode, which has to wait until the data in the target park is also successfully written before returning a successful prompt, this solution returns success when writing in the local park, and will not cause the feedback of the entire write operation to become slow due to possible write delays in the target park (such as network transmission problems, temporary busyness of the storage device in Park B, etc., which causes slower writing), thereby improving data throughput.

[0057] In this embodiment, when the data synchronization mode is concurrent mode, the target field includes a timestamp field, and when the data synchronization mode is batch mode, the target field includes a timestamp field and a delete field.

[0058] The timestamp field is a field used to record the information of a specific point in time of the data, usually the time when the data is generated or a specific operation (such as modification, synchronization, etc.) occurs. During the data synchronization process, the timestamp can be used to determine the order and timeliness of the data, which helps to correctly process and synchronize the data, such as determining which data to synchronize first. The delete field is a target field unique to batch mode, which is mainly used to identify relevant status information such as whether the data can be deleted. When batch processing data synchronization, it may involve operations such as cleaning up some data. This field can be used to clarify which data meets the deletion conditions, which is convenient for subsequent data management in the synchronization process and the target park.

[0059] For an implementation method, please refer to Figure 2 , Figure 2 This is another flow chart of the data synchronization method provided by the embodiment of the present application. This embodiment is optimized on the basis of the above embodiment. The specific optimization is that when the data synchronization mode is concurrent mode, the specific implementation process of "synchronizing the data to be synchronized to the target park according to the data synchronization mode and the target field" is explained in detail. Figure 2 The method of this embodiment includes but is not limited to the following steps:

[0060] S210: Determine thread parameters according to the amount of concurrently processed data, and start a processing thread to process the data to be synchronized according to the thread parameters.

[0061] When processing data to be synchronized, you first need to evaluate the amount of data to be processed concurrently. For example, by analyzing the amount of data in past data synchronization tasks, or estimating the amount of data to be processed this time based on the current system load. Assume that the system needs to synchronize 1,000 rows of data to be synchronized, and based on experience and the current system resource status, it is determined that 100 rows of data can be processed simultaneously. In this embodiment, one processing thread processes one row of data to be synchronized.

[0062] Furthermore, the method of determining the thread-related parameters according to the amount of concurrently processed data is to set the thread parameter to 100, that is, to start 100 processing threads to process 100 rows of data during one data processing process; and then after the current batch of data is completed, start 100 processing threads to continue processing the next 100 rows of data, etc., to ensure efficient data processing.

[0063] S220. For the current processing thread, obtain the row key identifier of the data to be synchronized, and determine the target key value from the local park according to the row key identifier.

[0064] In a database, a row key is a key value that uniquely identifies a row of data in a table. It is usually a keyword used to uniquely identify a row of data. For example, in a database table, a row key may be the value of a column. Assuming that the data to be synchronized is stored in a table-like structure, each row has a unique identifier as a row key identifier, and the processing thread will read this identifier.

[0065] Determine the target key value from the local campus based on the obtained row key identifier. This may involve searching for the key value corresponding to the row key identifier in the data storage of the local campus. For example, in the key-value storage system of the local campus, search for the corresponding actual data value (target key value) based on the row key identifier. This target key value may be business data associated with the row key identifier, such as customer information, order details, etc.

[0066] S230. Synchronize the row key identifier and the corresponding target key value in each data to be synchronized to the target park based on the timestamp field.

[0067] After obtaining the row key identifier and the corresponding target key value, the processing thread will synchronize the row key identifier and the corresponding target key value in each data to be synchronized to the target park based on the timestamp field in the data. The timestamp field usually records time information such as the creation time or the last modification time of the data.

[0068] Assuming that the timestamp field of the data records the last modification time of the data, the processing thread will synchronize the data to the target campus in the order of the timestamp. For example, the data rows with earlier timestamps will be synchronized first to ensure that the order and time logic of the data in the target campus are correct. During the synchronization process, the row key identifier and the target key value will be sent to the target campus together, and the corresponding data storage or update operations will be performed in the target campus.

[0069] It should be noted that the solution provided by this embodiment will not use the key value corresponding to the current row key identifier to synchronize data to the target park after obtaining the row key identifier of the data to be synchronized in the processing thread; instead, the target key value is first determined from the local park based on the row key identifier, and then the row key identifier and the corresponding target key value in each data to be synchronized are synchronized to the target park. This method can avoid the target park data being in an intermediate state of update ("intermediate state of update" can be interpreted as, when new data comes to be written, the target park may be processing part of the data update received previously, and the data may be in an unstable state that is neither an old state nor a completely updated state.), ensuring that the data of the target park is highly consistent with the final data of the local park at all times, and improving the reliability and stability of the data of the entire data storage system between different parks. .

[0070] In a preferred implementation, the above step S230 can be implemented as follows: determine whether the current row key identifier in the target park has corresponding key value data; if so, obtain the first timestamp corresponding to the key value data and the second timestamp corresponding to the target key value, and synchronize the data to be synchronized corresponding to the current row key identifier according to the first timestamp and the second timestamp; if not, write the target key value into the row key identifier of the target park, and mark the field value corresponding to the timestamp field in the target park. This method of data synchronization by comparing timestamps can ensure that the data in the target park is always the latest version, whether there are multiple data sources synchronizing data to the target park, or the data is synchronized after being updated locally, this method can avoid the situation where old data overwrites new data, and ensures the consistency of data in the entire system.

[0071] In the data storage system of the target park, the current row key identifier is used as a keyword to perform a search operation to determine whether there is corresponding key-value data for the current row key identifier in the target park; if so, a first timestamp is recorded when the target park stores the key-value data corresponding to the current row key identifier; when the target key value corresponding to the current row key identifier is obtained from the local park, a second timestamp is recorded when the local park stores the target key value, thereby synchronizing the data to be synchronized corresponding to the current row key identifier by comparing the first timestamp and the second timestamp.

[0072] When the key value data is not stored in the current row key identifier in the target park in advance, it indicates that no data has been recorded in the current row key identifier in the historical time. The target key value to be synchronized is directly written to the position of the corresponding row key identifier in the target park, and the field value corresponding to the timestamp field is marked in the data record of the target park at the same time. This timestamp field value marks the writing time of the target key value in the target park.

[0073] Preferably, in this embodiment, the data to be synchronized corresponding to the current row key identifier is synchronized according to the first timestamp and the second timestamp, including: when the first timestamp is not greater than the second timestamp, the target key value is written into the row key identifier of the target park, and the field value corresponding to the timestamp field is marked in the target park; when the first timestamp is greater than the second timestamp, data synchronization fails. This method of deciding whether to write data by comparing timestamps can ensure that the data in the target park is always the latest version, and avoid old data overwriting new data.

[0074] If the second timestamp is greater than the first timestamp, it means that the data to be synchronized is a newer version. At this time, the data to be synchronized needs to overwrite the existing data in the target park. If the first timestamp is greater than or equal to the second timestamp, it may mean that the data in the target park is already the latest or does not need to be updated. In this case, according to the set rules, data synchronization fails at this time.

[0075] The data synchronization method provided in this embodiment determines thread parameters according to the amount of concurrently processed data, and can reasonably allocate thread resources according to the current load and resource conditions of the system. This adaptive resource allocation method can avoid excessive allocation of threads leading to exhaustion of system resources and ensure the stable operation of the system during data synchronization. Synchronizing data to the target park based on the timestamp field can ensure the order and integrity of the data in the target park.

[0076] For another implementation, please refer to Figure 3 , Figure 3 This is another flow chart of the data synchronization method provided by the embodiment of the present application. This embodiment is optimized on the basis of the above embodiment. The specific optimization is that when the data synchronization mode is in batch mode, the specific implementation process of "synchronizing the data to be synchronized to the target park according to the data synchronization mode and the target field" is explained in detail.

[0077] See also Figure 3 The data synchronization method of this embodiment includes but is not limited to the following steps:

[0078] S310: Group the data to be synchronized to obtain at least one data table.

[0079] The local park may involve data processing of multiple business systems. Here, in order to facilitate batch processing and synchronize data from the local park to the target park, this embodiment analyzes the data to be synchronized to obtain multiple data tables, and then analyzes each data table one by one for processing.

[0080] For example, in a database system of an enterprise, if the data to be synchronized includes customer information, order information and product information, the data to be synchronized can be divided into a customer information table, an order information table and a product information table according to the data type. Each data table includes at least one piece of data.

[0081] S320. For each data in the current data table, obtain the field value corresponding to the delete field in each data, and divide each data into a delete queue or a write queue according to the field value corresponding to the delete field.

[0082] For each piece of data in the current data table, the delete field DeleteFlag of the data is obtained, and the field value of each DeleteFlag field is obtained to determine whether the current data is written or deleted.

[0083] One implementation method is to obtain the field value corresponding to the deleted field in each data, and to divide each data into a deletion queue or a write queue according to the field value corresponding to the deleted field, including: for the current data, determine whether the field value corresponding to the deleted field is a preset value; if it is a preset value, determine that the current data is data to be deleted, obtain the field name contained in the current data, and add the field name to the deletion queue; if it is not a preset value, obtain the field name contained in the current data and the field value corresponding to the field name, and add the field name and the corresponding field value to the write queue. This solution determines the data to be deleted by determining whether the field value is a preset value, and can accurately identify and mark the data to be deleted and the data to be written, which helps to accurately analyze each piece of data and improve the accuracy of data synchronization.

[0084] Among them, the field value of the deletion field can be pre-set in two forms: 0 and 1. When it is 0, it means that the data does not need to be deleted. At this time, the field and value of the data are put into the write queue and prepared to be copied to the target park; when it is 1, it means that the row of data is to be deleted, and the field name is added to the deletion queue, that is, when it is data to be deleted, there is no need to care about the corresponding field value.

[0085] S330, traverse all the data in the current data table, and after each data has been added to the delete queue or write queue, synchronize the data of the target park according to the data information in the delete queue and the write queue.

[0086] After each data in the data table is divided into a deletion queue or a write queue, the data to be deleted can be obtained according to the field name in the deletion queue, and the data to be written can be obtained according to the field name and the corresponding field value in the write queue, so as to synchronize the two types of data, writing and deleting, in the same data table to the target park.

[0087] S340. After data synchronization is successful, mark the field value corresponding to the timestamp field in the current data table in the target park.

[0088] After the data synchronization operation (including writing and deleting) is completed and the synchronization is confirmed to be successful, the current system time is obtained as the field value corresponding to the timestamp field.

[0089] The data synchronization method provided in this embodiment accurately divides data into corresponding queues according to the value of the deletion field in batch mode, ensuring that only data that needs to be deleted is placed in the deletion queue, and data that needs to be retained is placed in the write queue. In this way, when data synchronization is performed in the target park, data deletion and writing operations can be accurately performed, avoiding erroneous deletion or retention of data, and ensuring the consistency of target park data with source data. After successful data synchronization, a timestamp is marked to record the time when the data was last synchronized. This helps to track and verify the data synchronization history later, thereby ensuring the accuracy and traceability of the data.

[0090] Specifically, both the local park and the target park have the same functional logic. In the main park and the target park, the data processing flow includes the following components: data initialization component, data storage component, data transfer component and data synchronization component.

[0091] When data enters the park from an external source, data is first written to enter the data initialization component, where the data is initialized, which may include setting initial values, format conversion, verification or adding target fields, etc.; the initialized data enters the data storage component for storing data, which may be stored in a local database or other storage medium; the stored data will pass through the data transfer component, which is responsible for data forwarding, routing and other operations, and pass the data to the next processing link; finally, the data enters the data synchronization component to synchronize the data between the local park and the target park through the data synchronization component to ensure data consistency between the two parks. The two parks are connected through the "data synchronization" mechanism, and the data is kept consistent between the local park and the target park through the data synchronization mechanism after being processed by each component. When the data synchronization method provided in this embodiment is executed based on the system architecture provided in this embodiment, data synchronization can support two-way replication between the local park and the target park, and the data in the parks on both sides can ensure the final consistency of the data, thereby ensuring the disaster recovery capability and high reliability of the entire storage system. And the solution provided by the present invention is to return success after the data is written to the local park, and the subsequent synchronization process of the data is handed over to the data synchronization module. This ensures high concurrency and response speed of data writing.

[0092] The data synchronization method provided in this embodiment, when receiving the data to be synchronized in the local park, first determines the adapted data synchronization method according to the interface type, and then initializes each data according to the synchronization method, so that each data contains the target field related to the synchronization method. When the data is successfully written in the local park, the data to be synchronized is synchronized to the target park by combining the synchronization method and the target field. In concurrent mode, the timestamp field in the target field can accurately mark the data order and version, effectively avoiding data conflicts; in batch mode, the timestamp field and the delete field work together to maintain the correctness of the data version and ensure the accurate execution of the deletion operation. Through the solution provided in this embodiment, whether it is to deal with message middleware failures or to deal with the situation where the order of data updates is reversed, the consistency and accuracy of the data between the two parks can be guaranteed, and the reliability of data processing can be improved.

[0093] Figure 4 is a structural diagram of a data synchronization device provided in an embodiment of the present application, and the device is suitable for executing the data synchronization method provided in an embodiment of the present application. Figure 4 As shown, the device may specifically include: a mode determination module 410, a data processing module 420 and a data synchronization module 430, wherein:

[0094] A mode determination module 410 is used to determine a data synchronization mode according to an interface type for receiving the data to be synchronized when the data to be synchronized is received in the local park;

[0095] The data processing module 420 is used to initialize each piece of data in the data to be synchronized according to the data synchronization mode, so that each piece of data includes a target field;

[0096] The data synchronization module 430 is used to synchronize the data to be synchronized to the target park according to the data synchronization method and the target field after the data to be synchronized is successfully written in the local park; when the data synchronization method is a concurrent mode, the target field includes a timestamp field, and when the data synchronization method is a batch mode, the target field includes a timestamp field and a deletion field.

[0097] The data synchronization device provided in the embodiment of the present application, when receiving the data to be synchronized in the local park, first determines the adapted data synchronization method according to the interface type, and then initializes each data according to the synchronization method, so that each data contains a target field related to the synchronization method. When the data is successfully written in the local park, the data to be synchronized is synchronized to the target park by combining the synchronization method and the target field. In concurrent mode, the timestamp field in the target field can accurately mark the data order and version, effectively avoiding data conflicts; in batch mode, the timestamp field and the delete field work together to maintain the correctness of the data version and ensure the accurate execution of the deletion operation. Through the solution provided in this embodiment, whether it is dealing with message middleware failures or dealing with the situation where the order of data updates is reversed, the consistency and accuracy of the data between the two parks can be guaranteed, thereby improving the reliability of data processing.

[0098] In one embodiment, the mode determination module 410 is specifically used to add a check timestamp flag to the data to be synchronized; determine a flag value corresponding to the check timestamp flag according to the interface type; and determine the data synchronization mode according to the flag value.

[0099] In one embodiment, when the data synchronization mode is the concurrent mode, the data synchronization module 430 includes: a data processing unit, a key value determination unit and a data synchronization unit, wherein:

[0100] A data processing unit, configured to determine thread parameters according to the amount of concurrently processed data, and start a processing thread to process the data to be synchronized according to the thread parameters, wherein one processing thread processes a row of data to be synchronized;

[0101] A key value determination unit, configured to obtain, for a current processing thread, a row key identifier of the data to be synchronized, and determine a target key value from the local park according to the row key identifier;

[0102] A data synchronization unit is used to synchronize the row key identifier and the corresponding target key value in each of the data to be synchronized to the target park based on the timestamp field.

[0103] In one embodiment, the data synchronization unit is specifically used to determine whether there is corresponding key value data for the current row key identifier in the target park; if so, obtain the first timestamp corresponding to the key value data and the second timestamp corresponding to the target key value, and synchronize the data to be synchronized corresponding to the current row key identifier according to the first timestamp and the second timestamp; if not, write the target key value into the row key identifier of the target park, and mark the field value corresponding to the timestamp field in the target park.

[0104] In one embodiment, the data synchronization unit is specifically used to write the target key value into the row key identifier of the target park and mark the field value corresponding to the timestamp field in the target park when the first timestamp is not greater than the second timestamp; when the first timestamp is greater than the second timestamp, data synchronization fails.

[0105] In one embodiment, when the data synchronization mode is the batch mode, the data synchronization module 430 further includes: a data grouping unit, a field value obtaining unit and a field value marking unit, wherein:

[0106] A data grouping unit, used for grouping the data to be synchronized to obtain at least one data table, each of which includes at least one piece of data;

[0107] A field value acquisition unit, used for acquiring, for each data in the current data table, a field value corresponding to the deletion field in each data, and classifying each data into a deletion queue or a write queue according to the field value corresponding to the deletion field;

[0108] A data synchronization unit, used for traversing all the data in the current data table, and after each data has been added to the delete queue or the write queue, synchronizing the data on the target park according to the data information in the delete queue and the write queue;

[0109] The field value marking unit is used to mark the field value corresponding to the timestamp field to the current data table in the target park after the data synchronization is successful.

[0110] In one embodiment, the field value acquisition unit is specifically used to determine whether the field value corresponding to the deletion field is a preset value for the current data; if it is a preset value, the current data is determined to be data to be deleted, the field name contained in the current data is obtained, and the field name is added to the deletion queue; if it is not a preset value, the field name contained in the current data and the field value corresponding to the field name are obtained, and the field name and the corresponding field value are added to the write queue.

[0111] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the functional modules described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0112] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data comply with relevant laws, regulations and standards in relevant regions.

[0113] An embodiment of the present application also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data synchronization method described in any embodiment of the present application.

[0114] An embodiment of the present application further provides a computer-readable medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data synchronization method described in any embodiment of the present application when executed.

[0115] Reference below Figure 5 , Figure 5 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, and shows a schematic diagram of the structure of a computer system 500 suitable for implementing the electronic device in an embodiment of the present application. Figure 5 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0116] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the system 500 are also stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0117] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.

[0118] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the system of the present application are executed.

[0119] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, etc., or any suitable combination of the above.

[0120] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0121] The modules and / or units involved in the embodiments described in the present application may be implemented by software or by hardware. The modules and / or units described may also be set in a processor, for example, it may be described as: a processor includes a mode determination module, a data processing module and a data synchronization module. The names of these modules do not constitute a limitation on the modules themselves in some cases.

[0122] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by a device, the device includes: when receiving the data to be synchronized in the local park, determining the data synchronization mode according to the interface type for receiving the data to be synchronized; initializing each data in the data to be synchronized according to the data synchronization mode so that each data includes a target field; when the data to be synchronized is successfully written in the local park, synchronizing the data to be synchronized to the target park according to the data synchronization mode and the target field; when the data synchronization mode is in concurrent mode, the target field includes a timestamp field, and when the data synchronization mode is in batch mode, the target field includes a timestamp field and a delete field.

[0123] According to the technical solution of this embodiment, in concurrent mode, the timestamp field in the target field can accurately mark the data order and version, effectively avoiding data conflicts; in batch mode, the timestamp field and the delete field work together to maintain the correctness of the data version and ensure the accurate execution of the delete operation. Through the solution provided by this embodiment, whether it is to deal with message middleware failures or to deal with the situation where the data update order is reversed, the consistency and accuracy of the data between the two parks can be guaranteed, and the reliability of data processing can be improved.

[0124] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A data synchronization method, characterized in that: include: When the local park receives the data to be synchronized, determining the data synchronization mode according to the type of the interface receiving the data to be synchronized; Initialize each piece of data in the data to be synchronized according to the data synchronization method so that each piece of data includes a target field; When the data to be synchronized is successfully written in the local park, synchronizing the data to be synchronized to the target park according to the data synchronization mode and the target field; When the data synchronization mode is a concurrent mode, the target field includes a timestamp field; when the data synchronization mode is a batch mode, the target field includes a timestamp field and a delete field.

2. The data synchronization method according to claim 1, characterized in that: The determining of the data synchronization mode according to the type of the interface receiving the data to be synchronized includes: Adding a verification timestamp flag to the data to be synchronized; Determine a flag bit value corresponding to the check timestamp flag bit according to the interface type; The data synchronization mode is determined according to the flag bit value.

3. The data synchronization method according to claim 1, characterized in that: When the data synchronization mode is the concurrent mode, synchronizing the to-be-synchronized data to the target park according to the data synchronization mode and the target field includes: Determine thread parameters according to the amount of concurrently processed data, start a processing thread to process the data to be synchronized according to the thread parameters, and one processing thread processes a row of data to be synchronized; For the current processing thread, obtaining the row key identifier of the data to be synchronized, and determining the target key value from the local park according to the row key identifier; The row key identifier and the corresponding target key value in each of the data to be synchronized are synchronized to the target park based on the timestamp field.

4. The data synchronization method according to claim 3, characterized in that: The step of synchronizing the row key identifier and the corresponding target key value in each of the to-be-synchronized data to the target park based on the timestamp field includes: Determine whether there is corresponding key value data for the current row key identifier in the target park; If it exists, obtain the first timestamp corresponding to the key value data and the second timestamp corresponding to the target key value, and synchronize the to-be-synchronized data corresponding to the current row key identifier according to the first timestamp and the second timestamp; If it does not exist, the target key value is written into the row key identifier of the target park, and the field value corresponding to the timestamp field is marked in the target park.

5. The data synchronization method according to claim 4, characterized in that: The synchronizing the to-be-synchronized data corresponding to the current row key identifier according to the first timestamp and the second timestamp includes: When the first timestamp is not greater than the second timestamp, writing the target key value into the row key identifier of the target park, and marking the field value corresponding to the timestamp field in the target park; When the first timestamp is greater than the second timestamp, data synchronization fails.

6. The data synchronization method according to claim 1, characterized in that: When the data synchronization mode is the batch mode, synchronizing the to-be-synchronized data to the target park according to the data synchronization mode and the target field includes: Grouping the data to be synchronized to obtain at least one data table, each of which includes at least one piece of data; For each data in the current data table, the field value corresponding to the deletion field in each data is obtained, and each data is divided into a deletion queue or a writing queue according to the field value corresponding to the deletion field; Traversing all the data in the current data table, after each data has been added to the delete queue or the write queue, synchronizing the data of the target park according to the data information in the delete queue and the write queue; After the data synchronization is successful, the field value corresponding to the timestamp field is marked in the current data table in the target park.

7. The data synchronization method according to claim 6, characterized in that: The obtaining of the field value corresponding to the deletion field in each data, and dividing each data into a deletion queue or a write queue according to the field value corresponding to the deletion field, includes: For the current data, determine whether the field value corresponding to the deleted field is a preset value; If it is a preset value, determining that the current data is data to be deleted, obtaining the field name contained in the current data, and adding the field name to the deletion queue; If it is not a preset value, obtain the field name contained in the current data and the field value corresponding to the field name, and add the field name and the corresponding field value to the write queue.

8. A data synchronization device, characterized in that: include: A mode determination module, used to determine the data synchronization mode according to the type of interface receiving the data to be synchronized when the data to be synchronized is received in the local park; A data processing module, used for initializing each piece of data in the data to be synchronized according to the data synchronization mode, so that each piece of data includes a target field; A data synchronization module, used for synchronizing the data to be synchronized to a target park according to the data synchronization mode and the target field after the data to be synchronized is successfully written in the local park; When the data synchronization mode is a concurrent mode, the target field includes a timestamp field; when the data synchronization mode is a batch mode, the target field includes a timestamp field and a delete field.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data synchronization method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the data synchronization method according to any one of claims 1 to 7.

Images