Data synchronization method, device, electronic device and storage medium

Abstract

The embodiments of the present application provide a data synchronization method, device, electronic device, and storage medium, the method comprising: obtaining multiple business data, storing the multiple business data in a first wide table; after each business data is stored in the first wide table, synchronizing each business data stored in the first wide table to a second wide table; in the process of synchronizing each business data stored in the first wide table to the second wide table, locking the business data in the first wide table; after each business data stored in the first wide table is synchronized to the second wide table, unlocking the business data in the first wide table, and synchronizing the business data in the second wide table to the peer end. Applying the technical solution provided by the embodiments of the present application can reduce the error rate of data synchronization.

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and in particular to a data synchronization method, device, electronic device, and storage medium. Background Art

[0002] With the development of the internet industry, business systems are becoming increasingly complex, and so is the data exchanged between different terminals. To facilitate business processing, full data synchronization is the primary solution. However, the sheer volume of full data leads to a high error rate in data synchronization. Summary of the Invention

[0003] The purpose of the embodiments of the present application is to provide a data synchronization method, device, electronic device, and storage medium to reduce the error rate of data synchronization. The specific technical solution is as follows:

[0004] In a first aspect of the present application, a data synchronization method is provided, the method comprising:

[0005] Acquire multiple pieces of business data, and store the multiple pieces of business data in a first wide table;

[0006] After storing each piece of business data in the first wide table, synchronizing each piece of business data stored in the first wide table to the second wide table;

[0007] In the process of synchronizing each piece of business data stored in the first wide table to the second wide table, locking the business data in the first wide table;

[0008] After synchronizing each piece of business data stored in the first wide table to the second wide table, the business data in the first wide table is unlocked, and the business data in the second wide table is synchronized to the peer end.

[0009] Optionally, the method further includes:

[0010] In the process of storing each piece of business data in the first wide table, setting the update flag field corresponding to each piece of business data to a first flag, wherein the first flag indicates that the business data storage is incomplete;

[0011] After storing each piece of business data in the first wide table, setting the update flag field corresponding to each piece of business data to a second flag, where the second flag indicates that the business data storage is completed;

[0012] After storing each piece of business data in the first wide table, synchronizing each piece of business data stored in the first wide table to the second wide table includes:

[0013] For each piece of business data in the first wide table, when it is detected that the update identification field corresponding to the business data in the first wide table is the second identification, the business data in the first wide table is synchronized to the second wide table.

[0014] Optionally, the business data includes multidimensional data;

[0015] The step of storing the plurality of business data in the first wide table includes:

[0016] Extract the data of the target data dimension from each piece of business data; store the data extracted from each piece of business data into the first wide table; or

[0017] Extract the target data dimension and the data of the associated dimension of the target data dimension from each piece of business data; and store the data extracted from each piece of business data in the first wide table accordingly.

[0018] Optionally, each piece of business data has a unique identifier, and the method further includes:

[0019] Determining a target business identifier from business identifiers of multiple pieces of business data;

[0020] Extracting the first service data corresponding to the target service identifier from the second wide table, and extracting the second service data corresponding to the target service identifier from the peer end;

[0021] If the data of the preset key field in the first business data does not match the data of the preset key field in the second business data, an operation corresponding to the data inconsistency is performed.

[0022] Optionally, the operation corresponding to the data inconsistency includes at least one of the following:

[0023] Output alarm information;

[0024] Resynchronize the first service data in the second wide table to the peer end.

[0025] Optionally, each piece of business data has a business identifier, and the step of obtaining the plurality of pieces of business data includes:

[0026] Obtaining a data update instruction, wherein the data update instruction includes the business data to be updated and an identifier to be updated;

[0027] The step of storing the plurality of business data in the first wide table includes:

[0028] If the third service data corresponding to the to-be-updated identifier is stored in the first wide table, updating the to-be-updated service data into the third service data;

[0029] If the third service data is not stored in the first wide table, the to-be-updated identifier and the to-be-updated service data are stored in the first wide table in correspondence.

[0030] Optionally, the method further includes:

[0031] Obtaining a data deletion instruction, where the data deletion instruction includes a service identifier to be deleted, and the service identifier to be deleted corresponds to the fourth service data;

[0032] Setting the fourth business data in the first wide table to a deleted state;

[0033] synchronizing the deletion status of the fourth business data in the first wide table to the second wide table, and locking the fourth business data in the first wide table during the process of synchronizing the deletion status of the fourth business data in the first wide table to the second wide table;

[0034] After synchronizing the deletion status of the fourth business data in the first wide table to the second wide table, the fourth business data in the first wide table is unlocked, and the deletion status of the fourth business data in the second wide table is synchronized to the peer end, so that the peer end deletes the fourth business data.

[0035] In a second aspect of the present application, a data synchronization device is provided, comprising:

[0036] A storage unit, configured to obtain multiple pieces of business data and store the multiple pieces of business data in a first wide table;

[0037] a first synchronization unit, configured to synchronize each piece of business data stored in the first wide table to a second wide table after each piece of business data is stored in the first wide table;

[0038] a locking unit, configured to lock each piece of business data stored in the first wide table during synchronization of the business data to the second wide table;

[0039] The second synchronization unit is configured to unlock the business data in the first wide table after synchronizing each business data stored in the first wide table to the second wide table, and synchronize the business data in the second wide table to the peer end.

[0040] Optionally, the device further includes:

[0041] an updating unit, configured to, during the process of storing each piece of business data into the first wide table, set an update identification field corresponding to each piece of business data to a first identification, the first identification indicating that storage of the business data is not complete; and after storage of each piece of business data into the first wide table is complete, set the update identification field corresponding to each piece of business data to a second identification, the second identification indicating that storage of the business data is complete;

[0042] The first synchronization unit is specifically configured to:

[0043] For each piece of business data in the first wide table, when it is detected that the update identification field corresponding to the business data in the first wide table is the second identification, the business data in the first wide table is synchronized to the second wide table.

[0044] Optionally, the business data includes multidimensional data;

[0045] The storage unit is specifically used for:

[0046] Extract the data of the target data dimension from each piece of business data; store the data extracted from each piece of business data into the first wide table; or

[0047] Extract the target data dimension and the data of the associated dimension of the target data dimension from each piece of business data; and store the data extracted from each piece of business data in the first wide table accordingly.

[0048] Optionally, each piece of business data has a business identifier, and the device further includes:

[0049] a determination unit, configured to determine a target service identifier from service identifiers of a plurality of service data;

[0050] an extraction unit, configured to extract the first service data corresponding to the target service identifier from the second wide table, and extract the second service data corresponding to the target service identifier from the opposite end;

[0051] An execution unit is configured to execute an operation corresponding to data inconsistency if data of a preset key field in the first business data does not match data of the preset key field in the second business data.

[0052] Optionally, the operation corresponding to the data inconsistency includes at least one of the following:

[0053] Output alarm information;

[0054] Resynchronize the first service data in the second wide table to the peer end.

[0055] Optionally, each piece of business data has a business identifier, and the storage unit is specifically used to:

[0056] Obtaining a data update instruction, wherein the data update instruction includes the business data to be updated and an identifier to be updated;

[0057] If the third service data corresponding to the to-be-updated identifier is stored in the first wide table, updating the to-be-updated service data into the third service data;

[0058] If the third service data is not stored in the first wide table, the to-be-updated identifier and the to-be-updated service data are stored in the first wide table in correspondence.

[0059] Optionally, the device further includes:

[0060] an acquiring unit, configured to acquire a data deletion instruction, wherein the data deletion instruction includes a to-be-deleted service identifier, and the to-be-deleted service identifier corresponds to the fourth service data;

[0061] a setting unit, configured to set the fourth business data in the first wide table to a deleted state;

[0062] The locking unit is further configured to synchronize the deletion status of the fourth business data in the first wide table to the second wide table, and lock the fourth business data in the first wide table during the synchronization of the deletion status of the fourth business data in the first wide table to the second wide table;

[0063] The second synchronization unit is further used to unlock the fourth business data in the first wide table after synchronizing the deletion status of the fourth business data in the first wide table to the second wide table, and synchronize the deletion status of the fourth business data in the second wide table to the opposite end, so that the opposite end deletes the fourth business data.

[0064] In a third aspect of the present application, an electronic device is provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0065] Memory for storing computer programs;

[0066] The processor is configured to implement any of the above-mentioned data synchronization methods when executing a program stored in the memory.

[0067] In a fourth aspect of the implementation of the present application, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned data synchronization methods is implemented.

[0068] In a fifth aspect of the implementation of the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned data synchronization methods.

[0069] In the technical solution provided by the embodiment of the present application, a first wide table and a second wide table are set up, and multiple business data are directly stored in the first wide table. The first wide table includes business data that has been stored and business data that is being stored; each business data that has been stored in the first wide table is synchronized to the second wide table. In the process of synchronizing a piece of business data to the second wide table, the business data in the first wide table is locked. In this way, it is possible to avoid the business data being modified during the synchronization process, resulting in errors in the data synchronized to the opposite end. After the synchronization of a piece of business data to the second wide table is completed, the business data in the second wide table is synchronized to the opposite end to complete the data synchronization. In addition, the business data in the first wide table is unlocked. In this way, while ensuring that the data synchronized this time is the required data, subsequent modifications to the business data can also be achieved. It can be seen that in the embodiment of the present application, by setting the first wide table and the second wide table, and locking the business data in the first wide table that is being synchronized to the second wide table, the data synchronization error rate is effectively reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art.

[0071] Figure 1 A first flow chart of a data synchronization method provided in an embodiment of the present application.

[0072] Figure 2 A second flow chart of a data synchronization method provided in an embodiment of the present application.

[0073] Figure 3 A first flow chart of a data monitoring method provided in an embodiment of the present application.

[0074] Figure 4 A third flow chart of a data synchronization method provided in an embodiment of the present application.

[0075] Figure 5 A first flow chart of a data deletion method provided in an embodiment of the present application.

[0076] Figure 6 A block diagram of a data synchronization system provided in an embodiment of the present application.

[0077] Figure 7 A structural diagram of a data synchronization device provided in an embodiment of the present application.

[0078] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the embodiments described are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application are within the scope of protection of this application.

[0080] With the development of the internet industry, business systems are becoming increasingly complex, and the data exchanged between different terminals is also becoming increasingly complex. To facilitate business processing, full data synchronization is the primary solution. However, the volume of full data is extremely large. Under this premise, there is a high probability that business data will be modified during the synchronization process. If business data is modified during the synchronization process, the synchronization data will be incorrect. Therefore, the error rate of data synchronization in related technologies is relatively high.

[0081] In order to solve the problem of high error rate of data synchronization, the embodiment of the present application provides a data synchronization method, such as Figure 1 As shown, the method includes the following steps:

[0082] Step S11: Acquire multiple pieces of business data and store the multiple pieces of business data in a first wide table.

[0083] Step S12: After each piece of business data is stored in the first wide table, each piece of business data stored in the first wide table is synchronized to the second wide table.

[0084] Step S13: In the process of synchronizing each piece of business data stored in the first wide table to the second wide table, the business data in the first wide table is locked.

[0085] Step S14: After synchronizing each piece of business data stored in the first wide table to the second wide table, unlock the business data in the first wide table, and synchronize the business data in the second wide table to the peer end.

[0086] In the technical solution provided by the embodiment of the present application, a first wide table and a second wide table are set up, and multiple business data are directly stored in the first wide table. The first wide table includes business data that has been stored and business data that is being stored; each business data that has been stored in the first wide table is stored in the second wide table. In the process of synchronizing a piece of business data to the second wide table, the business data in the first wide table is locked. In this way, it is possible to avoid the business data being modified during the synchronization process, resulting in errors in the data synchronized to the opposite end. After synchronizing a piece of business data to the second wide table, the business data in the second wide table is synchronized to the opposite end to complete the data synchronization. In addition, the business data in the first wide table is unlocked. In this way, while ensuring that the data synchronized this time is the required data, subsequent modifications to the business data can also be achieved. It can be seen that in the embodiment of the present application, by setting the first wide table and the second wide table, and locking the business data in the first wide table that is being synchronized to the second wide table, the data synchronization error rate is effectively reduced.

[0087] In addition, the capacity of the synchronization queue is limited and can only cache a small amount of business data. In the technical solution provided in the embodiment of the present application, the second wide table is used as a transition between the first wide table and the synchronization queue, that is, for the business data stored in the first wide table, after synchronizing the business data to the second wide table, the business data in the second wide table is synchronized to the opposite end, thereby avoiding the problem of adding a large amount of business data to the synchronization queue, thereby reducing the problem of loss of business data to be synchronized due to the capacity limitation of the synchronization queue, further improving the accuracy of data synchronization, reducing the data synchronization error rate, and thus improving the efficiency of data synchronization.

[0088] In the scenario of synchronizing massive business data, the technical solution provided by the embodiment of the present application is more effective in reducing the error rate of data synchronization.

[0089] The above-mentioned data synchronization method can be applied to servers, personal computers or other electronic devices with data processing functions. For ease of understanding, the following description is based on the electronic device as the execution subject, which does not serve as a limitation. In the embodiment of the present application, the electronic device can be a B (Business) end, and the electronic device can use a database engine to store data in a relational database such as an RDMS (Relational Database Management System) database. The counterpart of the electronic device is a device that receives synchronized data, and the counterpart can be a C (Consumer) end. The counterpart can use an inverted index at the memory level to store data in a flat large wide table.

[0090] Of course, the electronic device can also use the inverted index at the memory level to store data in a flat large wide table, and the other end can also use the database engine to store data in a relational database. Alternatively, both the electronic device and the other end use the inverted index at the memory level to store data in a flat large wide table, or both the electronic device and the other end use the database engine to store data in a relational database. There is no limitation on this.

[0091] In step S11, a first wide table is set in the electronic device. The first wide table can be called a Creative table (i.e., a large wide table) and can be used to store flattened data. The electronic device can obtain multiple business data from a data source and store the obtained multiple business data in the first wide table.

[0092] Business data can be advertising business data or retrieval business data, and there is no limitation to this. Different business data has different data sources. For example, if the business data is advertising business data, the data source may be the advertising business end. Business data can include data in multiple data dimensions. The number and types of data dimensions can be the same or different for different types of business data. Furthermore, the data dimensions accessed by users vary for different types of business data. For ease of description and understanding, the data dimensions accessed by users are referred to as target data dimensions below.

[0093] Take advertising business data as an example. Advertising business data can include advertiser data (the parent data structure of the advertising campaign, involving basic account information, industry, affiliation, account opening status and other series of information), advertising campaign data (mainly describing the delivery strategy), advertising plan data (mainly describing the delivery target data), advertising creative data (mainly describing the delivered materials, which is also the data directly accessed by users) and other data dimensions. For advertising business data, the target data dimension is the advertising creative data dimension, that is, the data accessed by users is the material information on the advertising creative data, the advertising creative data is associated with advertising plan data, the advertising plan data is associated with advertising campaign data, and the advertising campaign data is associated with advertiser data. It can be seen that advertising creative data is the basic data of electronic devices and peers. Electronic devices and peers can align data based on advertising creative data, that is, electronic devices can synchronize advertising creative data to the peer.

[0094] In an optional embodiment, the data dimension that the user is interested in is the target data dimension. In this case, in step S11, the step of storing multiple business data in the first wide table can be: extracting data of the target data dimension from each business data; and storing the data extracted from each business data in the first wide table.

[0095] In another optional embodiment, the data dimensions that the user is concerned about include other data dimensions in addition to the target data dimension. The other data dimensions here can be called associated dimensions of the target data dimension, and the number of other data dimensions can be 1 or more. In this case, in step S11, the step of storing multiple business data in the first wide table can be: extracting the data of the target data dimension and the associated dimensions of the target data dimension from each business data; and storing the data extracted from each business data in the first wide table accordingly. Here, the data of the target data dimension and the associated dimensions of the target data dimension are both stored in the first wide table. It can be understood that the first wide table stores the data of the target data dimension, and the data of this target data dimension includes the data of the original target data dimension and the data of the associated dimensions.

[0096] Taking advertising business data as an example, advertising business data 1 includes advertising creative data 1 and advertising plan data 1 associated with advertising creative data 1. The electronic device can store advertising creative data 1 and advertising plan data 1 in a corresponding manner in a first wide table. The first wide table actually stores only one piece of advertising creative data 1', which includes advertising creative data 1 and advertising plan data 1.

[0097] In step S12 above, a second wide table is set up in the electronic device. The second wide table can be called the Creative_result table (i.e., the result table). The first wide table and the second wide table have the same structure, that is, the Creative table and the Creative_result table have the same structure. The data in the second wide table can be used as the final inverted basic data. The electronic device synchronizes each business data stored in the first wide table to the second wide table.

[0098] In the embodiment of the present application, the second wide table is the basis for the electronic device to query business data. Specifically, the electronic device can use a hash storage engine to query and obtain business data from the second wide table in an O(1) manner. For example, still taking advertising business data as an example, the electronic device can use a hash storage engine to query and obtain one or more advertising creative data from the second wide table in an O(1) manner.

[0099] In embodiments of the present application, electronic devices can be configured with multiple data change protocols, such as a data insertion protocol, a data update protocol, a data deletion protocol, a data monitoring protocol, and a data trigger update protocol. All data change protocols have adaptability, allowing them to be applied to both ends with inconsistent underlying storage. Furthermore, to facilitate the processing of business data, each of the aforementioned data change protocols can be abstracted into an interface, which is then used to implement each of the aforementioned data change protocols.

[0100] In an optional embodiment, the electronic device may set a data query protocol and run the data query protocol to query and obtain service data from the second wide table.

[0101] In step S13, for each piece of business data stored in the first wide table, the electronic device synchronizes the business data to the second wide table. The electronic device gradually synchronizes the entire content of the business data to the second wide table. During the synchronization process, the electronic device may lock the business data in the first wide table.

[0102] When the business data in the first wide table is locked, the electronic device cannot modify or delete the business data, thereby avoiding the problem that the data synchronized to the other end is incorrect due to the business data being modified during the synchronization process.

[0103] In the above step S14, for each piece of business data stored in the first wide table, the electronic device unlocks the business data in the first wide table after synchronizing the business data to the second wide table. In this way, the electronic device can continue to modify the business data to achieve real-time update of the business data.

[0104] In addition, for each piece of business data stored in the first wide table, after synchronizing the business data to the second wide table, the electronic device synchronizes the business data in the second wide table to the peer end, completing data synchronization between the electronic device and the peer end.

[0105] In an optional embodiment, to facilitate the storage and management of business data, the peer end may pre-set a third wide table with the same structure as the second wide table. The third wide table may also be called a Creative_result table. After receiving the business data synchronized by the electronic device, the peer end stores the business data in the third wide table.

[0106] In one optional embodiment, using advertising service data as an example, the electronic device can synchronize the advertising service data in the second wide table with the peer's advertising engine through Binlog information synchronization. The advertising engine then converts the service data into data on the peer's memory storage medium for use by the peer's users. Binlog is a database data change log, and using Binlog information synchronization can achieve master-slave synchronization between the electronic device and the peer.

[0107] In an optional embodiment, the electronic device can use MQ (Message Queue) to obtain business data from the data source and synchronize the business data in the second wide table to the peer end using MQ. Applying this embodiment can solve problems such as decoupling between the electronic device and the data source, and between the electronic device and the peer end, asynchronous messaging, and traffic peak shaving, thereby achieving a high-performance, high-availability, scalable, and eventually consistent architecture.

[0108] In one embodiment of the present application, the present application embodiment also provides a data synchronization method, such as Figure 2 As shown, the method may include steps S21 to S26, step S24 is an implementation of step S12, and steps S21, S25, and S26 are the same as steps S11, S13, and S14, and are not repeated here.

[0109] Step S22 : During the process of storing each piece of business data in the first wide table, the update flag field corresponding to each piece of business data is set to a first flag, where the first flag indicates that the business data storage is incomplete.

[0110] Step S23 : After each piece of business data is stored in the first wide table, the update flag field corresponding to each piece of business data is set to a second flag, where the second flag indicates that the business data storage is completed.

[0111] Step S24 , for each piece of business data in the first wide table, when it is detected that the update identification field corresponding to the business data in the first wide table is the second identification, the business data in the first wide table is synchronized to the second wide table.

[0112] In the technical solution provided in the embodiment of the present application, the electronic device identifies the business data in the first wide table by updating the identifier to determine whether the business data has been stored. This can effectively ensure that the business data stored in the first wide table is synchronized to the second wide table in a timely manner, and ensure that the business data synchronized to the second wide table is business data that has been stored. This further avoids the problem of synchronization data errors caused by incomplete data storage, and reduces the error rate of data synchronization.

[0113] In step S22 above, for each piece of business data, the electronic device stores the acquired business data in the first wide table. The electronic device gradually acquires the content of the business data, and therefore gradually stores the content of the business data in the first wide table. During the process of storing the acquired business data in the first wide table, the electronic device may set the update identification field corresponding to the business data to the first identification. The first identification can be represented by 0 or 1, without limitation.

[0114] When the update identification field corresponding to the business data is the first identification, the electronic device will not perform other processing on the business data in the first wide table, that is, will not synchronize the business data in the first wide table to the second wide table.

[0115] In the embodiment of the present application, the update identification field may be represented as an update_flag field. The update_flag field may be set at the end or head of each service data record in the first wide table, or may be managed by an independent register, which is not limited.

[0116] In step S23, after the electronic device stores each piece of service data in the first wide table, it can set the update flag field corresponding to the service data to the second flag. The second flag can be 0 or 1, without limitation, as long as it is different from the first flag.

[0117] For each business data in the first wide table, when the update identification field corresponding to the business data is the second identification, the electronic device processes the business data in the first wide table, that is, executes step S24 to synchronize the business data in the first wide table to the second wide table.

[0118] In an embodiment of the present application, the electronic device may set a data-triggered update protocol. The electronic device runs the data-triggered update protocol to detect whether the update identification field corresponding to each piece of service data is the second identification, and executes step S24 when detecting that the update identification field corresponding to a piece of service data is the second identification.

[0119] In the above step S24, the service data corresponding to the second identifier is the stored service data. The electronic device can obtain the service data corresponding to the second identifier from the first wide table in real time and store the obtained service data in the second wide table.

[0120] In this embodiment of the present application, the electronic device may set a data update protocol. When detecting that the update identification field corresponding to the service data is the second identification, the electronic device may run the data update protocol to execute step S24 to synchronize the service data corresponding to the second identification in the first wide table to the second wide table.

[0121] In order to ensure data synchronization between the two devices, the embodiment of the present application also provides a data monitoring method, in which each piece of business data has a business ID (Identity). To ensure the accuracy of data synchronization and monitoring, the business ID of each piece of business data is unique. Figure 3 As shown, the data monitoring method may include the following steps:

[0122] Step S31: Determine a target service identifier from the service identifiers of multiple pieces of service data.

[0123] Step S32: extract the first service data corresponding to the target service identifier from the second wide table, and extract the second service data corresponding to the target service identifier from the opposite end.

[0124] Step S33: If the data of the preset key field in the first business data does not match the data of the preset key field in the second business data, an operation corresponding to the data inconsistency is performed.

[0125] In the technical solution provided in the embodiment of the present application, the electronic device samples the business data in the second wide table and the business data synchronized to the opposite end. When it is detected that the data of the preset key field in the first business data does not match the data of the preset key field in the second business data, the operation corresponding to the data inconsistency is performed, thereby reducing the probability of data processing errors caused by data synchronization errors.

[0126] In step S31 above, each piece of business data has a business ID, which facilitates business data management. When monitoring business data, the electronic device can periodically randomly select a business ID from the business IDs of multiple business data pieces as the target business ID. The number of target business IDs can be one or more, and the specific number of target business IDs can be set according to actual needs. In one example, the number of target business IDs can be a preset percentage of the total number of business data in the second wide table. The preset percentage can be 5%, 6%, or 7%, etc.

[0127] In the above step S32, the electronic device can extract the business data corresponding to each target business ID from the second wide table as the first business data; and extract the business data corresponding to each target business ID from the business data of the other end (such as the above third wide table) as the second business data.

[0128] For each target service ID, the electronic device may compare data of preset key fields in the first service data and the second service data corresponding to the target service ID to determine whether the first service data and the second service data match.

[0129] In an embodiment of the present application, the electronic device can use a hash storage engine to extract the first business data corresponding to the target business ID from the second wide table in an O(1) manner, and use a hash storage engine to extract the second business data corresponding to the target business ID from the opposite end in an O(1) manner; the electronic device can also use other methods to extract the first business data corresponding to the target business ID from the second wide table and the opposite end, and there is no limitation on this.

[0130] In step S33, the electronic device may pre-set key fields, i.e., preset key fields. The preset key fields may be set based on actual needs. For example, using advertising business data as an example, the key fields may include the price field, location field, and duration field in the advertising creative data.

[0131] After obtaining the first business data and the second business data corresponding to each target business ID, the electronic device compares the data of the preset key fields in the first business data and the second business data corresponding to each target business ID to determine whether the data of the preset key fields of the first business data and the second business data match. If the data of the preset key fields of the first business data and the second business data are inconsistent, that is, the data of the preset key fields of the first business data and the second business data do not match, the electronic device performs the operation corresponding to the data inconsistency.

[0132] In this embodiment of the present application, the operations corresponding to data inconsistency may include at least one of the following:

[0133] 1) Output alarm information. This alarm information is used to remind users of the current data synchronization error, so that users can solve the data synchronization error problem in time and improve data synchronization efficiency.

[0134] 2) Resynchronize the first service data in the second wide table to the peer end.

[0135] In this embodiment of the present application, the target service ID may be one or more. When there are multiple target service IDs, the service data whose data in the preset key fields does not match as determined in steps S31-S33 may be one or more. For ease of description, the service data whose data in the preset key fields does not match is referred to as unmatched service data. For each piece of first service data that does not match the second service data of the peer end, the electronic device may resynchronize the first service data in the second wide table to the peer end to ensure data synchronization between the electronic device and the peer end.

[0136] The operations corresponding to the above-mentioned various data inconsistencies can be combined arbitrarily. For example, for each first business data that does not match the second business data of the other end, the electronic device can resynchronize the first business data in the second wide table to the other end. If the first business data in the electronic device still does not match the second business data of the other end after resynchronization for a preset number of times, an alarm message is output.

[0137] In the embodiment of the present application, the operations corresponding to data inconsistency may include various other achievable operations, which are not limited to this.

[0138] If the data of the preset key fields of the first business data and the second business data are consistent, that is, the data of the preset key fields of the first business data and the second business data match, it means that the data of the electronic device and the other end are consistent, and the electronic device does not need to perform any processing.

[0139] In an embodiment of the present application, the electronic device can set a data monitoring protocol and a data query protocol. The electronic device and the opposite end respectively abstract the data query protocol into interfaces, and the electronic device can run the data monitoring protocol and execute the above steps S31-S33 to extract the first business data corresponding to the target business ID from the second wide table through the interface abstracted from the data query protocol, and extract the second business data corresponding to the target business ID from the opposite end, and then use the comparison result of the first business data and the second business data as the benchmark for data synchronization to determine whether it is necessary to perform operations corresponding to data inconsistency. In an embodiment of the present application, the electronic device reads both the first business data and the second business data onto the interface that implements the data monitoring protocol, and completes the comparison of the business data on the interface, completing the data adaptation, that is, shielding the problem of different underlying storage, and solving the problem that data comparison cannot be achieved due to the inconsistency between the underlying storage of the electronic device and the opposite end.

[0140] If the comparison result of the first service data and the second service data indicates data inconsistency, the electronic device may execute a data update protocol, a data insertion protocol, a data deletion protocol, etc., to ensure that the data in the preset key fields of the first service data and the second service data match. For details on the execution of the data update protocol, please refer to the relevant description of step S24. Details on the execution of the data insertion protocol and the data deletion protocol are provided below and are not detailed here.

[0141] The above steps S31-S33 can also be executed by the opposite end. The embodiment of the present application is only described with the electronic device as the execution subject, and does not serve as a limitation.

[0142] In one embodiment of the present application, the present application embodiment also provides a data synchronization method, in which each business data has a business identifier, such as Figure 4 As shown, steps S41 to S43 are an implementation of step S11, and steps S44 to S46 are the same as the above-mentioned steps S12 to S14, and are not repeated here.

[0143] Step S41: obtaining a data update instruction, where the data update instruction includes the business data to be updated and an identifier to be updated.

[0144] Step S42: If the third service data corresponding to the to-be-updated identifier is stored in the first wide table, the to-be-updated service data is updated into the third service data.

[0145] Step S43: If the third service data is not stored in the first wide table, the to-be-updated identifier and the to-be-updated service data are stored in the first wide table in correspondence.

[0146] In the technical solution provided by the embodiment of the present application, when the opposite end needs to update its business data, the electronic device uses a data update instruction to write the business data to be updated in the first wide table, and synchronizes the business data to be updated to the opposite end through the second wide table to update the business data of the opposite end, thereby ensuring the accuracy of the data synchronized to the opposite end and reducing the error rate of data synchronization.

[0147] In step S41, a user may input a data update instruction into the electronic device, which then receives the instruction. The data update instruction may include the service data to be updated and the ID to be updated. The service data corresponding to the ID to be updated is the third service data, which is the service data to be written into the first wide table.

[0148] After obtaining the data update instruction, the electronic device may obtain the service data to be updated and the ID to be updated from the data update instruction, and then search the first wide table for the third service data corresponding to the ID to be updated.

[0149] If it is found that the third business data corresponding to the ID to be updated is stored in the first wide table, the business data to be updated will be updated to the third business data, and after the business data to be updated is updated to the third business data, the third business data will be synchronized to the second wide table. In the process of synchronizing the third business data to the second wide table, the third business data is locked. After the synchronization of the third business data to the second wide table is completed, the third business data is unlocked and synchronized to the peer end. In this way, the peer end will update the third business data corresponding to the ID to be updated, that is, update the original third business data to the third business data with the business data to be updated inserted.

[0150] If the data is not found, that is, the first wide table does not store the third business data corresponding to the ID to be updated, then the ID to be updated and the business data to be updated are stored in the first wide table accordingly. After the ID to be updated and the business data to be updated are stored in the first wide table accordingly, the ID to be updated and the business data to be updated are synchronized to the second wide table. In the process of synchronizing the ID to be updated and the business data to be inserted to the second wide table, the ID to be updated and the business data to be updated are locked. After the ID to be updated and the business data to be updated are synchronized to the second wide table, the ID to be updated and the business data to be updated are unlocked, and the ID to be updated and the business data to be updated are synchronized to the peer end.

[0151] In the embodiment of the present application, the electronic device can set a data insertion protocol and a data update protocol. The electronic device can run the data insertion protocol and the data update protocol, obtain a data update instruction, and then execute the above steps S41-S42 to synchronize the service data of the electronic device and the peer end.

[0152] In one embodiment of the present application, the present application embodiment also provides a data deletion method, in which, Figure 5 As shown, the data deletion method may include the following steps:

[0153] Step S51: Obtain a data deletion instruction, where the data deletion instruction includes a service identifier to be deleted, and the service identifier to be deleted corresponds to the fourth service data.

[0154] A user may input a data deletion instruction into the electronic device, and the electronic device receives the data deletion instruction input by the user. The data deletion instruction may include the service ID to be deleted. The service data corresponding to the service ID to be deleted is the fourth service data, and the fourth service data is the service data to be deleted.

[0155] Step S52: Set the fourth business data in the first wide table to a deleted state.

[0156] After receiving the data deletion instruction, the electronic device can obtain the service ID to be deleted from the data deletion instruction, determine the fourth service data corresponding to the service ID to be deleted from the first wide table, and set the fourth service data to a deleted state.

[0157] Step S53: Synchronize the deletion status of the fourth business data in the first wide table to the second wide table. In the process of synchronizing the deletion status of the fourth business data in the first wide table to the second wide table, lock the fourth business data in the first wide table.

[0158] Step S54: After the deletion status of the fourth business data in the first wide table is synchronized to the second wide table, the fourth business data in the first wide table is unlocked, and the deletion status of the fourth business data in the second wide table is synchronized to the peer end, so that the peer end deletes the fourth business data.

[0159] Step S53 and step S54 are as detailed as the above-mentioned step S13 and step S14. For the specific implementation process, please refer to the relevant description of the above-mentioned step S13 and step S14.

[0160] In the embodiment of the present application, the electronic device may be configured with a data deletion protocol, and may run the data deletion protocol and execute the above steps S51-S54 to synchronize the service data between the electronic device and the peer end.

[0161] In the technical solution provided in the embodiment of the present application, the electronic device uses a data deletion instruction to synchronize the deletion status of the fourth business data between the electronic device and the counterpart. In the process of synchronizing the deletion status of the fourth business data to the second wide table, the fourth business data in the first wide table is locked. In this way, the fourth business data can be prevented from being modified during the synchronization process, thereby effectively reducing the data synchronization error rate.

[0162] In addition, in the technical solution provided in the embodiment of the present application, only the fourth business data of the other end is deleted. For the fourth business data in the electronic device, only the deletion status is set for the fourth business data, and the fourth business data is not deleted. The fourth business data is subsequently adjusted to improve data synchronization efficiency.

[0163] The following combination Figure 6 The data synchronization system block diagram shown in the figure provides a detailed description of the data synchronization method provided in the embodiment of the present application. Figure 6 Taking advertising business data as an example, the data source of the entire data synchronization system is the advertising business data storage system (i.e., the advertising business side). The Creative table and Creative_result table are set up on the B side, and the Creative_result table is also set up on the C side. The structure of the Creative table and Creative_result table on the B side is the same as that of the Creative_result table on the C side. The settings of the fields in the Creative table and Creative_result table refer to the above. Figure 1-Figure 5 The configuration of the first wide table and the second wide table will not be described here. The monitoring alarm system can be set at the B end or the C end, or can be independently located on a single device.

[0164] The advertising business data storage system stores advertising business data. The advertising business data storage system uses MQ to synchronize advertising business data to the B-side.

[0165] After receiving the advertising business data, Side B stores the creative data in the Creative table and sets the update flag corresponding to the record of the creative data in the Creative table to the first flag. After the creative data is stored in the Creative table, Side B sets the update flag corresponding to the record of the creative data in the Creative table to the second flag. Based on the second flag, Side B synchronizes the creative data from the Creative table to the Creative_result table. Side B uses MQ to synchronize the creative data in the Creative_result table to Side C.

[0166] The C-side receives the advertising creative data synchronized by the B-side and stores the advertising creative data in the Creative_result table on the C-side.

[0167] The monitoring alarm system calls the interface abstracted from the data query protocol of the B-end and the C-end, randomly extracts 5% (this value can be set according to demand) of the advertising creative data corresponding to the business ID, compares the preset key fields in the extracted advertising creative data, determines whether the advertising creative data of the B-end and the C-end match, and performs the corresponding operation for data inconsistency when the two do not match.

[0168] Corresponding to the above data synchronization method, the embodiment of the present application also provides a data synchronization device, such as Figure 7 As shown, the device includes:

[0169] The storage unit 71 is used to obtain multiple business data and store the multiple business data in a first wide table;

[0170] A first synchronization unit 72 is configured to synchronize each piece of business data stored in the first wide table to the second wide table after each piece of business data is stored in the first wide table;

[0171] A locking unit 73 is configured to lock each piece of business data stored in the first wide table during synchronization of the business data to the second wide table;

[0172] The second synchronization unit 74 is configured to unlock the business data in the first wide table after synchronizing each business data stored in the first wide table to the second wide table, and synchronize the business data in the second wide table to the peer end.

[0173] In the technical solution provided by the embodiment of the present application, a first wide table and a second wide table are set up, and multiple business data are directly stored in the first wide table. The first wide table includes business data that has been stored and business data that is being stored; each business data that has been stored in the first wide table is synchronized to the second wide table. In the process of synchronizing a piece of business data to the second wide table, the business data in the first wide table is locked. In this way, it is possible to avoid the business data being modified during the synchronization process, resulting in errors in the data synchronized to the opposite end. After the synchronization of a piece of business data to the second wide table is completed, the business data in the second wide table is synchronized to the opposite end to complete the data synchronization. In addition, the business data in the first wide table is unlocked. In this way, while ensuring that the data synchronized this time is the required data, subsequent modifications to the business data can also be achieved. It can be seen that in the embodiment of the present application, by setting the first wide table and the second wide table, and locking the business data in the first wide table that is being synchronized to the second wide table, the data synchronization error rate is effectively reduced.

[0174] Optionally, the data synchronization device may further include:

[0175] An updating unit is configured to set an update identification field corresponding to each piece of business data to a first identification during the process of storing each piece of business data in the first wide table, the first identification indicating that the business data storage is not completed; and after the storage of each piece of business data in the first wide table is completed, set the update identification field corresponding to each piece of business data to a second identification, the second identification indicating that the business data storage is completed;

[0176] The first synchronization unit is specifically configured to:

[0177] For each piece of business data in the first wide table, when it is detected that the update identification field corresponding to the business data in the first wide table is the second identification, the business data in the first wide table is synchronized to the second wide table.

[0178] Optionally, the business data includes multidimensional data;

[0179] Storage unit, specifically used for:

[0180] Extract the data of the target data dimension from each piece of business data; store the data extracted from each piece of business data into the first wide table; or

[0181] Extract the target data dimension and the data of the associated dimensions of the target data dimension from each piece of business data; and store the data extracted from each piece of business data in the first wide table.

[0182] Optionally, each piece of business data has a business identifier, and the data synchronization device may further include:

[0183] a determination unit, configured to determine a target service identifier from service identifiers of a plurality of service data;

[0184] an extraction unit, configured to extract first service data corresponding to the target service identifier from the second wide table, and extract second service data corresponding to the target service identifier from the opposite end;

[0185] The execution unit is configured to execute an operation corresponding to data inconsistency if data of a preset key field in the first business data does not match data of a preset key field in the second business data.

[0186] Optionally, operations corresponding to data inconsistency include at least one of the following:

[0187] Output alarm information;

[0188] Resynchronize the first service data in the second wide table to the peer end.

[0189] Optionally, each piece of business data has a business identifier and a storage unit, which is specifically used to:

[0190] Obtain a data update instruction, which includes the business data to be inserted and the identifier to be inserted;

[0191] If the first wide table stores third business data corresponding to the identifier to be inserted, the business data to be inserted is updated into the third business data;

[0192] If the third business data is not stored in the first wide table, the identifier to be inserted and the business data to be inserted are stored in the first wide table correspondingly.

[0193] Optionally, the data synchronization device may further include:

[0194] an acquiring unit, configured to acquire a data deletion instruction, the data deletion instruction including a to-be-deleted service identifier, the to-be-deleted service identifier corresponding to the fourth service data;

[0195] A setting unit, configured to set the fourth business data in the first wide table to a deletion state;

[0196] The locking unit may also be configured to synchronize the deletion status of the fourth business data in the first wide table to the second wide table, and lock the fourth business data in the first wide table during the synchronization of the deletion status of the fourth business data in the first wide table to the second wide table;

[0197] The second synchronization unit can also be used to unlock the fourth business data in the first wide table after the deletion status of the fourth business data in the first wide table is synchronized to the second wide table, and synchronize the deletion status of the fourth business data in the second wide table to the opposite end, so that the opposite end deletes the fourth business data.

[0198] The present application also provides an electronic device, such as Figure 8 As shown, it includes a processor 801, a communication interface 802, a memory 803 and a communication bus 804, wherein the processor 801, the communication interface 802, and the memory 803 communicate with each other through the communication bus 804.

[0199] Memory 803, used for storing computer programs;

[0200] The processor 801 is configured to implement any of the above-mentioned data synchronization methods when executing the program stored in the memory 803 .

[0201] The communication bus mentioned in the electronic device mentioned above may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0202] The communication interface is used for communication between the above electronic device and other devices.

[0203] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0204] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0205] In another embodiment provided by the present application, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data synchronization method described in any one of the above embodiments is implemented.

[0206] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the data synchronization method described in any one of the above embodiments.

[0207] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0208] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0209] Each embodiment in this specification is described in a related manner. Similar portions between embodiments can be referenced to each other. Each embodiment focuses on the differences between other embodiments. In particular, the device, storage medium, and computer program product embodiments are generally similar to the method embodiments, so their descriptions are relatively simplified. For related portions, reference can be made to the descriptions of the method embodiments.

[0210] The above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the scope of protection of the present application.

Claims

1. A data synchronization method, characterized in that: Applied to an electronic device, the electronic device is arranged between a data source and a peer end, the electronic device is provided with a first wide table and a second wide table, the method comprising: Acquire multiple pieces of business data from the data source, and store the multiple pieces of business data in the first wide table; After storing each piece of business data in the first wide table, synchronizing each piece of business data stored in the first wide table to the second wide table; In the process of synchronizing each piece of business data stored in the first wide table to the second wide table, locking the business data in the first wide table; After synchronizing each piece of business data stored in the first wide table to the second wide table, the business data in the first wide table is unlocked, and the business data in the second wide table is synchronized to the peer end.

2. The method according to claim 1, characterized in that The method further comprises: In the process of storing each piece of business data in the first wide table, setting the update flag field corresponding to each piece of business data to a first flag, wherein the first flag indicates that the business data storage is incomplete; After storing each piece of business data in the first wide table, setting the update flag field corresponding to each piece of business data to a second flag, where the second flag indicates that the business data storage is completed; After storing each piece of business data in the first wide table, synchronizing each piece of business data stored in the first wide table to the second wide table includes: For each piece of business data in the first wide table, when it is detected that the update identification field corresponding to the business data in the first wide table is the second identification, the business data in the first wide table is synchronized to the second wide table.

3. The method according to claim 1, characterized in that The business data includes multi-dimensional data; The step of storing the plurality of business data in the first wide table includes: Extract the data of the target data dimension from each piece of business data; store the data extracted from each piece of business data into the first wide table; or Extract the target data dimension and the data of the associated dimension of the target data dimension from each piece of business data; and store the data extracted from each piece of business data in the first wide table accordingly.

4. The method according to any one of claims 1 to 3, characterized in that Each piece of business data has a business identifier, and the method further includes: Determining a target business identifier from business identifiers of multiple pieces of business data; Extracting the first service data corresponding to the target service identifier from the second wide table, and extracting the second service data corresponding to the target service identifier from the peer end; If the data of the preset key field in the first business data does not match the data of the preset key field in the second business data, an operation corresponding to the data inconsistency is performed.

5. The method according to claim 4, characterized in that The operations corresponding to the data inconsistency include at least one of the following: Output alarm information; Resynchronize the first service data in the second wide table to the peer end.

6. The method according to any one of claims 1 to 3, characterized in that Each piece of business data has a business identifier, and the step of obtaining multiple pieces of business data includes: Obtaining a data update instruction, wherein the data update instruction includes the business data to be updated and an identifier to be updated; The step of storing the plurality of business data in the first wide table includes: If the third service data corresponding to the to-be-updated identifier is stored in the first wide table, updating the to-be-updated service data into the third service data; If the third service data is not stored in the first wide table, the to-be-updated identifier and the to-be-updated service data are stored in the first wide table in correspondence.

7. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Obtaining a data deletion instruction, where the data deletion instruction includes a service identifier to be deleted, and the service identifier to be deleted corresponds to the fourth service data; Setting the fourth business data in the first wide table to a deleted state; synchronizing the deletion status of the fourth business data in the first wide table to the second wide table, and locking the fourth business data in the first wide table during the process of synchronizing the deletion status of the fourth business data in the first wide table to the second wide table; After synchronizing the deletion status of the fourth business data in the first wide table to the second wide table, the fourth business data in the first wide table is unlocked, and the deletion status of the fourth business data in the second wide table is synchronized to the peer end, so that the peer end deletes the fourth business data.

8. A data synchronization device, characterized in that: The device is applied to an electronic device, the electronic device being arranged between a data source and a peer end, the electronic device being provided with a first wide table and a second wide table, and the device comprising: a storage unit, configured to obtain a plurality of pieces of business data from the data source, and store the plurality of pieces of business data in the first wide table; a first synchronization unit, configured to synchronize each piece of business data stored in the first wide table to the second wide table after each piece of business data is stored in the first wide table; a locking unit, configured to lock each piece of business data stored in the first wide table during synchronization of the business data to the second wide table; The second synchronization unit is configured to unlock the business data in the first wide table after synchronizing each business data stored in the first wide table to the second wide table, and synchronize the business data in the second wide table to the peer end.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; A processor, configured to implement the method steps described in any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps according to any one of claims 1 to 7 are implemented.

Images