Cross-boundary data synchronization method and device, computer equipment and storage medium

Abstract

The invention relates to a cross-boundary data synchronization method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring a source record table of source equipment and a target record table of target equipment; taking the incremental data of the same batch as the incremental data of the current batch, and determining the size relationship between the source statistical information of the incremental data of the current batch and the target statistical information of the incremental data of the current batch; under the condition that the target statistical information of the current batch of incremental data is not equal to the source statistical information of the current batch of incremental data, updating the target state information corresponding to the received data according to the source state information corresponding to the sent data, and the target device synchronizes the current batch of incremental data according to the target state information corresponding to the received data. By adopting the method, the data consistency of the incremental data of the same batch in the source equipment and the target equipment can be ensured, and the accuracy of data synchronization is further ensured.

Description

Technical Field

[0001] This application relates to the technical field of big data processing, and in particular to a cross-border data synchronization method, device, computer device, and storage medium. Background Art

[0002] Data synchronization is to synchronize data from the business database to the data lake, which is a basic link in big data processing. It is necessary to ensure the quality of the synchronized data to provide fresh and accurate data for subsequent data model construction and data analysis. In some classified scenarios, there are many network domains, and there are security boundaries between the network domains and they cannot be directly connected, which makes it difficult to ensure the accuracy of data synchronization and difficult to detect data problems in a timely manner.

[0003] In traditional technologies, since the security boundary completely separates the source network domain from the target network domain, when the data in the source network domain changes in real time, the target network domain cannot perceive it. When performing data reconciliation after data synchronization, it is also difficult to ensure the consistency of data reconciliation, thereby affecting the accuracy of data synchronization. Summary of the Invention

[0004] Based on this, in view of the above technical problems, it is necessary to provide a cross-border data synchronization method, device, computer device, and storage medium that can ensure the accuracy of data synchronization.

[0005] In a first aspect, this application provides a cross-border data synchronization method, including:

[0006] Obtain the source record table of the source device and the target record table of the target device; the source record table records the source statistical information of each batch of incremental data sent by the source device to the target device and the source status information corresponding to the sent data; the target record table records the target statistical information of the corresponding batch of incremental data and the target status information corresponding to the received data; there is a security boundary between the source network domain to which the source device belongs and the target network domain to which the target device belongs;

[0007] Take the incremental data of the same batch as the current batch of incremental data, and determine the magnitude relationship between the source statistical information of the current batch of incremental data and the target statistical information of the current batch of incremental data;

[0008] In the case where the target statistical information of the current batch of incremental data is not equal to the source statistical information of the current batch of incremental data, update the target status information corresponding to the received data according to the source status information corresponding to the sent data, so that the target device synchronizes the current batch of incremental data according to the target status information corresponding to the received data.

[0009] In one embodiment, when the target statistical information of the current batch of incremental data is not equal to the source statistical information of the current batch of incremental data, the target status information corresponding to the received data is updated according to the source status information corresponding to the sent data, including:

[0010] When the target statistical information of the current batch of incremental data is less than the source statistical information of the current batch of incremental data, the target status information corresponding to the received data is directly updated according to the source status information corresponding to the sent data;

[0011] When the target statistical information of the current batch of incremental data is greater than the source statistical information of the current batch of incremental data, based on the size relationship between the target statistical information and the preset standard statistical information, the target status information corresponding to the received data is updated according to the source status information corresponding to the sent data.

[0012] In one embodiment, based on the size relationship between the target statistical information and the preset standard statistical information, the target status information corresponding to the received data is updated according to the source status information corresponding to the sent data, including:

[0013] Determine the size relationship between the target statistical information of the current batch of incremental data and the preset standard statistical information;

[0014] When the target statistical information is greater than the standard statistical information, the target status information corresponding to the received data is updated according to the source status information corresponding to the sent data.

[0015] In one embodiment, the target status information corresponding to the received data is updated according to the source status information corresponding to the sent data, including:

[0016] In the target record table, obtain the first status information of the received data corresponding to the current batch of incremental data, and the second status information of the received data corresponding to the previous batch of incremental data;

[0017] Select the source status information that is less than the first status information and not less than the second status information as the third status information;

[0018] Update the target status information corresponding to the received data in the target record table according to the third status information.

[0019] In one embodiment, the method further includes:

[0020] When the target statistical information of the current batch of incremental data is equal to the source statistical information of the corresponding batch of incremental data, obtain the target statistical information of the historical batch of incremental data;

[0021] Determine the data quality of the current batch of incremental data according to the degree of difference between the target statistical information of the current batch of incremental data and the target statistical information of the historical batch of incremental data.

[0022] In one embodiment, determining the data quality of the current batch of incremental data according to the degree of difference between the target statistical information of the current batch of incremental data and the target statistical information of the historical batch of incremental data includes:

[0023] Determine the statistical difference between the target statistical information of the current batch of incremental data and the target statistical information of the historical batch of incremental data;

[0024] In the case where the statistical difference exceeds the preset standard difference range, determine that the data quality of the current batch of incremental data is unqualified.

[0025] In a second aspect, the present application also provides a cross-border data synchronization device, including:

[0026] A record acquisition module, configured to acquire a source record table of a source device and a target record table of a target device; the source record table records the source statistical information of each batch of incremental data sent by the source device to the target device and the source status information corresponding to the sent data; the target record table records the target statistical information of the corresponding batch of incremental data and the target status information corresponding to the received data; there is a security boundary between the source network domain to which the source device belongs and the target network domain to which the target device belongs;

[0027] A relationship determination module, configured to use the incremental data of the same batch as the current batch of incremental data, and determine the magnitude relationship between the source statistical information of the current batch of incremental data and the target statistical information of the current batch of incremental data;

[0028] A status update module, configured to, in the case where the target statistical information of the current batch of incremental data is not equal to the source statistical information of the current batch of incremental data, update the target status information corresponding to the received data according to the source status information corresponding to the sent data, so that the target device synchronizes the current batch of incremental data according to the target status information corresponding to the received data.

[0029] In a third aspect, the present application also provides a computer device, including a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0030] Obtain the source record table of the source device and the target record table of the target device; the source record table records the source statistical information of each batch of incremental data sent from the source device to the target device and the source status information corresponding to the sent data; the target record table records the target statistical information of the corresponding batch of incremental data and the target status information corresponding to the received data; there is a security boundary between the source network domain to which the source device belongs and the target network domain to which the target device belongs;

[0031] Take the incremental data of the same batch as the current batch of incremental data, and determine the size relationship between the source statistical information of the current batch of incremental data and the target statistical information of the current batch of incremental data;

[0032] In the case where the target statistical information of the current batch of incremental data is not equal to the source statistical information of the current batch of incremental data, update the target status information corresponding to the received data according to the source status information corresponding to the sent data, so that the target device synchronizes the current batch of incremental data according to the target status information corresponding to the received data.

[0033] Fourthly, the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0034] Obtain the source record table of the source device and the target record table of the target device; the source record table records the source statistical information of each batch of incremental data sent from the source device to the target device and the source status information corresponding to the sent data; the target record table records the target statistical information of the corresponding batch of incremental data and the target status information corresponding to the received data; there is a security boundary between the source network domain to which the source device belongs and the target network domain to which the target device belongs;

[0035] Take the incremental data of the same batch as the current batch of incremental data, and determine the size relationship between the source statistical information of the current batch of incremental data and the target statistical information of the current batch of incremental data;

[0036] In the case where the target statistical information of the current batch of incremental data is not equal to the source statistical information of the current batch of incremental data, update the target status information corresponding to the received data according to the source status information corresponding to the sent data, so that the target device synchronizes the current batch of incremental data according to the target status information corresponding to the received data.

[0037] Fifthly, the present application also provides a computer program product, including a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0038] Obtain the source record table of the source device and the target record table of the target device; the source record table records the source statistical information of each batch of incremental data sent by the source device to the target device and the source status information corresponding to the sent data; the target record table records the target statistical information of the corresponding batch of incremental data and the target status information corresponding to the received data; there is a security boundary between the source network domain to which the source device belongs and the target network domain to which the target device belongs.

[0039] Use the incremental data of the same batch as the current batch of incremental data, and determine the size relationship between the source statistical information of the current batch of incremental data and the target statistical information of the current batch of incremental data.

[0040] In the case that the target statistical information of the current batch of incremental data is not equal to the source statistical information of the current batch of incremental data, update the target status information corresponding to the received data according to the source status information corresponding to the sent data, so that the target device synchronizes the current batch of incremental data according to the target status information corresponding to the received data.

[0041] The above cross-boundary data synchronization method, device, computer device and storage medium obtain the source record table of the source device and the target record table of the target device; the source record table records the source statistical information of each batch of incremental data sent by the source device to the target device and the source status information corresponding to the sent data; the target record table records the target statistical information of the corresponding batch of incremental data and the target status information corresponding to the received data; there is a security boundary between the source network domain to which the source device belongs and the target network domain to which the target device belongs; use the incremental data of the same batch as the current batch of incremental data, and determine the size relationship between the source statistical information of the current batch of incremental data and the target statistical information of the current batch of incremental data; in the case that the target statistical information of the current batch of incremental data is not equal to the source statistical information of the current batch of incremental data, update the target status information corresponding to the received data according to the source status information corresponding to the sent data, so that the target device synchronizes the current batch of incremental data according to the target status information corresponding to the received data.

[0042] In this embodiment, through the source record table and the target record table, the data statistical information and data status information of the synchronized batch of incremental data are recorded, and data reconciliation is performed through the data statistical information and data status information. In the case that the target statistical information is not equal to the source statistical information, it proves that there may be changes in the sent data in the source device that are unknown to the target device. Therefore, according to the source status information corresponding to the sent data, the target status information corresponding to the received data is updated, so as to ensure the data consistency of the same batch of incremental data between the source device and the target device, and further ensure the accuracy of data synchronization. Description of the Drawings

[0043] To more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the related art. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0044] Figure 1 It is an application environment diagram of a cross-border data synchronization method provided in this embodiment;

[0045] Figure 2 It is a schematic flowchart of a cross-border data synchronization method provided in this embodiment;

[0046] Figure 3 It is a schematic flowchart of the first state information update step provided in this embodiment;

[0047] Figure 4 It is a schematic flowchart of the second state information update step provided in this embodiment;

[0048] Figure 5 It is a structural block diagram of a cross-border data synchronization device provided in this embodiment;

[0049] Figure 6 It is an internal structure diagram of a computer device provided in this embodiment. Detailed implementation manners

[0050] In order to make the purpose, technical solutions and advantages of the present application more clear, the following further details the present application in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] The cross-border data synchronization method provided by the embodiments of the present application can be applied to an application environment as Figure 1 shown. Among them,

[0052] The source domain can be the domain for initiating data synchronization, and the target domain can be the domain for receiving synchronized data. There is a security boundary between the source domain and the target domain. The source device 102 can be the device for initiating data synchronization. The source device can perform operations of adding, deleting, modifying, and querying the incremental data to be synchronized. The target device 103 can be the device for receiving synchronized data. The target device can perform operations of adding, deleting, modifying, and querying the received data according to the target status information after data synchronization to ensure data synchronization of the corresponding data in the source device and the target device. The data synchronization device 101 can receive the incremental data sent by the source device 102, and, when there is a need for the target device 103 to receive incremental data, send the corresponding incremental data to the target device 103. The data synchronization device 101 in the target domain obtains the source record table of the source device 102 and the target record table of the target device 103. The source record table records the source statistical information of each batch of incremental data sent by the source device 101 to the target device 103 and the source status information corresponding to the sent data. The target record table records the target statistical information of the corresponding batch of incremental data and the target status information corresponding to the received data. There is a security boundary between the source domain to which the source device 102 belongs and the target domain to which the target device 103 belongs. Taking the same batch of incremental data as the current batch of incremental data, determine the magnitude relationship between the source statistical information of the current batch of incremental data and the target statistical information of the current batch of incremental data. When the target statistical information of the current batch of incremental data is not equal to the source statistical information of the current batch of incremental data, update the target status information corresponding to the received data according to the source status information corresponding to the sent data, so that the target device synchronizes the current batch of incremental data according to the target status information corresponding to the received data.

[0053] In an exemplary embodiment, as Figure 2 shown, a cross-boundary data synchronization method is provided. Taking the data synchronization device 101 in Figure 1 as an example for illustration, it includes the following steps 201 to 203. Among them:

[0054] Step 201, obtain the source record table of the source device and the target record table of the target device.

[0055] Among them, the source record table records the source statistical information of each batch of incremental data sent by the source device to the target device and the source status information corresponding to the sent data. The target record table records the target statistical information of the corresponding batch of incremental data and the target status information corresponding to the received data. There is a security boundary between the source domain to which the source device belongs and the target domain to which the target device belongs.

[0056] Among them, the source record table may include a source statistics table and a source status table; the source statistics table records the source statistical information of each batch of incremental data sent by the source device to the target device; the source status table records the source status information of each batch of incremental data that has been sent.

[0057] Among them, the target record table may include a target statistics table and a target status table; the target statistics table records the target statistical information of each batch of incremental data received by the target device from the source device; the target status table records the target status information of each batch of incremental data that has been sent.

[0058] Among them, the source statistical information may be the data statistical information of each batch of incremental data in the source device. The target statistical information may be the data statistical information of each batch of incremental data in the target device. The data statistical information may include the total number of data rows, the number of different field values, the minimum field value, the maximum field value, and the number of null field values.

[0059] Among them, the source status information may be the data status information of each batch of incremental data that has been sent in the source device; the target status information may be the data status information of each batch of incremental data that has been received in the target device; the data status information is the information for changing the add, delete, modify, and query status of the incremental data. Exemplarily, the data status information of incremental data 1 may be the delete status.

[0060] Specifically, the data synchronization device sends a source record table acquisition request to the source device, the source device receives the source record table acquisition request, and feeds back the source statistics table and the source status table to the data synchronization device; the data synchronization device receives the source statistics table and the source status table. Similarly, the data synchronization device sends a target record table acquisition request to the target device, the target device receives the target record table acquisition request, and feeds back the target statistics table and the target status table to the data synchronization device; the data synchronization device receives the target statistics table and the target status table.

[0061] It should be noted that the data synchronization device can synchronize data between the source device and the target device periodically; it can also synchronize data between the source device and the target device when there is a data synchronization requirement for the target device. This application does not make any limitations in this regard.

[0062] Step 202: Use the incremental data of the same batch as the current batch of incremental data, and determine the magnitude relationship between the source statistical information of the current batch of incremental data and the target statistical information of the current batch of incremental data.

[0063] Specifically, for the data statistical information of each dimension, determine the magnitude relationship between the source statistical information of the current batch of incremental data and the target statistical information of the current batch of incremental data; determine the number of dimensions of the data statistical information where the source statistical information is greater than the target statistical information; if the number of dimensions is greater than the preset quantity threshold, then determine that the source statistical information is greater than the target statistical information.

[0064] Step 203, in the case where the target statistical information of the current batch of incremental data is not equal to the source statistical information of the current batch of incremental data, update the target status information corresponding to the received data according to the source status information corresponding to the sent data, so that the target device synchronizes the current batch of incremental data according to the target status information corresponding to the received data.

[0065] Specifically, in the case where the target statistical information of the current batch of incremental data is greater than the source statistical information of the current batch of incremental data, or in the case where the target statistical information of the current batch of incremental data is less than the source statistical information of the current batch of incremental data, at this time, it is proved that there is an abnormal reconciliation between the source device and the target device, then overwrite and update the target status information corresponding to the received data with the source status information corresponding to the sent data; the data synchronization device sends the target record table (or target status table) with the updated target status information to the target device; the target device changes the current batch of incremental data according to the target status information corresponding to the received data in the target record table (or target status table) to complete data synchronization.

[0066] Exemplarily, if the target device receives that the incremental data 1 in the target status table is in a deleted state, while the corresponding incremental quantity 1 in the target device has not been deleted, then delete the incremental data 1 to complete the data synchronization between the source device and the target device.

[0067] For the above cross-border data synchronization method, obtain the source record table of the source device and the target record table of the target device; the source record table records the source statistical information of each batch of incremental data sent by the source device to the target device and the source status information corresponding to the sent data; the target record table records the target statistical information of the corresponding batch of incremental data and the target status information corresponding to the received data; there is a security boundary between the source network domain to which the source device belongs and the target network domain to which the target device belongs; use the incremental data of the same batch as the current batch of incremental data, and determine the magnitude relationship between the source statistical information of the current batch of incremental data and the target statistical information of the current batch of incremental data; in the case where the target statistical information of the current batch of incremental data is not equal to the source statistical information of the current batch of incremental data, update the target status information corresponding to the received data according to the source status information corresponding to the sent data, so that the target device synchronizes the current batch of incremental data according to the target status information corresponding to the received data.

[0068] In this embodiment, the source record table and the target record table are used to record the data statistical information and data status information of the incremental data of the synchronization batch. Data reconciliation is performed through the data statistical information and data status information. When the target statistical information is not equal to the source statistical information, it is proved that there may be changes in the data sent by the source device that are unknown to the target device. Therefore, according to the source status information corresponding to the sent data of the sent data, the target status information corresponding to the received data is updated, so as to ensure the data consistency of the incremental data of the same batch between the source device and the target device, and further ensure the accuracy of data synchronization.

[0069] It should be noted that the method for the source device to send incremental data to the target device in this application may include: the target device extracts a preset number of incremental data slices (such as 10,000 pieces of data) from the business database; converts the extracted incremental data slices into a preset format file (such as serialization format, avro format) to obtain the target incremental data; puts the target incremental data into a pre-configured data channel, and sends the target incremental data to the data synchronization device through the data channel. The data synchronization device detects whether there is new target incremental data in the data channel. If so, it receives the target incremental data and sends the target incremental data to the corresponding daily partition in the target device. Among them, the data channel may be a channel corresponding to the File Transfer Protocol (FTP) with a configured transmission policy.

[0070] It should be noted that after the data synchronization device receives the reminder that the incremental data reception is completed feedback from the target device, it proves that the incremental data synchronization task is completed at this time, and feeds back the task completion time of the incremental data synchronization task to the source device. At this time, the source device needs to complete the following operations: persist the task completion time and the source status information of this batch of incremental data into the source status table maintained locally, and at the same time, the source status table can be synchronized to the target device through the data synchronization device by the data channel. Persist the data statistical information within the preset time range into the source statistical table maintained locally, and at the same time, the source statistical table can be synchronized to the target device by the data channel. Among them, the preset time range may include the time range corresponding to the source status information of the previous batch of incremental data to the source status information of this batch of incremental data. At this time, the target needs to complete the following operations: select the corresponding table partition according to the task completion time, determine the target statistical information in the table partition, and write the target statistical information into the target statistical table maintained locally by the target device.

[0071] Figure 3It is a schematic flowchart of the status information update step in an embodiment. In this embodiment, the step of updating the target status information corresponding to the received data according to the source status information corresponding to the sent data when the target statistical information of the current batch of incremental data is not equal to the source statistical information of the current batch of incremental data in the above embodiment is refined, and an optional method for updating the status information is given, including the following steps:

[0072] Step 301, when the target statistical information of the current batch of incremental data is less than the source statistical information of the current batch of incremental data, directly update the target status information corresponding to the received data according to the source status information corresponding to the sent data.

[0073] One optional implementation method: when the target statistical information of the current batch of incremental data is less than the source statistical information of the current batch of incremental data, it proves that there may be some unreceived incremental data in the target device at this time. Therefore, it is necessary to overwrite and update the target status information corresponding to the received data with the source status information corresponding to the sent data.

[0074] Another optional implementation method: in the target record table, obtain the first status information of the received data corresponding to the current batch of incremental data, and the second status information of the received data corresponding to the previous batch of incremental data; select the source status information that is less than the first status information and not less than the second status information as the third status information; update the target status information corresponding to the received data in the target record table according to the third status information.

[0075] Specifically, obtain the first status information of the current batch of incremental data and the second status information of the previous batch of incremental data from the target record table; select the source status information that is less than the first status information of the received data corresponding to the current batch of incremental data and not less than the second status information of the received data corresponding to the previous batch of incremental data as the third status information; overwrite and update the target status information corresponding to the received data in the target record table according to the third status information.

[0076] Step 302, when the target statistical information of the current batch of incremental data is greater than the source statistical information of the current batch of incremental data, update the target status information corresponding to the received data according to the source status information corresponding to the sent data based on the size relationship between the target statistical information and the preset standard statistical information.

[0077] In one optional implementation method: based on the size relationship between the target statistical information and the preset standard statistical information, determine the corresponding update method; through the update method, update the target status information corresponding to the received data according to the source status information corresponding to the sent data.

[0078] In another alternative implementation: Determine the magnitude relationship between the target statistical information of the current batch of incremental data and the preset standard statistical information; in the case where the target statistical information is greater than the standard statistical information, update the target status information corresponding to the received data according to the source status information corresponding to the sent data.

[0079] It should be noted that due to the existence of the security boundary, the incremental data between the target device and the source device on both sides of the security boundary cannot be synchronized in time, and it is possible that the source device deletes some incremental data, while the target device cannot perceive this deletion operation. In this case, the situation where the target statistical information of the current batch of incremental data is greater than the source statistical information of the current batch of incremental data will occur. However, since all the incremental data has been completely received in the target device and there is no un-received incremental data, therefore, for some incremental data that has not been deleted or modified, it can be temporarily not modified, or it can be determined that the reconciliation is normal.

[0080] Furthermore, due to the different device performances of the target device, therefore, the target device can set the preset standard statistical information. When the target statistical information is greater than the standard statistical information, it proves that the incremental information stored in the target device exceeds the limit, and there may be a situation where new incremental data cannot be received. Therefore, it is necessary to further complete the data synchronization between the source device and the target device, so as to determine which incremental data can be deleted from the source device, thereby further releasing the data storage space in the target device to accommodate more new incremental data.

[0081] In this embodiment, for the situation where the magnitude of the target statistical information of the current batch of incremental data is different from the source statistical information of the current batch of incremental data, different ways of updating the target status information corresponding to the received data are set, further ensuring the timeliness and accuracy of data synchronization.

[0082] Figure 4 It is a schematic flowchart of the status information update step in an embodiment. In this embodiment, the step of updating the target status information corresponding to the received data according to the source status information corresponding to the sent data in the above embodiment when the target statistical information of the current batch of incremental data is not equal to the source statistical information of the current batch of incremental data is refined, and an alternative way of status information update is given, including the following steps:

[0083] Step 401, when the target statistical information of the current batch of incremental data is equal to the source statistical information of the corresponding batch of incremental data, obtain the target statistical information of the historical batch of incremental data.

[0084] Specifically, when the target statistical information of the current batch of incremental data is equal to the source statistical information of the corresponding batch of incremental data, the data synchronization device obtains the target statistical information of a preset number of historical batches of incremental data.

[0085] Step 402: Determine the data quality of the current batch of incremental data according to the degree of difference between the target statistical information of the current batch of incremental data and the target statistical information of the historical batch of incremental data.

[0086] In an alternative implementation: According to the degree of difference between the target statistical information of the current batch of incremental data and the target statistical information of the historical batch of incremental data; determine the preset data quality corresponding to the degree of difference; use the preset data quality as the data quality of the current batch of incremental data.

[0087] Another alternative implementation: Determine the statistical difference between the target statistical information of the current batch of incremental data and the target statistical information of the historical batch of incremental data; when the statistical difference exceeds the preset standard difference range, determine that the data quality of the current batch of incremental data is unqualified.

[0088] Specifically, determine the statistical difference between the target statistical information of the current batch of incremental data and the target statistical information of the historical batch of incremental data; when the statistical difference exceeds the preset standard difference range, determine that the data quality of the current batch of incremental data is unqualified; when the statistical difference does not exceed the preset standard difference range, determine that the data quality of the current batch of incremental data is qualified.

[0089] Still another alternative implementation: For the target statistical information of each dimension, determine the dimension difference between the target statistical information of the current batch of incremental data and the target statistical information of the historical batch of incremental data in this dimension; perform a weighted sum of the dimension differences of each dimension and the preset dimension weights corresponding to the respective dimensions to obtain a statistical difference; when the statistical difference exceeds the preset standard difference range, determine that the data quality of the current batch of incremental data is unqualified; when the statistical difference does not exceed the preset standard difference range, determine that the data quality of the current batch of incremental data is qualified.

[0090] In this embodiment, when the target statistical information of the current batch of incremental data is equal to the source statistical information of the corresponding batch of incremental data, obtain the target statistical information of the historical batch of incremental data; according to the degree of difference between the target statistical information of the current batch of incremental data and the target statistical information of the historical batch of incremental data, more accurately determine the data quality of the current batch of incremental data.

[0091] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are shown in sequence according to the indications of the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, there is no strict order limit for the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0092] Based on the same inventive concept, an embodiment of the present application further provides a cross-border data synchronization device for implementing the above-mentioned cross-border data synchronization method. The implementation solutions provided by this device to solve problems are similar to the implementation solutions described in the above method. Therefore, the specific limitations in one or more embodiments of the cross-border data synchronization device provided below can refer to the limitations on the cross-border data synchronization method in the above text, and will not be repeated here.

[0093] In an exemplary embodiment, as Figure 5 shown, a cross-border data synchronization device is provided, including: a record acquisition module 10, a relationship determination module 11, and a status update module 12, where:

[0094] The record acquisition module 10 is used to acquire the source record table of the source device and the target record table of the target device; the source record table records the source statistical information of each batch of incremental data sent by the source device to the target device and the source status information corresponding to the sent data; the target record table records the target statistical information of the corresponding batch of incremental data and the target status information corresponding to the received data; there is a security boundary between the source network domain to which the source device belongs and the target network domain to which the target device belongs;

[0095] The relationship determination module 11 is used to use the incremental data of the same batch as the current batch of incremental data, and determine the magnitude relationship between the source statistical information of the current batch of incremental data and the target statistical information of the current batch of incremental data;

[0096] The status update module 12 is used to, when the target statistical information of the current batch of incremental data is not equal to the source statistical information of the current batch of incremental data, update the target status information corresponding to the received data according to the source status information corresponding to the sent data, so that the target device synchronizes the current batch of incremental data according to the target status information corresponding to the received data.

[0097] In some embodiments, the status update module 12 further includes:

[0098] First, an update unit, configured to directly update the target status information corresponding to the received data according to the source status information corresponding to the sent data when the target statistical information of the current batch of incremental data is less than the source statistical information of the current batch of incremental data;

[0099] Second, an update unit, configured to update the target status information corresponding to the received data according to the source status information corresponding to the sent data based on the magnitude relationship between the target statistical information and the preset standard statistical information when the target statistical information of the current batch of incremental data is greater than the source statistical information of the current batch of incremental data.

[0100] In some embodiments, the second update unit is further configured to determine the magnitude relationship between the target statistical information of the current batch of incremental data and the preset standard statistical information; and update the target status information corresponding to the received data according to the source status information corresponding to the sent data when the target statistical information is greater than the standard statistical information.

[0101] In some embodiments, the status update module 12 is further configured to obtain, in the target record table, the first status information of the received data corresponding to the current batch of incremental data, and the second status information of the received data corresponding to the previous batch of incremental data; select the source status information that is less than the first status information and not less than the second status information as the third status information; and update the target status information corresponding to the received data in the target record table according to the third status information.

[0102] In some embodiments, the cross-border data synchronization device further includes:

[0103] A historical acquisition module, configured to obtain the target statistical information of the historical batch of incremental data when the target statistical information of the current batch of incremental data is equal to the source statistical information of the corresponding batch of incremental data;

[0104] A quality determination module, configured to determine the data quality of the current batch of incremental data according to the degree of difference between the target statistical information of the current batch of incremental data and the target statistical information of the historical batch of incremental data.

[0105] In some embodiments, the quality determination module is further configured to determine the statistical difference between the target statistical information of the current batch of incremental data and the target statistical information of the historical batch of incremental data; and determine that the data quality of the current batch of incremental data is unqualified when the statistical difference exceeds the preset standard difference range.

[0106] Each module in the above cross - border data synchronization device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above modules can be embedded in the processor of the computer device in hardware form or be independent of it, or can be stored in the memory of the computer device in software form, so as to facilitate the processor to call and execute the operations corresponding to each of the above modules.

[0107] In an exemplary embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as Figure 6 shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non - volatile storage medium and an internal memory. The non - volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non - volatile storage medium. The database of the computer device is used to store data. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through a network connection. When the computer program is executed by the processor, it implements a cross - border data synchronization method.

[0108] Those skilled in the art can understand that Figure 6 the structure shown in

[0109] is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have a different component layout.

[0110] In an embodiment, a computer device is further provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above - mentioned method embodiments are implemented.

[0111] In an embodiment, a computer - readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, the steps in the above - mentioned method embodiments are implemented.

[0112] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0113] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., and are not limited thereto. The processors involved in the embodiments provided in this application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., and are not limited thereto.

[0114] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the various technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0115] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A cross-border data synchronization method, characterized in that, The method includes: Obtaining a source record table of a source device and a target record table of a target device; the source record table records source statistical information of each batch of incremental data sent from the source device to the target device and source status information corresponding to the sent data; the target record table records target statistical information of the corresponding batch of incremental data and target status information corresponding to the received data; there is a security boundary between the source network domain to which the source device belongs and the target network domain to which the target device belongs; Taking the incremental data of the same batch as the current batch of incremental data, and determining the magnitude relationship between the source statistical information of the current batch of incremental data and the target statistical information of the current batch of incremental data; When the target statistical information of the current batch of incremental data is not equal to the source statistical information of the current batch of incremental data, updating the target status information corresponding to the received data according to the source status information corresponding to the sent data, so that the target device synchronizes the current batch of incremental data according to the target status information corresponding to the received data.

2. The method according to claim 1, wherein When the target statistical information of the current batch of incremental data is not equal to the source statistical information of the current batch of incremental data, updating the target status information corresponding to the received data according to the source status information corresponding to the sent data includes: When the target statistical information of the current batch of incremental data is less than the source statistical information of the current batch of incremental data, directly updating the target status information corresponding to the received data according to the source status information corresponding to the sent data; When the target statistical information of the current batch of incremental data is greater than the source statistical information of the current batch of incremental data, based on the magnitude relationship between the target statistical information and a preset standard statistical information, updating the target status information corresponding to the received data according to the source status information corresponding to the sent data.

3. The method according to claim 2, wherein Based on the magnitude relationship between the target statistical information and a preset standard statistical information, updating the target status information corresponding to the received data according to the source status information corresponding to the sent data includes: Determining the magnitude relationship between the target statistical information of the current batch of incremental data and the preset standard statistical information; When the target statistical information is greater than the standard statistical information, updating the target status information corresponding to the received data according to the source status information corresponding to the sent data.

4. The method according to claim 2 or 3, characterized in that Updating the target status information corresponding to the received data according to the source status information corresponding to the sent data includes: In the target record table, obtaining first status information of the received data corresponding to the current batch of incremental data and second status information of the received data corresponding to the previous batch of incremental data; Selecting source status information that is less than the first status information and not less than the second status information as third status information; Updating the target status information corresponding to the received data in the target record table according to the third status information.

5. The method according to claim 1, characterized in that, The method further includes: When the target statistical information of the current batch of incremental data is equal to the source statistical information of the corresponding batch of incremental data, obtaining the target statistical information of the historical batch of incremental data; Determine the data quality of the current batch of incremental data according to the degree of difference between the target statistical information of the current batch of incremental data and the target statistical information of the historical batch of incremental data.

6. The method according to claim 5, wherein Determine the data quality of the current batch of incremental data according to the degree of difference between the target statistical information of the current batch of incremental data and the target statistical information of the historical batch of incremental data, including: Determine the statistical difference between the target statistical information of the current batch of incremental data and the target statistical information of the historical batch of incremental data; When the statistical difference exceeds the preset standard difference range, determine that the data quality of the current batch of incremental data is unqualified.

7. A cross-border data synchronization device, characterized in that, The device includes: A record acquisition module, configured to acquire a source record table of a source device and a target record table of a target device; the source record table records the source statistical information of each batch of incremental data sent by the source device to the target device and the source status information corresponding to the sent data; the target record table records the target statistical information of the corresponding batch of incremental data and the target status information corresponding to the received data; there is a security boundary between the source network domain to which the source device belongs and the target network domain to which the target device belongs; A relationship determination module, configured to use the incremental data of the same batch as the current batch of incremental data, and determine the magnitude relationship between the source statistical information of the current batch of incremental data and the target statistical information of the current batch of incremental data; A status update module, configured to, when the target statistical information of the current batch of incremental data is not equal to the source statistical information of the current batch of incremental data, update the target status information corresponding to the received data according to the source status information corresponding to the sent data, so that the target device synchronizes the current batch of incremental data according to the target status information corresponding to the received data.

8. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images