Data synchronization method, electronic equipment and storage medium

Abstract

The invention discloses a data synchronization method, electronic equipment and a storage medium, and relates to the technical field of data synchronization. The data synchronization method comprises the following steps: acquiring historical synchronization information of a data source end, and determining quality index information of data pushing based on the historical synchronization information; determining a pushing credibility of the data source end based on the quality index information, and determining a data interaction mode of the data source end according to the credibility; and obtaining a to-be-synchronized task based on the data interaction mode, and synchronizing the to-be-synchronized task to a target receiving end based on a preset data synchronization function. According to the embodiment of the invention, the problem of a trust black box existing in existing data synchronization is solved, and meanwhile, different data interaction modes are determined based on different push trust degrees, so that the safety and convenience of obtaining the to-be-synchronized task are improved, the to-be-synchronized task is conveniently and quickly synchronized, and the use experience of a user is improved.

Description

Technical Field

[0001] The present invention relates to the field of data synchronization technology, and in particular to a data synchronization method, electronic equipment, and storage medium. Background Art

[0002] Currently, when data is transmitted, aggregated, and integrated between different data sources, in order to ensure the stability and security of the received data, temporary table data is often set up for physical isolation. The data provider pushes the data to be updated to the temporary table for transfer, and the data receiver then synchronizes it to the corresponding data table. Because the data from both parties comes from different databases and effective interaction judgment rules are not set or lacking, the data receiver cannot promptly and accurately understand that the data synchronization task has been completed, which will lead to problems such as data synchronization delays and inaccuracies.

[0003] For data providers and data receivers, since there is no data information communication between the two, it is unknown to both parties whether the transferred data is ready before the sequence-based scheduled task is completed. This will cause a trust black box problem. Therefore, how to solve the trust black box problem in existing data synchronization and improve the efficiency of task synchronization has become an urgent problem to be solved. Summary of the Invention

[0004] The present invention provides a data synchronization method, an electronic device and a storage medium to solve the trust black box problem and the low efficiency problem of task synchronization in existing data synchronization.

[0005] According to one aspect of the present invention, a data synchronization method is provided, wherein the method comprises:

[0006] Acquire historical synchronization information of a data source, and determine quality indicator information of data push based on the historical synchronization information;

[0007] Determining the push trust of the data source end based on the quality indicator information, and determining the data interaction mode of the data source end according to the trust;

[0008] The tasks to be synchronized are acquired based on the data interaction mode, and the tasks to be synchronized are synchronized to a target receiving end based on a preset data synchronization function.

[0009] According to another aspect of the present invention, a data synchronization device is provided, wherein the device includes:

[0010] An information determination module is used to obtain historical synchronization information of a data source and determine quality indicator information of data push based on the historical synchronization information;

[0011] A mode determination module, configured to determine the push trust of the data source end based on the quality indicator information, and determine the data interaction mode of the data source end according to the trust;

[0012] The data synchronization module is used to obtain the tasks to be synchronized based on the data interaction mode, and synchronize the tasks to be synchronized to the target receiving end based on a preset data synchronization function.

[0013] According to another aspect of the present invention, an electronic device is provided, comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the data synchronization method described in any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data synchronization method described in any embodiment of the present invention when executed.

[0018] The technical solution of the embodiment of the present invention solves the trust black box problem existing in existing data synchronization by obtaining historical synchronization information of the data source, determining the quality indicator information of data push based on the historical synchronization information, determining the push trust of the data source based on the quality indicator information, and determining the data interaction mode of the data source according to the trust; obtaining the task to be synchronized based on the data interaction mode, synchronizing the task to be synchronized to the target receiving end based on a preset data synchronization function, and determining different data interaction modes based on different push trusts, thereby improving the security and convenience of obtaining the task to be synchronized, facilitating the rapid synchronization of the task to be synchronized, and improving the user experience.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 This is a flow chart of a data synchronization method provided according to the first embodiment of the present invention;

[0022] Figure 2 This is a flow chart of a data synchronization method provided according to the second embodiment of the present invention;

[0023] Figure 3 This is a flowchart of a data synchronization method provided according to Embodiment 3 of the present invention;

[0024] Figure 4 This is a structural diagram of a data synchronization device provided according to a fourth embodiment of the present invention;

[0025] Figure 5 It is a structural diagram of an electronic device for implementing the data synchronization method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] Example 1

[0029] Figure 1 This is a flow chart of a data synchronization method provided according to the first embodiment of the present invention. This embodiment is applicable to the case of synchronizing data of multi-source temporary tables. The method can be executed by a data synchronization device, which can be implemented in the form of hardware and / or software and can be configured in an electronic device. Figure 1As shown, the method includes:

[0030] S110: Acquire historical synchronization information of the data source, and determine quality indicator information of data push based on the historical synchronization information.

[0031] The data source refers to the data transmitter. The data source can push the data to be updated to the database of the target receiving end, generating a task to be synchronized. In one embodiment, the number of data sources can be one or more, and the data source can be a database. Historical synchronization information refers to historical information about the synchronization of a dataset at the data source end. For example, this historical synchronization information may include, but is not limited to, the number of historical synchronizations, the number of historical synchronization successes, the arrival time of historical temporary table data, the maximum synchronization time, the planned arrival time of historical temporary table data, and a preset time offset value. In one embodiment, this historical synchronization information may also include information such as the scheduled task setting time, synchronization failure events, and the maximum number of temporary tables that can be supported before a daily scheduled task. Data push refers to the process by which the data source pushes the data to be updated to the temporary table of the database of the target receiving end. Quality indicator information can be understood as indicator information that measures the quality of data synchronization at the data source end. For example, this quality indicator information may include, but is not limited to, the first-pass pass rate, the accuracy rate, and the planned execution aggregation rate. In actual operation, the first-time pass rate can be determined according to the historical number of synchronizations and the historical number of successful synchronizations; the accuracy rate can be determined according to the historical temporary table data arrival time, the maximum synchronization time, and the historical number of synchronizations; and the plan execution aggregation rate can be determined according to the historical temporary table data arrival time, the historical temporary table data planned arrival time, and the preset time offset value.

[0032] In an embodiment, historical synchronization information can be extracted based on a time series, including the number of historical synchronizations at the data source, the number of historical successful synchronizations, the arrival time of historical temporary table data, the maximum synchronization time, the planned arrival time of historical temporary table data, and a preset time offset value. Alternatively, information such as the number of historical synchronizations and the number of historical successful synchronizations can be analyzed based on synchronization failure events and the maximum number of temporary tables before daily scheduled tasks. Quality indicators can also be used to determine the first-pass pass rate, accuracy rate, and planned execution rate of data push based on this historical synchronization information.

[0033] In the actual operation process, the number of historical synchronizations and the number of historical successful synchronizations within a time period can be determined, and the ratio of the number of historical successful synchronizations within a time period to the number of historical synchronizations can be used as the one-time pass rate. The number of times the arrival time of historical temporary table data in a time period is less than the maximum synchronization time can be determined, and the ratio of the number of times the arrival time of historical temporary table data is less than the maximum synchronization time to the number of historical synchronizations can be used as the accuracy rate. The difference between the planned arrival time of historical temporary table data and the preset time offset value can be determined as the first threshold, and the sum of the planned arrival time of historical temporary table data and the preset time offset value can be determined as the second threshold. The number of times the arrival time of historical temporary table data in a time period is greater than the first threshold and less than the second threshold can be determined, and the ratio of the number of times the arrival time of historical temporary table data is greater than the first threshold and less than the second threshold to the number of historical synchronizations can be used as the plan execution aggregation rate. The one-time pass rate, accuracy rate, and plan execution aggregation rate can be used as quality indicator information.

[0034] S120: Determine the push trust of the data source end based on the quality indicator information, and determine the data interaction mode of the data source end according to the trust.

[0035] The push trust level refers to the target receiving end's level of trust in the pending synchronization tasks pushed by the data source. Different push trust levels correspond to different data interaction modes. A data interaction mode can be understood as the mode in which the target receiving end and the data source exchange data. In one embodiment, data interaction modes may include, but are not limited to, a one-way data interaction mode based on data broadcasting; a one-way data interaction mode based on data perception; and a two-way data interaction mode based on data verification.

[0036] In an embodiment, the weight coefficient of each quality indicator information can be determined, the product of each quality indicator information and the corresponding weight coefficient can be determined, and the sum of each product can be used as the push trust. A trust interval can be pre-set for the push trust, and each data interaction mode corresponds to a trust interval. Exemplarily, a high trust interval, a medium trust interval and a low trust interval can be set. When it is determined that the push trust is in the high trust interval, the data interaction mode can be determined to be a unidirectional data interaction mode based on data broadcasting; when it is determined that the push trust is in the medium trust interval, the data interaction mode can be determined to be a one-way data interaction mode based on data perception; when it is determined that the push trust is in the low trust interval, the data interaction mode can be determined to be a two-way data interaction mode based on data verification. In one embodiment, the high trust interval may include but is not limited to less than 1 and greater than or equal to 0.75; the medium trust interval may include but is not limited to less than 0.75 and greater than or equal to 0.3; the low trust interval may include but is not limited to less than 0.3 and greater than or equal to 0.

[0037] In one embodiment, various types of quality indicator information can be formed into an indicator evaluation set based on expert opinions, and the specific quality indicator information in the evaluation set can be sorted by importance to form an evaluation indicator set with a determined order relationship, that is, y1>y2>...>y M Determine the relative importance of adjacent indicators. Suppose the experts have the following evaluation indicators: j The importance value is set to ω j , then the importance ratio of two adjacent evaluation indicators can be expressed as r i =ω j-1 / ω j ,according to Can be obtained: ω j-1 =r i *ω j To improve the rationality of weight coefficient settings, one can consult the opinions of multiple experts to determine the order relationship of various indicator sets, and then develop different indicator weight distribution schemes. The average of these expert opinions is then used as the final weighting result. Alternatively, the weight information of each quality indicator information can be determined using the entropy weight method; or, alternatively, the order relationship and the entropy weight method can be combined to determine the weight information of each quality indicator information.

[0038] S130: Acquire tasks to be synchronized based on the data interaction mode, and synchronize the tasks to be synchronized to the target receiving end based on a preset data synchronization function.

[0039] Among them, pending synchronization tasks refer to synchronization tasks generated by data to be updated and requiring synchronization. Preset data synchronization functions can be understood as pre-set data synchronization functions. Preset data synchronization functions are used to process corresponding business data and perform front-end updates without any notice.

[0040] In an embodiment, tasks to be synchronized can be received according to the data interaction mode corresponding to the data source, and a preset data synchronization function can be extracted. The tasks to be synchronized can then be synchronized to the target receiving end according to the preset data synchronization function. In actual operation, to reduce the probability of data synchronization errors, a synchronization form can be configured for the tasks to be synchronized, and then the tasks to be synchronized can be synchronized according to the information in the synchronization form, facilitating the backtracking of synchronization tasks. Furthermore, different methods for obtaining tasks to be synchronized exist for different data interaction modes. Exemplarily, when the data interaction mode is determined to be a unidirectional data interaction mode based on data broadcasting, the log information broadcasted by the data source is received, and the temporary table data with typical discrimination fields is extracted as the task to be synchronized; when the data interaction mode is determined to be a one-way data interaction mode based on data perception, all temporary table data are traversed through the typical discrimination fields according to the preset data perception function, and the temporary table data with typical discrimination fields are taken as the task to be synchronized; when the data interaction mode is determined to be a two-way data interaction mode based on data verification, all temporary table data are verified according to the preset interaction password information, and the temporary table data that has passed the verification are traversed through the typical discrimination fields according to the preset data perception function, and the temporary table data with typical discrimination fields are taken as the task to be synchronized.

[0041] The embodiments of the present invention solve the trust black box problem existing in existing data synchronization by acquiring historical synchronization information of the data source, determining the quality indicator information of data push based on the historical synchronization information, determining the push trust of the data source based on the quality indicator information, and determining the data interaction mode of the data source according to the trust; acquiring tasks to be synchronized based on the data interaction mode, synchronizing the tasks to be synchronized to the target receiving end based on a preset data synchronization function, and determining different data interaction modes based on different push trusts, thereby improving the security and convenience of acquiring tasks to be synchronized, facilitating the rapid synchronization of tasks to be synchronized, and enhancing the user experience.

[0042] Example 2

[0043] Figure 2 This is a flow chart of a data synchronization method provided according to the second embodiment of the present invention. This embodiment is a further description of a data synchronization method based on the above embodiment. Figure 2 As shown, the method includes:

[0044] S201 , extracting the historical synchronization times of the data source, the historical synchronization success times, the historical temporary table data arrival time, the maximum synchronization time, the historical temporary table data planned arrival time and the preset time offset value as historical synchronization information.

[0045] The historical synchronization count refers to the cumulative number of synchronizations completed within a certain period. The historical one-time synchronization success count refers to the number of times the pending data was synchronized once. The historical temporary table data arrival time refers to the arrival time of historically completed temporary table data. The maximum synchronization time refers to the preset maximum time during which data synchronization can be performed. The historical temporary table data planned arrival time refers to the historically planned arrival time of temporary table data. The preset time offset refers to the preset offset for the arrival of temporary table data.

[0046] In an embodiment, the historical synchronization times, historical synchronization success times, historical temporary table data arrival time, maximum synchronization time, historical temporary table data planned arrival time and preset time offset value of the data source end can be determined, and the historical synchronization times, historical synchronization success times, historical temporary table data arrival time, maximum synchronization time, historical temporary table data planned arrival time and preset time offset value of the data source end can be used as historical synchronization information.

[0047] S202: Determine a one-time pass rate according to the number of historical synchronizations and the number of historical one-time successful synchronizations.

[0048] Among them, the first-pass rate represents the accuracy of pushing temporary table data.

[0049] In an embodiment, the number of historical synchronizations and the number of historical successful synchronizations within a time period can be determined, and the ratio of the number of historical successful synchronizations within a time period to the number of historical synchronizations can be used as the first pass rate. The calculation logic is the ratio of the number of times that the data to be pushed in the temporary table data of the jth type of data in the data source end i can be synchronized once and applied to the total number of arrivals N within a time period. The calculation formula is:

[0050] in,

[0051] S203: Determine the accuracy rate according to the historical temporary table data arrival time, the maximum synchronization time, and the historical synchronization times.

[0052] The accuracy rate indicates the timeliness of pushing temporary table data.

[0053] In one embodiment, the number of times that the arrival time of the historical temporary table data is less than the maximum synchronization time in a time period can be determined, and the ratio of the number of times that the arrival time of the historical temporary table data is less than the maximum synchronization time to the number of historical synchronizations can be used as the accuracy rate. The calculation logic is that within a period of time, the arrival time of the data to be pushed in the temporary table data of the jth type of data in the data source end i is less than the maximum synchronization time T max The ratio of the number of times to the total number of arrivals N is calculated as follows:

[0054] in

[0055] S204: Determine the plan execution aggregation rate according to the historical temporary table data arrival time, the historical temporary table data planned arrival time and the preset time offset value, and use the first-time pass rate, the accurate arrival rate and the plan execution aggregation rate as quality indicator information.

[0056] Among them, the plan execution clustering rate represents the degree of clustering between the actual arrival time of temporary table data and the planned arrival time. The higher the clustering degree, the smaller the data fluctuation and the better the long-term stability. Its calculation logic is that within a period of time, the arrival time of temporary table data of category j in data source end i is placed around the planned arrival time T plan is the center of mass, with the time offset value T d It is the ratio of the number of circles that can deviate from the radius to the total number of arrivals N, and its calculation formula is: in, After determining the first-pass rate, the accuracy rate, and the plan execution clustering rate, the first-pass rate, the accuracy rate, and the plan execution clustering rate can be used as quality indicator information. In one embodiment, the first-pass rate, the accuracy rate, and the plan execution clustering rate can be vectors.

[0057] S205 : Determine the weight coefficient of each quality indicator information based on the order relationship-entropy weight method, determine the product of each quality indicator information and the corresponding weight coefficient, and use the sum of each product as the push confidence.

[0058] Among them, the order relationship-entropy weight method refers to a method of determining weights from the order relationship analysis method and the entropy weight method, which takes into account both subjective experience judgment and objective data information.

[0059] In an embodiment, the weight coefficients of each quality indicator information can be determined according to the order relationship and entropy weight method respectively, and then the average of the two can be determined as the weight coefficient of each quality indicator information. In one embodiment, the order relationship can be used to determine the weight coefficient by forming an indicator evaluation set based on expert opinions for each type of quality indicator information, and sorting the specific quality indicator information in the evaluation set by importance to form an evaluation indicator set with a determined order relationship, that is, y1>y2>...>y M Determine the relative importance of adjacent indicators. Suppose the experts have the following evaluation indicators: j The importance value is set to ω j , then the importance ratio of two adjacent evaluation indicators can be expressed as r i =ω j-1 / ω j ,according to Can be obtained ω j-1 =ri *ω j To improve the rationality of the weight coefficient setting, we will refer to the opinions of multiple experts to obtain the order relationship of various indicator sets, and then obtain different indicator weight distribution schemes. The average of the various expert opinions is then used as the final weight result. For example, the value of ri can adopt the scale in Table 1.

[0060] Table 1 Ratio scale of relative importance

[0061] <![CDATA[Ratio scale of relative importance (r k )]]> Relative importance 1 The importance of two adjacent evaluation indicators is the same 1.2 <![CDATA[The previous indicator y j-1 is slightly more important than the subsequent indicator y j > 1.4 <![CDATA[The previous index y j-1 is significantly more important than the subsequent index y j > 1.6 <![CDATA[Previous indicator y j-1 is more important than the subsequent indicator y j Strongly important]]> 1.8 <![CDATA[Previous indicator y j-1 is extremely important compared to the subsequent indicator y j > 1.1，1.3，1.5，1.7 The median of two adjacent judgments

[0062] The entropy weight method is then used to determine the weight coefficients of each quality indicator information. This objective weighting method, based on the principle of data difference driving, can determine the indicator weight by the amount of information provided by each quality indicator information. In the entropy weight method, if the entropy value of an indicator is small, it means that the variation process of the indicator data sequence is large. The greater the role of the evaluation indicator in the entire evaluation model, the greater its weight.

[0063] For the three quality index information (first pass rate Fpy i , Rate_accuracy i and the planned execution aggregation rate Rate_aggregation i ), the above three eigenvectors can be defined as x i,j (j=1,2,3), weight coefficient k Fpy 、k rate_ac 、k rate_agg Defined as k j (j=1,2,3), the calculation steps are as follows: first, pre-process the j-th (j=1,2,3) quality indicator information, and calculate the proportion of the i-th type of data source under the j-th quality indicator information. Then calculate the entropy weight of the jth (j=1,2,3) eigenvector: Calculate the weight coefficient of the jth (j=1,2,3) eigenvector: Then, the weight coefficient of each quality index information is determined by combining the order relationship and entropy weight method. Fpy +k rate_ac +k rate_agg =1, that is, the sum of the weight coefficients of the three quality indicator information is 1. Determine the product of each quality indicator information and the corresponding weight coefficient, and use the sum of each product as the push confidence. The calculation formula is as follows:

[0064]

[0065] S206: Determine a preset trust interval to which the push trust belongs, and extract a data interaction mode corresponding to the preset trust interval as the data interaction mode at the data source end.

[0066] The data interaction mode includes at least one of the following: a one-way data interaction mode based on data broadcasting; a one-way data interaction mode based on data perception; and a two-way data interaction mode based on data verification. A one-way data interaction mode based on data broadcasting indicates that the information provided by the data provider can efficiently meet the requirements of the data recipient; a one-way data interaction mode based on data perception indicates that the information provided by the data provider basically meets the requirements of the data recipient; and a two-way data interaction mode based on data verification indicates that the information provided by the data provider does not fully meet the data requirements of the data recipient.

[0067] Among them, the preset trust interval can be understood as a pre-set interval for confirming the data interaction mode. In one embodiment, there can be three preset trust intervals, corresponding to the unidirectional data interaction mode based on data broadcasting, the unidirectional data interaction mode based on data perception, and the bidirectional data interaction mode based on data verification. Among them, the push trust corresponding to the unidirectional data interaction mode based on data broadcasting is the highest, and the push trust corresponding to the bidirectional data interaction mode based on data verification is the lowest. In one embodiment, the preset trust interval of the unidirectional data interaction mode based on data broadcasting may include but is not limited to less than 1 and greater than or equal to 0.75; the preset trust interval of the unidirectional data interaction mode based on data perception may include but is not limited to less than 0.75 and greater than or equal to 0.3; the preset trust interval of the bidirectional data interaction mode based on data verification may include but is not limited to less than 0.3 and greater than or equal to 0.

[0068] In an embodiment, a preset confidence interval for push confidence attribution may be determined, and a data interaction mode of the preset confidence interval for push confidence attribution may be used as the data interaction mode at the data source end.

[0069] S207: Determine a typical discrimination field associated with the data interaction mode and receive tasks to be synchronized according to the data interaction mode and the typical discrimination field in a preset inspection period.

[0070] Among them, the typical discrimination field refers to the field information used to distinguish the tasks to be synchronized. For each data interaction mode, the corresponding typical discrimination field is different. For example, the typical discrimination field of the single data interaction mode based on data broadcasting can be a time field (such as the timestamp of this data, the timestamp of the last data, etc.). For data that needs to be synchronized multiple times, the synchronization number field can be used as the basis for judgment (such as the planned synchronization number, the actual synchronization number, etc.). In addition to the above fields, the one-way data interaction mode based on data perception adds the judgment of data quality, such as the number of data entries (such as the seven-day average number of entries, the number of entries in this statistics, the degree of deviation, etc.), the integrity rate of key field information, etc. In addition to the above typical discrimination fields, the two-way data interaction mode based on data verification adds a protective verification link for the source end with a low trust coefficient for data push, which is generally composed of preset interaction password information (such as interaction password, handshake password, etc.) set by both parties before the data.

[0071] The preset inspection cycle refers to a pre-set cycle for automatically obtaining tasks to be synchronized, which is used to realize the cyclic execution of the two tasks of data interaction and instant synchronization. Due to the existence of synchronization status in the data interaction and data synchronization link table, the preset inspection cycle can be set to be small enough.

[0072] In an embodiment, a typical discrimination field associated with a data interaction mode may be determined, and tasks to be synchronized may be received according to the data interaction mode and the typical discrimination field in a preset inspection cycle through the typical discrimination field.

[0073] In one embodiment, receiving tasks to be synchronized according to a data interaction mode and a typical identification field in a preset inspection period includes:

[0074] When the data interaction mode is determined to be a single data interaction mode based on data broadcast, the log information broadcast by the data source is received, and temporary table data with typical discrimination fields is extracted as tasks to be synchronized;

[0075] When the data interaction mode is determined to be a one-way data interaction mode with data perception, all temporary table data is traversed through typical discrimination fields according to the preset data perception function, and temporary table data with typical discrimination fields is selected as tasks to be synchronized;

[0076] When the data interaction mode is determined to be a two-way data interaction mode based on data verification, all temporary table data are verified according to the preset interaction password information, and the temporary table data that has passed the verification is traversed through all temporary table data through typical discrimination fields according to the preset data perception function, and the temporary table data of the typical discrimination fields are used as tasks to be synchronized.

[0077] In an embodiment, for a one-way data interaction mode based on data broadcasting, when temporary table data arrives, the data source end broadcasts it in the form of writing a log, and the log information broadcast by the data source end can be received, and the temporary table data with typical discrimination fields can be extracted as tasks to be synchronized. For a one-way data interaction mode based on data perception, a preset data perception function can be set, and the target receiving end uses this function to traverse the typical fields of all temporary table data, and the temporary table data with typical discrimination fields can be used as tasks to be synchronized. For a two-way data interaction mode based on data verification, when temporary table data arrives, the data source end needs to synchronously provide preset interaction password information such as a data interaction password or other data handle password information to trigger the data target receiving end to carry out data verification. After the preset interaction password information is verified to be correct, the target receiving end uses the preset data perception function to traverse all temporary table data through the typical discrimination fields, and the temporary table data with typical discrimination fields can be used as tasks to be synchronized.

[0078] S208: Determine the data to be updated in the task to be synchronized, and set the synchronization status of the task to be synchronized to pending synchronization.

[0079] In an embodiment, the data to be updated in the task to be synchronized may be extracted, and the synchronization state of the task to be synchronized may be set to be synchronized.

[0080] S209: Determine the preset data synchronization function and the data source end of the task to be synchronized, the temporary table data name, the business type, the data interaction mode, the interaction time and the application status.

[0081] In an embodiment, a preset data synchronization function may be extracted, and the data source end, temporary table data name, business type, data interaction mode, interaction time, and application status of the task to be synchronized may be determined.

[0082] S210: Update the synchronization form according to the data source, temporary table data name, business type, data interaction mode, data to be updated, interaction time, application status, typical identification fields, and synchronization status.

[0083] The synchronization form can be understood as a table used to perform data synchronization. The synchronization status can be set to pending synchronization, synchronizing, synchronization completed, synchronization failed, etc.

[0084] In an embodiment, the synchronization form can be updated through the data source, temporary table data name, business type, data interaction mode, data to be updated, interaction time, application status, typical discrimination fields and synchronization status.

[0085] In one embodiment, the synchronization form is updated according to the data source, temporary table data name, business type, data interaction mode, data to be updated, interaction time, application status, typical identification fields, and synchronization status, including:

[0086] Update the data interaction execution status table in the synchronization form according to the data source end, temporary table data name, business type, data interaction mode, interaction time, and typical discrimination fields of the task to be synchronized;

[0087] Update the data synchronization configuration relationship table in the synchronization form according to the temporary table data name, business type, application status, and preset data synchronization function;

[0088] Update the data interaction and data synchronization link table in the synchronization form according to the temporary table data name, data to be updated, business type, and synchronization status.

[0089] Among them, the data interaction execution status table is mainly used to determine whether the temporary table data has been updated and can be applied to subsequent synchronization operations. For data sources with different interaction modes, the main differences are different typical judgment fields and different write field information; the data synchronization configuration relationship table is mainly used to determine the logical comparison relationship between temporary table data and the corresponding preset data synchronization function; the data interaction and data synchronization link table is mainly used to solve the extreme situation where multiple temporary tables arrive at the same time node at the same time, to ensure the continuous implementation of data synchronization tasks, and to avoid system freezes caused by the simultaneous triggering of multiple parallel tasks.

[0090] In an embodiment, after determining the data source, temporary table data name, business type, data interaction mode, data to be updated, interaction time, application status, typical identification fields, and synchronization status, the data source, temporary table data name, business type, data interaction mode, interaction time, and typical identification fields of the task to be synchronized can be extracted to update the data interaction execution status table in the synchronization form. The temporary table data name, business type, application status, and preset data synchronization function can also be extracted to update the data synchronization configuration relationship table in the synchronization form. The temporary table data name, data to be updated, business type, and synchronization status can also be extracted to update the data interaction and data synchronization link table in the synchronization form.

[0091] S211. Update the tasks to be synchronized based on the synchronization form and the presets.

[0092] In one embodiment, updating the tasks to be synchronized based on the synchronization form and the presets includes:

[0093] Set the synchronization status of the task to be synchronized to "synchronizing" and extract the preset data synchronization function in the data synchronization configuration relationship table in the synchronization form;

[0094] Synchronize the data to be updated in the synchronization task based on the preset data synchronization function.

[0095] In one embodiment, the synchronization status of the pending synchronization task can be set to "synchronizing," and the preset data synchronization function in the data synchronization configuration relationship table in the synchronization form can be extracted to start the corresponding synchronization task, process the corresponding business data, and perform a seamless update on the front end. In one embodiment, for tasks that are synchronizing, the remaining pending synchronization tasks can enter a waiting state and no longer enter the immediate synchronization function, ensuring the sequential access of the preset data synchronization function.

[0096] In an embodiment of the present invention, the historical synchronization times, the historical successful synchronization times, the historical temporary table data arrival time, the maximum synchronization time, the historical temporary table data planned arrival time and the preset time offset value of the historical synchronization information are extracted from the data source end, and the one-time pass rate is determined according to the historical synchronization times and the historical successful synchronization times, the accurate arrival rate is determined according to the historical temporary table data arrival time, the maximum synchronization time and the historical synchronization times, and the planned execution aggregation rate is determined according to the historical temporary table data arrival time, the historical temporary table data planned arrival time and the preset time offset value. The one-time pass rate, accurate arrival rate and planned execution aggregation rate are used as quality indicator information to achieve reasonable determination of quality indicator information; the weight coefficient of each quality indicator information is determined based on the order relationship-entropy weight method, the product of each quality indicator information and the corresponding weight coefficient is determined, the sum of each product is used as the push trust, the preset trust interval to which the push trust belongs is determined, and the data interaction mode corresponding to the preset trust interval is extracted as the data source end. The data interaction mode is used to carry out data synchronization work in a targeted manner to make the synchronization work more complete. The typical discrimination fields associated with the data interaction mode are determined. In the preset inspection cycle, the tasks to be synchronized are received according to the data interaction mode and the typical discrimination fields. The data to be updated in the tasks to be synchronized is determined. The synchronization status of the tasks to be synchronized is set to pending synchronization. The preset data synchronization function and the data source end, temporary table data name, business type, data interaction mode, interaction time and application status of the tasks to be synchronized are determined. The synchronization form is updated according to the data source end, temporary table data name, business type, data interaction mode, data to be updated, interaction time, application status, typical discrimination fields and synchronization status. The synchronization tasks to be synchronized are updated based on the synchronization form, realizing the transformation of the data processing mode from a centralized, timed processing mode to a distributed, instant processing mode, and increasing the data perception time limit from the traditional hourly level to the real-time level, realizing the on-the-go data processing, and the front-end data application can be updated without feeling, thereby optimizing the customer application experience.

[0097] In one embodiment, after obtaining the task to be synchronized based on the data interaction mode and synchronizing the task to be synchronized to the target receiving end based on the preset data synchronization function, the method further includes:

[0098] Determine the synchronization start time and synchronization end time of the task to be synchronized, update the data interaction and data synchronization link table associated with the task to be synchronized according to the synchronization start time and synchronization end time, and set the synchronization status to completed;

[0099] Update the typical discrimination field according to the synchronization end time.

[0100] The synchronization start time refers to the time when data synchronization starts, and the synchronization end time refers to the time when data synchronization ends.

[0101] In an embodiment, the synchronization start time and synchronization end time of the task to be synchronized can be determined, and the synchronization start time and synchronization end time can be synchronized and updated to the data interaction and data synchronization link table associated with the task to be synchronized, and the synchronization status can be modified to completed. Since the typical discrimination field contains a time field, the synchronization end time can be used as the last data timestamp as a typical discrimination field. In one embodiment, after the synchronization is completed, the information of each data synchronization task is written into the data perception and data synchronization link table, and the corresponding synchronization task status flag is changed to completed, so as to release subsequent tasks to be synchronized and achieve precise and targeted synchronization. In one embodiment, the typical discrimination field information of each daily synchronization data can be stored, and the one-time pass rate, accuracy rate, and plan execution aggregation rate of each push data can be recalculated according to the monthly / quarterly / yearly period. At the same time, the weights of various types of data are corrected according to the actual situation, and the dynamic iterative update of the trust coefficient of each source end is achieved, thereby optimizing the data interaction strategy and ensuring the timeliness of the data interaction mode.

[0102] Example 3

[0103] Figure 3 This is a flow chart of a data synchronization method provided according to the third embodiment of the present invention. This embodiment further illustrates a data synchronization method based on the above embodiments.

[0104] Step 1: Sort out the historical synchronization information stored in the time series-based scheduled task mode.

[0105] In one embodiment, historical record information such as the actual arrival time of each temporary table data synchronization, the scheduled task setting time, synchronization failure events, the maximum number of temporary tables carried before the daily scheduled task, and the longest synchronization time can be collected when the scheduled task mode is based on the time series, so as to prepare data for the subsequent determination of the characteristic vector (quality indicator information) representing the trust level.

[0106] Step 2: Determine quality indicator information that characterizes the push quality of each data source based on historical synchronization information.

[0107] For the data receiver, during the data synchronization process, a data source will push multiple types of data. However, no matter what type of data, it is necessary to focus on the security, accuracy, punctuality and long-term stability of the data push. Considering that the security of data push is a factor to be considered in the process of system development and application, it will not be analyzed separately in this invention. For three of these characteristics, combined with the historical synchronization information stored in the time series-based scheduled task mode, the one-time pass rate of data push Fpy is calculated. i,j , Rate_accuracy i,j , Plan execution aggregation rate Rate_aggregation i,j etc. as the feature vectors of the above characteristics.

[0108] Step 3: Determine the data interaction mode that should be adopted by each data source.

[0109] (1) Determine the trust factor (trust level) of data push from each source end.

[0110] Define the trust coefficient of data push for data source i as Trust_coefficient i , the data push trust coefficient for the j-th type of data in data source i is Trust_coefficient i,j Where k Fpy 、k rate_ac 、k rate_agg are three eigenvectors Fpy i 、Rate_accuracy i 、Rate_aggregation i The weight coefficient reflects the degree of attention or importance of the three characteristics of data one-time pass rate, accuracy rate, and plan execution aggregation rate in data source i; ω j is the weight coefficient of the jth type of data in data source i, reflecting the degree of attention or importance of a certain type of data in the data source.

[0111] (2) Calculation of weight coefficient

[0112] The weight coefficient can generally be calculated using two methods: subjective experience method and objective evaluation method. The subjective experience method includes expert experience, hierarchical analysis method, sequence relationship method, etc.; the objective evaluation method includes entropy weight method, etc.

[0113] The weight of each type of data to be pushed ω j It is highly correlated with current key tasks, work arrangements of superior units and other factors, and has strong timeliness and guidance. It can be obtained using subjective evaluation methods. The present invention can use the calculation of weight coefficients based on sequence relationships.

[0114] The weight coefficient k of the three feature vectors representing the push characteristics Fpy , k rate_ac , k rate_agg The stability is high and can be calculated by objective evaluation method. The present invention can also use the calculation of data push feature vector weight based on entropy weight method.

[0115] (3) Calculating the evaluation results of each data source

[0116] Through the above algorithm, k can be calculated Fpy , k rate_ac , k rate_agg and the weights ω of various data j , thereby calculating the trust coefficient of a certain data source i. From the calculation steps, it can be seen that the data push trust evaluation result established by this algorithm is a weighted combination of the push data type weight and the feature vector weight, while taking into account the influence of expert experience and objective data, ensuring the scientificity and reliability of the trust evaluation.

[0117] (4) Selecting a data interaction mode based on evaluation results

[0118] The present invention sets the range of data push trust coefficient values ​​into high trust, medium trust, and low trust intervals. By calculating the values ​​of the trust coefficients of different data sources, the intervals they fall into are determined, and different real-time synchronization strategies are set for different trust intervals. The classification of data push trust evaluation results is shown in Table 2:

[0119] Table 2 Classification of data push trust evaluation results

[0120]

[0121] Step 4: Generate synchronization forms based on different data interaction modes and subsequent synchronization operations.

[0122] The forms required for data interaction and subsequent synchronization operations based on different data sources mainly include the data interaction execution status table, the data synchronization configuration relationship table, and the data interaction and data synchronization link table.

[0123] (1) Data interaction execution status table

[0124] The data interaction execution status table is primarily used to determine whether temporary table data has been updated and is available for subsequent synchronization operations. This table primarily includes information such as the ID, data source, interaction mode, temporary table name, business type, typical identification fields, and interaction time. Data sources with different interaction modes differ primarily in their typical identification fields and the field information they write.

[0125] The typical discrimination field of the one-way data interaction mode based on data broadcasting can be the time field (such as the timestamp of this data, the timestamp of the last data, etc.). For data that needs to be synchronized multiple times, the synchronization count field can be used as the basis for discrimination (such as the planned synchronization count, the actual synchronization count, etc.). In addition to the above fields, the one-way data interaction mode based on data perception adds fields for judging data quality, such as the number of data entries (such as the seven-day average number of entries, the number of entries in this statistics, the degree of deviation, etc.), the integrity rate of key field information, etc. In addition to the above typical discrimination fields, the two-way data interaction mode based on data verification adds a protective verification link for the source end with a low trust coefficient for data push, which is generally composed of preset interaction password information (such as interaction password, handshake password, etc.) set by both parties to the data.

[0126] (2) Data synchronization configuration relationship table

[0127] The data synchronization configuration relationship table is mainly used to determine the logical comparison relationship between temporary table data and the corresponding synchronization function. It mainly includes the ID, temporary table name, business type, application status, corresponding preset data synchronization function, etc.

[0128] When the same synchronization function updates data across multiple temporary tables, the name of the temporary table whose data arrives last generally prevails. By establishing synchronization configuration logic, you can divide the full temporary data push tables from multiple source terminals on the platform into multiple node tasks, significantly reducing the maximum number of synchronized temporary tables on a single node.

[0129] (3) Data interaction and data synchronization link table

[0130] The data interaction and data synchronization link table is primarily used to address the extreme situation where multiple temporary tables arrive at the same time. This ensures the continuous execution of data synchronization tasks and avoids system lags caused by the simultaneous triggering of multiple parallel tasks. This table primarily contains fields such as the ID, temporary table name, business type, synchronization information, synchronization status, synchronization start time, and synchronization end time. The synchronization status flag can be set to pending synchronization, synchronizing, synchronization completed, or synchronization failed.

[0131] Step 5: Complete the preparation work for data synchronization on multiple data sources.

[0132] In an embodiment, it is possible to determine the data to be updated of all source data, fill in the information that needs to be manually entered in advance in three data forms, including the data to be updated and the corresponding temporary data table, typical discrimination fields, etc., to ensure that the synchronization plan is not missed and complete the corresponding data preparation work.

[0133] Step 6: Start the automatic inspection task.

[0134] In one embodiment, a preset inspection cycle can be set as needed to initiate an automatic inspection task. The self-inspection task is a scheduled task trigger, primarily used to implement the cyclic execution of data interaction and real-time synchronization tasks. Due to the presence of synchronization status in the data interaction and data synchronization link tables, the inspection cycle of the self-inspection task can be set to be sufficiently short. Compared to traditional time-reduced scheduled task models, the self-inspection task only calls the synchronization function for tasks to be synchronized, rather than calling the full synchronization function. This significantly reduces algorithm complexity and redundancy, effectively preventing program idling.

[0135] Step 7: Data interaction triggering.

[0136] In one embodiment, for a one-way data interaction mode based on data broadcasting, when temporary table data arrives, the data provider broadcasts it by writing a log, updates the latest information in the execution status table field, and sets the synchronization status corresponding to the temporary table to be updated to pending synchronization.

[0137] For the one-way data interaction mode based on data perception, a data perception function is set. The data recipient uses this function to traverse the typical fields of all temporary tables corresponding to the interaction mode in the data execution status table, update and improve the latest field information, and determine whether there is any temporary table update. If there is an update, the synchronization status corresponding to the temporary table to be updated is set to pending synchronization.

[0138] For the two-way data interaction mode based on data verification, when the temporary table data arrives, the data provider is required to synchronously provide the data interaction password or other data handle password information to trigger the data receiver to perform data verification. After the password information is verified to be correct, the data receiver uses the perception function to update the latest field information for the corresponding temporary table to determine whether the temporary table is to be synchronized. If so, the synchronization status corresponding to the temporary table to be updated is set to pending synchronization.

[0139] Step 8: Connect the tasks to be synchronized sequentially.

[0140] In one embodiment, the state corresponding to the temporary table that first enters the preset data synchronization function can be set to synchronized. For tasks that are synchronized, the remaining tasks to be synchronized enter the waiting state and no longer enter the immediate synchronization function, ensuring the sequential access of the synchronization function. Among them, the preset data synchronization function is an immediate synchronization function.

[0141] Step 9: Accurately synchronize data based on synchronization status flag constraints.

[0142] In one embodiment, a synchronization function can be applied in conjunction with the data synchronization configuration relationship table to initiate the corresponding synchronization task, process the corresponding business data, and perform a seamless front-end update. After synchronization is complete, the information about each data synchronization task is written to the data perception and data synchronization link table, and the corresponding synchronization task status flag is changed to completed, allowing subsequent tasks to be synchronized to proceed, achieving precise and targeted synchronization.

[0143] Step 10: Reset the typical discrimination field information.

[0144] In one embodiment, when the daily synchronization task is completed, the data reset phase is entered, and information such as the typical judgment fields in the data perception execution status table is updated or reset (such as writing the current data timestamp content into the previous data timestamp field, setting the current data timestamp field to empty, etc.), and when entering a new daily cycle, the logical judgment is re-performed.

[0145] Step 11: Dynamically iterate and update the trust coefficient of each source based on data synchronization.

[0146] In one embodiment, typical discrimination field information of each daily synchronized data can be stored, and the one-time pass rate, accuracy rate, and plan execution aggregation rate of each pushed data can be recalculated according to monthly / quarterly / annual periods. At the same time, the weights of various types of data can be corrected according to actual conditions to achieve dynamic iterative updates of the trust coefficients of each source end, thereby optimizing the data interaction strategy and ensuring the timeliness of the data interaction mode.

[0147] This embodiment introduces the concept of trust coefficient, and takes the accuracy, punctuality and long-term stability in the data push process as feature vectors to solve the trust black box problem existing in existing data synchronization; the data type weights in the trust function are weighted using the order relationship, and the feature vector weights are weighted using the entropy weight method. Through weighted synthesis, the influence of expert experience and objective data is taken into account at the same time to ensure the scientific nature and reliability of the trust evaluation; according to different trust coefficients, two-way interactive or one-way interactive data synchronization schemes are set to construct an interactive data synchronization mode; through the program design of "three data forms, two execution functions, and one self-inspection task", the association relationship between data interaction and data synchronization is established, and the data processing mode is transformed from a centralized, timed processing mode to a distributed, real-time processing mode, solving the problem of poor matching of existing data synchronization and optimizing the front-end customer application perception.

[0148] Example 4

[0149] Figure 4 FIG. 1 is a structural diagram of a data synchronization device provided according to a fourth embodiment of the present invention. Figure 4 As shown, the device includes: an information determination module 41, a mode determination module 42 and a data synchronization module 43.

[0150] The information determination module 41 is used to obtain historical synchronization information of the data source and determine the quality indicator information of the data push based on the historical synchronization information;

[0151] A mode determination module 42 is configured to determine the push trustworthiness of the data source end based on the quality indicator information, and determine the data interaction mode of the data source end according to the trustworthiness;

[0152] The data synchronization module 43 is configured to obtain tasks to be synchronized based on the data interaction mode, and synchronize the tasks to be synchronized to a target receiving end based on a preset data synchronization function.

[0153] The technical solution of the embodiment of the present invention obtains historical synchronization information of the data source end through the information determination module, determines the quality indicator information of the data push based on the historical synchronization information, and the mode determination module determines the push trust of the data source end based on the quality indicator information, and determines the data interaction mode of the data source end according to the trust, thereby solving the trust black box problem existing in the existing data synchronization; the data synchronization module obtains the task to be synchronized based on the data interaction mode, and synchronizes the task to be synchronized to the target receiving end based on the preset data synchronization function. By determining different data interaction modes based on different push trusts, the security and convenience of obtaining the task to be synchronized are improved, the rapid synchronization of the task to be synchronized is facilitated, and the user experience is enhanced.

[0154] In one embodiment, the information determination module 41 includes:

[0155] An information extraction unit is used to extract the historical synchronization times of the data source, the historical synchronization success times, the historical temporary table data arrival time, the maximum synchronization time, the historical temporary table data planned arrival time, and the preset time offset value as historical synchronization information;

[0156] A pass rate determination unit is used to determine a pass rate according to the number of historical synchronizations and the number of historical synchronization successes;

[0157] An accuracy rate determination unit, configured to determine an accuracy rate according to a historical temporary table data arrival time, a maximum synchronization time, and a historical synchronization count;

[0158] The information determination unit is used to determine the plan execution aggregation rate according to the historical temporary table data arrival time, the historical temporary table data planned arrival time and the preset time offset value, and use the first-time pass rate, accurate arrival rate and plan execution aggregation rate as quality indicator information.

[0159] In one embodiment, the mode determination module 42 includes:

[0160] A trust degree determination unit is used to determine the weight coefficient of each quality indicator information based on the order relationship-entropy weight method, determine the product of each quality indicator information and the corresponding weight coefficient, and use the sum of each product as the push trust degree;

[0161] An interaction mode determination unit is used to determine a preset trust interval to which the push trust belongs, and extract a data interaction mode corresponding to the preset trust interval as the data interaction mode at the data source end; wherein the data interaction mode includes at least one of the following: a one-way data interaction mode based on data broadcasting; a one-way data interaction mode based on data perception; a two-way data interaction mode based on data verification.

[0162] In one embodiment, the data synchronization module 43 includes:

[0163] A synchronization task receiving unit, configured to determine a typical discrimination field associated with a data interaction mode and receive tasks to be synchronized according to the data interaction mode and the typical discrimination field in a preset inspection cycle;

[0164] A state modification unit, configured to determine the data to be updated in the task to be synchronized, and set the synchronization state of the task to be synchronized to pending synchronization;

[0165] An information determination unit is used to determine the preset data synchronization function and the data source end of the task to be synchronized, the temporary table data name, the business type, the data interaction mode, the interaction time and the application status;

[0166] The form update unit is used to update the synchronization form according to the data source, temporary table data name, business type, data interaction mode, data to be updated, interaction time, application status, typical discrimination fields and synchronization status;

[0167] The data synchronization unit is used to update the synchronization tasks based on the synchronization form and presets.

[0168] In one embodiment, the synchronization task receiving unit is specifically configured to:

[0169] When the data interaction mode is determined to be a single data interaction mode based on data broadcast, the log information broadcast by the data source is received, and temporary table data with typical discrimination fields is extracted as tasks to be synchronized;

[0170] When the data interaction mode is determined to be a one-way data interaction mode with data perception, all temporary table data is traversed through typical discrimination fields according to the preset data perception function, and temporary table data with typical discrimination fields is selected as tasks to be synchronized;

[0171] When the data interaction mode is determined to be a two-way data interaction mode based on data verification, all temporary table data are verified according to the preset interaction password information, and the temporary table data that has passed the verification is traversed through all temporary table data through typical discrimination fields according to the preset data perception function, and the temporary table data of the typical discrimination fields are used as tasks to be synchronized.

[0172] In one embodiment, the form updating unit is specifically configured to:

[0173] Update the data interaction execution status table in the synchronization form according to the data source end, temporary table data name, business type, data interaction mode, interaction time, and typical discrimination fields of the task to be synchronized;

[0174] Update the data synchronization configuration relationship table in the synchronization form according to the temporary table data name, business type, application status, and preset data synchronization function;

[0175] Update the data interaction and data synchronization link table in the synchronization form according to the temporary table data name, data to be updated, business type, and synchronization status.

[0176] In one embodiment, the data synchronization unit is specifically configured to:

[0177] Set the synchronization status of the task to be synchronized to "synchronizing" and extract the preset data synchronization function in the data synchronization configuration relationship table in the synchronization form;

[0178] Synchronize the data to be updated in the synchronization task based on the preset data synchronization function.

[0179] In one embodiment, the data synchronization method further includes:

[0180] The data update module is used to determine the synchronization start time and synchronization end time of the task to be synchronized, update the data interaction and data synchronization link table associated with the task to be synchronized according to the synchronization start time and synchronization end time, and set the synchronization status to completed;

[0181] The field update module is used to update the typical discrimination field according to the synchronization end time.

[0182] The data synchronization device provided in the embodiment of the present invention can execute the data synchronization method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0183] Example 5

[0184] Figure 5Schematic diagram of the structure of an electronic device that implements the data synchronization method of an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0185] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0186] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0187] The processor 11 may be a variety of general and / or specialized processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the data synchronization method.

[0188] In some embodiments, the data synchronization method can be implemented as a computer program that is tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data synchronization method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the data synchronization method in any other suitable manner (e.g., by means of firmware).

[0189] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0190] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0191] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0192] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0193] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0194] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0195] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0196] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A data synchronization method, characterized in that: include: Acquire historical synchronization information of a data source, and determine quality indicator information of data push based on the historical synchronization information; Determine the push trust of the data source end based on the quality indicator information, and determine the data interaction mode of the data source end according to the trust; The tasks to be synchronized are acquired based on the data interaction mode, and the tasks to be synchronized are synchronized to a target receiving end based on a preset data synchronization function.

2. The method according to claim 1, characterized in that The acquiring of historical synchronization information of the data source, and determining quality indicator information of data push based on the historical synchronization information, includes: Extract the historical synchronization times, historical synchronization success times, historical temporary table data arrival time, maximum synchronization time, historical temporary table data planned arrival time, and preset time offset value of the data source as historical synchronization information; Determine a one-time pass rate according to the historical synchronization times and the historical one-time synchronization success times; Determine the accuracy rate according to the arrival time of the historical temporary table data, the maximum synchronization time and the number of historical synchronizations; The plan execution aggregation rate is determined according to the arrival time of the historical temporary table data, the planned arrival time of the historical temporary table data and the preset time offset value, and the one-time pass rate, the accurate arrival rate and the plan execution aggregation rate are used as quality indicator information.

3. The method according to claim 1, characterized in that The step of determining the push trust of the data source end based on the quality indicator information, and determining the data interaction mode of the data source end according to the trust, includes: Determine the weight coefficient of each quality indicator information based on the order relationship-entropy weight method, determine the product of each quality indicator information and the corresponding weight coefficient, and use the sum of each product as the push trust; Determine the preset trust interval to which the push trust belongs, and extract the data interaction mode corresponding to the preset trust interval as the data interaction mode at the data source end; wherein the data interaction mode includes at least one of the following: a one-way data interaction mode based on data broadcasting; a one-way data interaction mode based on data perception; a two-way data interaction mode based on data verification.

4. The method according to claim 1, characterized in that The acquiring the task to be synchronized based on the data interaction mode, and synchronizing the task to be synchronized to the target receiving end based on a preset data synchronization function, includes: Determine a typical discrimination field associated with the data interaction mode and receive tasks to be synchronized according to the data interaction mode and the typical discrimination field in a preset inspection cycle; Determine the data to be updated in the task to be synchronized, and set the synchronization state of the task to be synchronized to pending synchronization; Determine the preset data synchronization function and the data source end, temporary table data name, business type, data interaction mode, interaction time and application status of the task to be synchronized; Update the synchronization form according to the data source, the temporary table data name, the business type, the data interaction mode, the data to be updated, the interaction time, the application status, the typical identification field and the synchronization status; The tasks to be synchronized are updated based on the synchronization form and the presets.

5. The method according to claim 4, characterized in that The receiving the task to be synchronized according to the data interaction mode and the typical identification field in the preset inspection cycle includes: When it is determined that the data interaction mode is a single data interaction mode based on data broadcast, receiving log information broadcasted by the data source end, and extracting temporary table data containing the typical discrimination field as tasks to be synchronized; When it is determined that the data interaction mode is a data-aware one-way data interaction mode, all temporary table data are traversed through typical discrimination fields according to a preset data perception function, and temporary table data with the typical discrimination fields are taken as tasks to be synchronized; When it is determined that the data interaction mode is a two-way data interaction mode based on data verification, all temporary table data are verified according to the preset interaction password information, and the temporary table data that have passed the verification are traversed through all temporary table data through typical discrimination fields according to the preset data perception function, and the temporary table data of the typical discrimination fields are used as tasks to be synchronized.

6. The method according to claim 4, characterized in that The updating of the synchronization form according to the data source, the temporary table data name, the business type, the data interaction mode, the data to be updated, the interaction time, the application status, the typical discrimination field and the synchronization status includes: Update the data interaction execution status table in the synchronization form according to the data source end of the task to be synchronized, the temporary table data name, the business type, the data interaction mode, the interaction time and the typical discrimination field; Update the data synchronization configuration relationship table in the synchronization form according to the temporary table data name, the business type, the application status and the preset data synchronization function; The data interaction and data synchronization link table in the synchronization form are updated according to the temporary table data name, the data to be updated, the business type and the synchronization status.

7. The method according to claim 4, characterized in that The updating of the task to be synchronized based on the synchronization form and the preset includes: The synchronization state of the task to be synchronized is set to being synchronized, and a preset data synchronization function in the data synchronization configuration relationship table in the synchronization form is extracted; The data to be updated in the task to be synchronized is synchronized based on the preset data synchronization function.

8. The method according to claim 1, characterized in that After acquiring the task to be synchronized based on the data interaction mode and synchronizing the task to be synchronized to the target receiving end based on a preset data synchronization function, the method further includes: Determine the synchronization start time and synchronization end time of the task to be synchronized, update the data interaction and data synchronization link table associated with the task to be synchronized according to the synchronization start time and synchronization end time, and set the synchronization status to completed; The typical discrimination field is updated according to the synchronization end time.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data synchronization method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data synchronization method according to any one of claims 1 to 7 when executed.

Images