A data synchronization transmission system and method for heterogeneous databases

Abstract

The present invention discloses a data synchronization transmission system and method for heterogeneous databases, and belongs to the technical field of data transmission. The present invention comprises: S10: storing each target incremental data obtained after processing, and the position information of each sub-incremental data in each target incremental data in the data to be changed in the temporary data storage area of ​​the corresponding sub-database; S20: predicting the transmission similarity between two target incremental data; S30: analyzing the data synchronization transmission situation of each sub-database; S40: analyzing the transmission time of each target incremental data. The present invention calculates the transmission similarity between the two target incremental data through the characteristic vector of the target incremental data of each sub-database, and realizes the calculation of the data transmission time difference between the two target incremental data, which is conducive to realizing the effective analysis of the synchronization transmission situation of the target incremental data stored in the temporary data storage area of ​​each sub-database.

Description

Technical Field

[0001] The present invention relates to the technical field of data transmission, and in particular to a data synchronization transmission system and method for heterogeneous databases. Background Art

[0002] Heterogeneous data refers to data of different types, different versions or different structures. A heterogeneous database system is a collection of multiple related database systems that already existed before joining the heterogeneous database system, and each database system maintains its own autonomy while achieving data sharing and transparent access.

[0003] In the prior art, the data to be updated is usually overwritten with the data to be synchronized, but this method requires all the data to be synchronized to be transmitted, which reduces the update rate of each sub-database for the updated data. In addition, since the structures of the data stored in each sub-database are different, it is impossible to directly analyze the data synchronization transmission status of each sub-database, which increases the processing time of the data to be synchronized. Summary of the invention

[0004] The object of the present invention is to provide a data synchronization transmission system and method for heterogeneous databases to solve the problems raised in the above background technology.

[0005] In order to solve the above technical problems, the present invention provides the following technical solution: a data synchronization transmission method for heterogeneous databases, the method comprising:

[0006] S10: Capture the data to be synchronized in the source database, compare the captured data to be synchronized with the data to be changed stored in each sub-database in the heterogeneous database, obtain the target incremental data of each sub-database, process the target incremental data of each sub-database, and store the target incremental data obtained after the processing and the position information of each sub-incremental data in the data to be changed in each target incremental data in the temporary data storage area of ​​the corresponding sub-database;

[0007] S20: extracting features of the target incremental data stored in each temporary data storage area to obtain a feature vector of each target incremental data, and predicting the transmission similarity between two target incremental data based on the obtained feature vector;

[0008] S30: Analyze the data synchronization transmission status of each sub-database;

[0009] S40: Determine the transmission time of the target incremental data corresponding to the target feature vector according to the data synchronization boundary time of each sub-database, and analyze the transmission time of other target incremental data based on the determination result.

[0010] Further, the S10 includes:

[0011] A sub-database is randomly selected from the sub-databases of the heterogeneous database, and the captured data to be synchronized is compared with the data to be changed stored in the selected sub-database. In the data to be synchronized, the data to be added in the data to be changed stored in the selected sub-database is marked, and the marked data is deleted from the data to be synchronized;

[0012] The data to be synchronized after the deletion process is compared with the data to be changed stored in the selected sub-database, and the inconsistent data is marked in the data to be synchronized after the deletion process. The set of each group of deletion mark data and each mark data stored in the data to be synchronized after the deletion process is the target incremental data of the selected sub-database, and the number of bits of each mark data in the data to be changed stored in the selected sub-database and the number of associated bits of each group of deletion mark data are obtained;

[0013] According to the data format of the data stored in the selected sub-database in the heterogeneous database, the target incremental data of the selected sub-database is processed, and based on the processing result, the target incremental data of the selected sub-database and the number of bits of each sub-incremental data in the target incremental data of the selected sub-database in the data to be changed stored in the selected sub-database or the number of associated bits of each sub-incremental data are stored in the temporary data storage area of ​​the selected sub-database. The data stored in the temporary data storage area is:

[0014] {(F1,max(U1,Z1)),(F2,max(U2,Z2)),…,(F b ,max(U b ,Z b ))}, where F b Indicates the value corresponding to the bth sub-incremental data in the target incremental data, U b Indicates the number of bits of the b-th incremental data in the data to be updated stored in the selected sub-database, Z b Indicates the number of associated digits of the b-th sub-incremental data, b indicates the total number of sub-incremental data contained in the target incremental data of the selected sub-database. If the b-th sub-incremental data has associated digits, then U b =0, Z b =The number of associated bits of the b-th sub-incremental data. If there is no associated bit number for the b-th sub-incremental data, then U b =The number of bits of the b-th sub-incremental data in the data to be updated stored in the selected sub-database, Z b =0, max indicates the maximum value symbol;

[0015] Repeat the above operation to determine the data stored in the temporary data storage area of ​​the remaining sub-databases in the heterogeneous database except the selected sub-database. By determining the target incremental data and the number of bits of each sub-incremental data in the target incremental data in the data to be changed or the number of associated bits of each sub-incremental data, the content to be changed of each sub-database is determined, which is conducive to reducing the data update time of each sub-database.

[0016] Further, the S20 includes:

[0017] S201: Perform numbering processing on each sub-database, and the numbering processing result is: i=1, 2, ..., n; n represents the total number of numbers, and the data stored in the temporary data storage area of ​​the sub-database numbered i is: {(F 1i→I ,max(U 1i→i ,Z 1i→i )),(F 2i→I ,max(U 2i→i ,Z 2i→i )),…,(F di→I ,max(U di→i ,Z di→i ))}, where di represents the total number of sub-incremental data contained in the target incremental data of the sub-database numbered i, and F di→i Indicates the value corresponding to the dth sub-incremental data in the target incremental data of the sub-database numbered i, U di→i The number of bits of the dth incremental data in the target incremental data of the sub-database numbered i in the data to be changed stored in the sub-database numbered i, Z di→i Indicates the number of bits associated with the dth sub-incremental data in the target incremental data of the sub-database numbered i;

[0018] According to max(U ji→i ,Z ji→i ) in order from small to large, for {(F 1i→I ,max(U 1i→i ,Z 1i→i )),(F 2i→I ,max(U 2i→i ,Z 2i→i )),…,(F di→I ,max(U di→i ,Z di→i ))}, wherein j=1, 2, ..., d, indicating that each sub-incremental data is numbered according to the recording order of the target incremental data, and d indicates the total number of sub-incremental data included in the target incremental data;

[0019] S202: Construct a feature vector model Q, where Q = {W1, W2, ..., W x},W xrepresents the value corresponding to the xth component in the feature vector model, x represents the maximum value of the total number of data to be changed stored in each sub-database, k = 1, 2, ..., x, represents the number corresponding to each component in the feature vector model, if there is k = max(U ji→i ,Z ji→i ) or k=max(U ji→i ,Z ji→i )+0.5, then W k =1, if k = max(U ji→i ,Z ji→i ) and k=max(U ji→i ,Z ji→i )+0.5 do not exist, then W k =0,W k Represents the value corresponding to the kth component in the feature vector model, realizing the unified processing of the data stored in each temporary data storage area, and the feature vector Q of the target incremental data of the sub-database numbered i i = {W 1i ,W 2i ,…,W xi}; The characteristic vector of each target incremental data is determined by the number of bits of each sub-incremental data in the target incremental data in the data to be changed, and the characteristic vector only contains data 1 and data 0, which is conducive to improving the calculation rate of the transmission similarity between the two target incremental data;

[0020] S203: Randomly select a feature vector as the target feature vector, according to The transmission similarity between the target incremental data corresponding to the target feature vector and the target incremental data corresponding to the feature vector numbered j is calculated, where j = 1, 2, ..., m, represents the number corresponding to each feature vector except the target feature vector, m represents the total number of feature vectors except the target feature vector, X jk Y represents the value corresponding to the kth component in the eigenvector numbered j, k Represents the value corresponding to the kth component in the target feature vector. The data transmission time difference between the two target incremental data is calculated by the transmission similarity between the two target incremental data, which is conducive to the effective analysis of the synchronous transmission of the target incremental data stored in the temporary data storage area of ​​each sub-database.

[0021] Further, the S30 includes:

[0022] S301: Determine the target incremental data corresponding to the feature vector numbered j, and set up a transmission channel length L between the temporary data storage area storing the determined target incremental data and the sub-database corresponding to the determined target incremental data. j, and the transmission rate V of the data in the transmission channel established between the temporary data storage area storing the determined target incremental data and the sub-database corresponding to the determined target incremental data j To obtain, according to The time for the target incremental data corresponding to the target feature vector to be transmitted to the corresponding sub-database is calculated. V represents the transmission rate of the data in the transmission channel between the temporary data storage area storing the target incremental data corresponding to the target feature vector and the sub-database corresponding to the target feature vector. L represents the length of the transmission channel between the temporary data storage area storing the target incremental data corresponding to the target feature vector and the sub-database corresponding to the target feature vector. h represents the data length value of one bit of data. R k Indicates the total number of data stored in the sub-incremental data corresponding to the number of bits corresponding to the kth component in the target feature vector;

[0023] According to E j =T*(1-R j )-T′*(1-R j ), The transmission time difference between the target incremental data corresponding to the target feature vector and the target incremental data corresponding to the feature vector numbered j is calculated, V j represents the transmission rate of data in the transmission channel between the temporary data storage area of ​​the target incremental data corresponding to the feature vector with the storage number j and the corresponding sub-database, R kj Indicates the total number of data stored in the sub-incremental data corresponding to the number of bits corresponding to the kth component in the feature vector numbered j; through the transmission similarity between the two target incremental data, the synchronous transmission status of each target incremental data is analyzed, without the need to perform complex calculations on the synchronous transmission time of each target incremental data one by one, which is conducive to ensuring that each sub-database completes the data update before the data synchronization boundary time;

[0024] If E j > 0, it means that the target incremental data corresponding to the feature vector numbered j is transmitted before the target incremental data corresponding to the target feature vector is transmitted. j Time starts to transfer;

[0025] If E j <0, it means the target incremental data corresponding to the feature vector numbered j is transmitted in the target incremental data corresponding to the target feature vector E j Start transmission after time;

[0026] If E j =0, it means that the target incremental data corresponding to the feature vector numbered j has the same transmission time as the target incremental data corresponding to the target feature vector.

[0027] Furthermore, the step S40 includes: performing synchronization on the data synchronization limit time value G of the sub-database numbered i. i To obtain, the transmission time of the target incremental data corresponding to the target feature vector is minG i -T, according to U j =minG i -TE j The transmission time of the target incremental data corresponding to the feature vector numbered j is determined.

[0028] A data synchronization transmission system for heterogeneous databases, the system comprising a temporary data storage area data storage module, a transmission similarity prediction module, a data synchronization transmission situation analysis module and a transmission time analysis module;

[0029] The data storage module in the temporary data storage area is used to store the target incremental data obtained after processing, and the position information of each sub-incremental data in each target incremental data in the data to be changed in the temporary data storage area of ​​the corresponding sub-database;

[0030] The transmission similarity prediction module is used to predict the transmission similarity between two target incremental data;

[0031] The data synchronization transmission situation analysis module is used to analyze the data synchronization transmission situation of each sub-database;

[0032] The transmission time analysis module is used to analyze the transmission time of each target incremental data.

[0033] Further, the data storage module of the temporary data storage area includes a target incremental data determination unit and a transmission data determination unit;

[0034] The target incremental data determination unit randomly selects a sub-database from the sub-databases of the heterogeneous database, compares the captured data to be synchronized with the data to be changed stored in the selected sub-database, deletes the data to be synchronized based on the comparison result, performs a secondary comparison on the deleted data to be synchronized with the data to be changed stored in the selected sub-database, and determines the target incremental data of the selected sub-database based on the secondary comparison result;

[0035] The transmission data determination unit processes the target incremental data of the selected sub-database, and based on the processing result, stores the target incremental data of the selected sub-database and the number of bits of each sub-incremental data in the target incremental data of the selected sub-database in the data to be changed stored in the selected sub-database or the number of associated bits of each sub-incremental data in the temporary data storage area of ​​the selected sub-database, and determines the transmission data stored in the temporary data storage area.

[0036] Further, the transmission similarity prediction module includes an adjustment unit, a feature vector acquisition unit and a transmission similarity prediction unit;

[0037] The adjusting unit adjusts the storage position of each data stored in the temporary data storage area of ​​the sub-database corresponding to the target incremental data according to the number of bits of each sub-incremental data in the target incremental data in the data to be changed stored in the sub-database corresponding to the target incremental data and the number of bits associated with each sub-incremental data in the target incremental data;

[0038] The feature vector acquisition unit receives the adjustment result transmitted by the adjustment unit, and determines the feature vector of each target incremental data in combination with the constructed feature vector model;

[0039] The transmission similarity prediction unit predicts the transmission similarity between the target incremental data corresponding to the target feature vector and the target incremental data corresponding to other feature vectors through the constructed prediction model.

[0040] Further, the data synchronization transmission situation analysis module includes a data transmission time calculation unit, a data transmission time difference calculation unit and a data synchronization transmission situation analysis unit;

[0041] The data transmission time calculation unit calculates the time for transmitting the target incremental data corresponding to the target feature vector to the corresponding sub-database according to the constructed mathematical model;

[0042] The data transmission time difference calculation unit calculates the transmission time difference between the target incremental data corresponding to the target feature vector and the target incremental data corresponding to other feature vectors;

[0043] The data synchronous transmission situation analysis unit receives the transmission time difference value transmitted by the data transmission time difference calculation unit, compares the received data with 0, and determines the synchronous transmission situation of the target incremental data corresponding to other feature vectors according to the comparison result.

[0044] Further, the transmission time analysis module includes a transmission time analysis unit 1 and a transmission time analysis unit 2;

[0045] The transmission time analysis unit obtains the data synchronization boundary time of each sub-database, and determines the transmission time of the target incremental data corresponding to the target feature vector in combination with the calculation result transmitted by the data transmission time calculation unit;

[0046] The second transmission time analysis unit receives the transmission time transmitted by the first transmission time analysis unit, and determines the transmission time of the target incremental data corresponding to other characteristic vectors in combination with the calculation result transmitted by the data transmission time difference calculation unit.

[0047] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0048] 1. The present invention searches for the target incremental data of each sub-database, and calculates the transmission similarity between the two target incremental data through the characteristic vector of the target incremental data. The characteristic vector only contains data 1 and data 0, which is beneficial to improving the calculation rate of the transmission similarity between the two target incremental data. By calculating the transmission similarity, the data transmission time difference between the two target incremental data is calculated, which is beneficial to effectively analyze the synchronous transmission status of the target incremental data stored in the temporary data storage area of ​​each sub-database.

[0049] 2. The present invention determines the content to be changed in each sub-database by using the target incremental data and the number of bits of each sub-incremental data in the target incremental data in the data to be changed or the number of associated bits of each sub-incremental data. Compared with the method of overwriting the data to be updated with the data to be synchronized, the present invention reduces the amount of updated data for the data to be updated in each sub-database, which is beneficial to improving the update rate of the data to be updated in each sub-database.

[0050] 3. The present invention does not need to perform complex calculations on the time required for transmission of each target incremental data one by one, nor does it need to analyze the start transmission time of each incremental data one by one based on the calculated transmission time. The present invention only needs to analyze the start transmission time of the target incremental data corresponding to the target feature vector, and the start transmission time of other target incremental data is determined by the data transmission time difference between the target incremental data corresponding to the target feature vector and other target incremental data. The determination process is relatively simple, which improves the efficiency of data synchronization transmission analysis of heterogeneous databases. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0052] Figure 1 It is a schematic diagram of the working process of a data synchronization transmission system and method for heterogeneous databases of the present invention;

[0053] Figure 2 It is a schematic diagram of the working principle structure of a data synchronization transmission system and method for heterogeneous databases of the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] See also Figure 1 and Figure 2 The present invention provides a technical solution: a data synchronization transmission method for heterogeneous databases, the method comprising:

[0056] S10: Capture the data to be synchronized in the source database. The source database refers to a self-sufficient database that can directly provide original materials or specific data. Users do not need to consult other information sources. The captured data to be synchronized is compared with the data to be changed stored in each sub-database in the heterogeneous database to obtain the target incremental data of each sub-database. The target incremental data of each sub-database is processed and processed. The target incremental data obtained after the processing and the position information of each sub-incremental data in each target incremental data in the data to be changed are respectively stored in the temporary data storage area of ​​the corresponding sub-database. The target incremental data refers to the data that needs to be added and replaced in each sub-database. For example, if the target incremental data is (1, 3, 5), then 1, 3, and 5 are all sub-incremental data of the target incremental data.

[0057] S10 includes:

[0058] A sub-database is randomly selected from the sub-databases of the heterogeneous database, and the captured data to be synchronized is compared with the data to be changed stored in the selected sub-database. In the data to be synchronized, the data that needs to be added in the data to be changed stored in the selected sub-database is marked, and the marked data is deleted from the data to be synchronized. The deleted marked data is stored in a set M in the order in which the marked data is deleted, and the marked data stored in the first storage position and the second storage position in the set M are acquired. If the storage positions of the two acquired marked data in the data to be synchronized are adjacent, the marked data stored in the third storage position in the set M is acquired. If the marked data stored in the third storage position in the set M and the marked data stored in the second storage position in the set M are not adjacent in the storage positions in the data to be synchronized, the marked data stored in the first, second and third storage positions in the set M are a group of deleted marked data. If the storage positions of the two acquired marked data in the data to be synchronized are not adjacent, the marked data stored in the first storage position in the set M is a group of deleted marked data;

[0059] The data to be synchronized after the deletion processing is compared with the data to be changed stored in the selected sub-database. In the data to be synchronized after the deletion processing, the data that is inconsistent with the comparison is marked. The set of each group of deletion mark data and each mark data stored in the data to be synchronized after the deletion processing is the target incremental data of the selected sub-database. The number of each mark data in the data to be changed stored in the selected sub-database and the associated number of each group of deletion mark data are obtained. The method for obtaining the number of the mark data is: numbering each sub-data to be synchronized according to the storage order of each sub-data to be synchronized in the data to be synchronized after the deletion processing. The number of the sub-data to be synchronized is the number of the sub-data to be synchronized in the data to be changed stored in the selected sub-database. The mark data is the sub-data to be synchronized of the data to be synchronized after the deletion processing. The method for obtaining the associated number of the marks is: randomly selecting a group of deletion mark data, in the data to be synchronized, the data stored in the storage position before the storage position where the selected deletion mark data is located is obtained, and the number of the obtained data in the data to be synchronized after the deletion processing is obtained, then the associated number of the selected deletion mark data = the number of the obtained number + 0.5;

[0060] According to the data format of the data stored in the selected sub-database in the heterogeneous database, the target incremental data of the selected sub-database is processed. The processing refers to converting the target incremental data format of each sub-database into the data format of the data stored in the corresponding sub-database. Based on the processing result, the target incremental data of the selected sub-database and the number of bits of each sub-incremental data in the target incremental data of the selected sub-database in the data to be changed stored in the selected sub-database or the number of associated bits of each sub-incremental data are stored in the temporary data storage area of ​​the selected sub-database. The data stored in the temporary data storage area is:

[0061] {(F1,max(U1,Z1)),(F2,max(U2,Z2)),…,(F b ,max(U b ,Z b ))}, where F b Indicates the value corresponding to the bth sub-incremental data in the target incremental data, U b Indicates the number of bits of the b-th incremental data in the data to be updated stored in the selected sub-database, Z b Indicates the number of associated digits of the b-th sub-incremental data, b indicates the total number of sub-incremental data contained in the target incremental data of the selected sub-database. If the b-th sub-incremental data has associated digits, then U b =0, Z b =The number of associated bits of the b-th sub-incremental data. If there is no associated bit number for the b-th sub-incremental data, then U b =The number of bits of the b-th sub-incremental data in the data to be updated stored in the selected sub-database, Z b=0, max represents the maximum value symbol, and the sub-incremental data refers to a marked data or a group of deleted marked data in the target incremental data;

[0062] Repeat the above operation to determine the data stored in the temporary data storage area of ​​the remaining sub-databases in the heterogeneous database except the selected sub-database;

[0063] S20: extracting features of the target incremental data stored in each temporary data storage area to obtain a feature vector of each target incremental data, and predicting the transmission similarity between two target incremental data based on the obtained feature vector;

[0064] The S20 includes:

[0065] S201: Perform numbering processing on each sub-database, and the numbering processing result is: i=1, 2, ..., n; n represents the total number of numbers, and the data stored in the temporary data storage area of ​​the sub-database numbered i is: {(F 1i→I ,max(U 1i→i ,Z 1i→i )),(F 2i→I ,max(U 2i→i ,Z 2i→i )),…,(F di→I ,max(U di→i ,Z di→i ))}, where di represents the total number of sub-incremental data contained in the target incremental data of the sub-database numbered i, and F di→i Indicates the value corresponding to the dth sub-incremental data in the target incremental data of the sub-database numbered i, U di→i The number of bits of the dth incremental data in the target incremental data of the sub-database numbered i in the data to be changed stored in the sub-database numbered i, Z di→i Indicates the number of bits associated with the dth sub-incremental data in the target incremental data of the sub-database numbered i;

[0066] According to max(U ji→i ,Z ji→i ) in order from small to large, for {(F 1i→I ,max(U 1i→i ,Z 1i→i )),(F 2i→I ,max(U 2i→i ,Z 2i→i )),…,(F di→I ,max(U di→i ,Z di→i ))}, wherein j=1, 2, ..., d, indicating that each sub-incremental data is numbered according to the recording order of the target incremental data, and d indicates the total number of sub-incremental data included in the target incremental data;

[0067] S202: Construct a feature vector model Q, where Q = {W1, W2, ..., W x},W x represents the value corresponding to the xth component in the feature vector model, x represents the maximum value of the total number of data to be changed stored in each sub-database, k = 1, 2, ..., x, represents the number corresponding to each component in the feature vector model, if there is k = max(U ji→i ,Z ji→i ) or k=max(U ji→i ,Z ji→i )+0.5, then W k =1, if k = max(U ji→i ,Z ji→i ) and k=max(U ji→i ,Z ji→i )+0.5 do not exist, then W k =0,W k Represents the value corresponding to the kth component in the feature vector model, the feature vector Q of the target incremental data of the sub-database numbered i i = {W 1i ,W 2i ,…,W xi},W xi The value corresponding to the kth component in the feature vector model representing the target incremental data of the sub-database numbered i;

[0068] S203: Randomly select a feature vector as the target feature vector, according to The transmission similarity between the target incremental data corresponding to the target feature vector and the target incremental data corresponding to the feature vector numbered j is calculated, where j = 1, 2, ..., m, represents the number corresponding to each feature vector except the target feature vector, m represents the total number of feature vectors except the target feature vector, X jk Y represents the value corresponding to the kth component in the eigenvector numbered j, k Represents the value corresponding to the kth component in the target feature vector, R j represents the transmission similarity between the target incremental data corresponding to the target feature vector and the target incremental data corresponding to the feature vector numbered j;

[0069] S30: Analyze the data synchronization transmission status of each sub-database;

[0070] S30 includes:

[0071] S301: Determine the target incremental data corresponding to the feature vector numbered j, and set up a transmission channel length L between the temporary data storage area storing the determined target incremental data and the sub-database corresponding to the determined target incremental data. j , and the transmission rate V of the data in the transmission channel established between the temporary data storage area storing the determined target incremental data and the sub-database corresponding to the determined target incremental data j To obtain, according to The time for the target incremental data corresponding to the target feature vector to be transmitted to the corresponding sub-database is calculated. V represents the transmission rate of the data in the transmission channel between the temporary data storage area storing the target incremental data corresponding to the target feature vector and the sub-database corresponding to the target feature vector. L represents the length of the transmission channel between the temporary data storage area storing the target incremental data corresponding to the target feature vector and the sub-database corresponding to the target feature vector. h represents the data length value of one bit of data. R k Represents the total number of data stored in the sub-incremental data corresponding to the number of bits corresponding to the kth component in the target feature vector. If the sub-incremental data is labeled data, then R k =1, if the sub-incremental data is the deletion mark data, then R k =Total number of data present in the deleted marked data;

[0072] According to E j =T*(1-R j )-T′*(1-R j ), The transmission time difference between the target incremental data corresponding to the target feature vector and the target incremental data corresponding to the feature vector numbered j is calculated, V j represents the transmission rate of data in the transmission channel between the temporary data storage area of ​​the target incremental data corresponding to the feature vector with the storage number j and the corresponding sub-database, R kj represents the total number of data stored in the sub-incremental data corresponding to the number of bits corresponding to the kth component in the feature vector numbered j. If the sub-incremental data is marked data, then R kj =1, if the sub-incremental data is the deletion mark data, then R kj =Total number of data present in the deleted marked data;

[0073] If E j > 0, it means that the target incremental data corresponding to the feature vector numbered j is transmitted before the target incremental data corresponding to the target feature vector is transmitted. j Time starts to transfer;

[0074] If E j<0, it means the target incremental data corresponding to the feature vector numbered j is transmitted in the target incremental data corresponding to the target feature vector E j Start transmission after time;

[0075] If E j = 0, it means that the target incremental data corresponding to the feature vector numbered j has the same transmission time as the target incremental data corresponding to the target feature vector;

[0076] S40: determining the transmission time of the target incremental data corresponding to the target feature vector according to the data synchronization boundary time of each sub-database, and analyzing the transmission time of other target incremental data based on the determination result, where the other target incremental data refers to the target incremental data other than the target incremental data corresponding to the target feature vector;

[0077] S40 includes: synchronizing the data of the sub-database numbered i with the time limit G i The data synchronization limit time value refers to the latest update time of the data to be changed in the corresponding sub-database, and the transmission time of the target incremental data corresponding to the target feature vector is minG i -T, according to U j =minG i -TE j The transmission time of the target incremental data corresponding to the feature vector numbered j is determined, and min represents the minimum value symbol.

[0078] A data synchronization transmission system for heterogeneous databases, the system comprising a temporary data storage area data storage module, a transmission similarity prediction module, a data synchronization transmission situation analysis module and a transmission time analysis module;

[0079] The data storage module in the temporary data storage area is used to store the target incremental data obtained after processing, and the position information of each sub-incremental data in each target incremental data in the data to be changed in the temporary data storage area of ​​the corresponding sub-database;

[0080] The data storage module of the temporary data storage area includes a target incremental data determination unit and a transmission data determination unit;

[0081] The target incremental data determination unit randomly selects a sub-database from the sub-databases of the heterogeneous database, compares the captured data to be synchronized with the data to be changed stored in the selected sub-database, deletes the data to be synchronized based on the comparison result, performs a secondary comparison on the deleted data to be synchronized with the data to be changed stored in the selected sub-database, and determines the target incremental data of the selected sub-database based on the secondary comparison result;

[0082] The transmission data determination unit processes the target incremental data of the selected sub-database, and based on the processing result, stores the target incremental data of the selected sub-database, and the number of bits of each sub-incremental data in the target incremental data of the selected sub-database in the data to be changed stored in the selected sub-database or the number of bits associated with each sub-incremental data in a temporary data storage area of ​​the selected sub-database, and determines the transmission data stored in the temporary data storage area;

[0083] The transmission similarity prediction module is used to predict the transmission similarity between two target incremental data;

[0084] The transmission similarity prediction module includes an adjustment unit, a feature vector acquisition unit and a transmission similarity prediction unit;

[0085] The adjusting unit adjusts the storage position of each data stored in the temporary data storage area of ​​the sub-database corresponding to the target incremental data according to the number of bits of each sub-incremental data in the target incremental data in the data to be changed stored in the sub-database corresponding to the target incremental data and the number of bits associated with each sub-incremental data in the target incremental data;

[0086] The feature vector acquisition unit receives the adjustment result transmitted by the adjustment unit, and determines the feature vector of each target incremental data in combination with the constructed feature vector model;

[0087] The transmission similarity prediction unit predicts the transmission similarity between the target incremental data corresponding to the target feature vector and the target incremental data corresponding to other feature vectors through the constructed prediction model, and the other feature vectors refer to feature vectors other than the target feature vector;

[0088] The data synchronization transmission situation analysis module is used to analyze the data synchronization transmission situation of each sub-database;

[0089] The data synchronization transmission situation analysis module includes a data transmission time calculation unit, a data transmission time difference calculation unit and a data synchronization transmission situation analysis unit;

[0090] The data transmission time calculation unit calculates the time for transmitting the target incremental data corresponding to the target feature vector to the corresponding sub-database according to the constructed mathematical model;

[0091] The data transmission time difference calculation unit calculates the transmission time difference between the target incremental data corresponding to the target feature vector and the target incremental data corresponding to other feature vectors;

[0092] The data synchronization transmission situation analysis unit receives the transmission time difference value transmitted by the data transmission time difference calculation unit, compares the received data with 0, and determines the synchronization transmission situation of the target incremental data corresponding to other characteristic vectors according to the comparison result;

[0093] The transmission time analysis module is used to analyze the transmission time of each target incremental data;

[0094] The transmission time analysis module includes a transmission time analysis unit 1 and a transmission time analysis unit 2;

[0095] The transmission time analysis unit 1 obtains the data synchronization boundary time of each sub-database, and determines the transmission time of the target incremental data corresponding to the target feature vector in combination with the calculation result transmitted by the data transmission time calculation unit;

[0096] Transmission time analysis unit 2 receives the transmission time transmitted by transmission time analysis unit 1, and determines the transmission time of the target incremental data corresponding to other feature vectors in combination with the calculation result transmitted by the data transmission time difference calculation unit. Other feature vectors refer to feature vectors other than the target feature vector.

[0097] Example 1: Assume that the transmission rate V in the transmission channel between the temporary data storage area for storing the target incremental data corresponding to the target feature vector and the sub-database corresponding to the target feature vector is 2 bits / second, the length L of the transmission channel between the temporary data storage area for storing the target incremental data corresponding to the target feature vector and the sub-database corresponding to the target feature vector is 1 meter, the data length value h of one bit of data is 0.1 meter, the target feature vector is (1,0,1), and the total number of data stored in the sub-incremental data corresponding to the number of bits corresponding to the first component of the target feature vector is R k = 1 bit, the total number of data stored in the sub-incremental data corresponding to the number of bits corresponding to the third component in the target feature vector R k =5 bits, then the time for the target incremental data corresponding to the target feature vector to be transmitted to the corresponding sub-database is:

[0098]

[0099] Then, the time for transmitting the target incremental data corresponding to the target feature vector to the corresponding sub-database is 5 seconds.

[0100] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0101] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data synchronization transmission method for heterogeneous databases, characterized in that: The method comprises: S10: Capture the data to be synchronized in the source database, compare the captured data to be synchronized with the data to be changed stored in each sub-database in the heterogeneous database, obtain the target incremental data of each sub-database, process the target incremental data of each sub-database, and store the target incremental data obtained after the processing and the position information of each sub-incremental data in the data to be changed in each target incremental data in the temporary data storage area of ​​the corresponding sub-database; S20: extracting features of the target incremental data stored in each temporary data storage area to obtain a feature vector of each target incremental data, and predicting the transmission similarity between two target incremental data based on the obtained feature vector; S30: Analyze the data synchronization transmission status of each sub-database; S40: Determine the transmission time of the target incremental data corresponding to the target feature vector according to the data synchronization boundary time of each sub-database, and analyze the transmission time of other target incremental data based on the determination result.

2. A method for synchronous transmission of data for heterogeneous databases according to claim 1, characterized in that: The S10 includes: A sub-database is randomly selected from the sub-databases of the heterogeneous database, and the captured data to be synchronized is compared with the data to be changed stored in the selected sub-database. In the data to be synchronized, the data to be added in the data to be changed stored in the selected sub-database is marked, and the marked data is deleted from the data to be synchronized; The data to be synchronized after the deletion process is compared with the data to be changed stored in the selected sub-database, and the inconsistent data is marked in the data to be synchronized after the deletion process. The set of each group of deletion mark data and each mark data stored in the data to be synchronized after the deletion process is the target incremental data of the selected sub-database, and the number of bits of each mark data in the data to be changed stored in the selected sub-database and the number of associated bits of each group of deletion mark data are obtained; According to the data format of the data stored in the selected sub-database in the heterogeneous database, the target incremental data of the selected sub-database is processed, and based on the processing result, the target incremental data of the selected sub-database and the number of bits of each sub-incremental data in the target incremental data of the selected sub-database in the data to be changed stored in the selected sub-database or the number of associated bits of each sub-incremental data are stored in the temporary data storage area of ​​the selected sub-database. The data stored in the temporary data storage area are: {(F1,max(U1,Z1)),(F2,max(U2,Z2)),…,(F b ,max(U b ,Z b ))}, where F b Indicates the value corresponding to the bth sub-incremental data in the target incremental data, U b Indicates the number of bits of the b-th incremental data in the data to be updated stored in the selected sub-database, Z b Indicates the number of associated digits of the b-th sub-incremental data, b indicates the total number of sub-incremental data contained in the target incremental data of the selected sub-database. If the b-th sub-incremental data has associated digits, then U b =0, Z b =The number of associated bits of the b-th sub-incremental data. If there is no associated bit number for the b-th sub-incremental data, then U b =The number of bits of the b-th sub-incremental data in the data to be updated stored in the selected sub-database, Z b =0, max indicates the maximum value symbol; Repeat the above operation to determine the data stored in the temporary data storage area of ​​the remaining sub-databases in the heterogeneous database except the selected sub-database.

3. A method for synchronous transmission of data for heterogeneous databases according to claim 2, characterized in that: The S20 includes: S201: Perform numbering processing on each sub-database, and the numbering processing result is: i=1, 2, ..., n; n represents the total number of numbers, and the data stored in the temporary data storage area of ​​the sub-database numbered i is: {(F 1i→I ,max(U 1i→i ,Z 1i→i )),(F 2i→I ,max(U 2i→i ,Z 2i→i )),…,(F di→I ,max(U di→i ,Z di→i ))}, where di represents the total number of sub-incremental data contained in the target incremental data of the sub-database numbered i, and F di→i Indicates the value corresponding to the dth sub-incremental data in the target incremental data of the sub-database numbered i, U di→i The number of bits of the dth incremental data in the target incremental data of the sub-database numbered i in the data to be changed stored in the sub-database numbered i, Z di→i Indicates the number of bits associated with the dth sub-incremental data in the target incremental data of the sub-database numbered i; According to max(U ji→i ,Z ji→i ) in order from small to large, for {(F 1i→I ,max(U 1i→i ,Z 1i→i )),(F 2i→I ,max(U 2i→i ,Z 2i→i )),…,(F di→I ,max(U di→i ,Z di→i ))}, wherein j=1, 2, ..., d, indicating that each sub-incremental data is numbered according to the recording order of the target incremental data, and d indicates the total number of sub-incremental data included in the target incremental data; S202: Construct a feature vector model Q, where Q = {W1, W2, ..., W x },W x represents the value corresponding to the xth component in the feature vector model, x represents the maximum value of the total number of data to be changed stored in each sub-database, k = 1, 2, ..., x, represents the number corresponding to each component in the feature vector model, if there is k = max(U ji→i ,Z ji→i ) or k=max(U ji→i ,Z ji→i )+0.5, then W k =1, if k = max(U ji→i ,Z ji→i ) and k=max(U ji→i ,Z ji→i )+0.5 do not exist, then W k =0,W k Represents the value corresponding to the kth component in the feature vector model, the feature vector Q of the target incremental data of the sub-database numbered i i = {W 1i ,W 2i ,…,W xi }; S203: Randomly select a feature vector as the target feature vector, according to The transmission similarity between the target incremental data corresponding to the target feature vector and the target incremental data corresponding to the feature vector numbered j is calculated, where j = 1, 2, ..., m, represents the number corresponding to each feature vector except the target feature vector, m represents the total number of feature vectors except the target feature vector, X jk Y represents the value corresponding to the kth component in the eigenvector numbered j, k Represents the value corresponding to the kth component in the target feature vector.

4. A method for synchronous transmission of data for heterogeneous databases according to claim 3, characterized in that: The S30 includes: S301: Determine the target incremental data corresponding to the feature vector numbered j, and set up a transmission channel length L between the temporary data storage area storing the determined target incremental data and the sub-database corresponding to the determined target incremental data. j , and the transmission rate V of the data in the transmission channel established between the temporary data storage area storing the determined target incremental data and the sub-database corresponding to the determined target incremental data j To obtain, according to The time for the target incremental data corresponding to the target feature vector to be transmitted to the corresponding sub-database is calculated. V represents the transmission rate of the data in the transmission channel between the temporary data storage area storing the target incremental data corresponding to the target feature vector and the sub-database corresponding to the target feature vector. L represents the length of the transmission channel between the temporary data storage area storing the target incremental data corresponding to the target feature vector and the sub-database corresponding to the target feature vector. h represents the data length value of one bit of data. R k Indicates the total number of data stored in the sub-incremental data corresponding to the number of bits corresponding to the kth component in the target feature vector; According to E j =T*(1-R j )-T′*(1-R j ), The transmission time difference between the target incremental data corresponding to the target feature vector and the target incremental data corresponding to the feature vector numbered j is calculated, V j represents the transmission rate of data in the transmission channel between the temporary data storage area of ​​the target incremental data corresponding to the feature vector with the storage number j and the corresponding sub-database, R kj Indicates the total number of data stored in the sub-incremental data corresponding to the number of bits corresponding to the kth component in the feature vector numbered j; If E j > 0, it means that the target incremental data corresponding to the feature vector numbered j is transmitted before the target incremental data corresponding to the target feature vector is transmitted. j Time starts to transfer; If E j <0, it means the target incremental data corresponding to the feature vector numbered j is transmitted in the target incremental data corresponding to the target feature vector E j Start transmission after time; If E j =0, it means that the target incremental data corresponding to the feature vector numbered j has the same transmission time as the target incremental data corresponding to the target feature vector.

5. A method for synchronous transmission of data for heterogeneous databases according to claim 4, characterized in that: The step S40 includes: synchronizing the data of the sub-database numbered i with the time limit G i To obtain, the transmission time of the target incremental data corresponding to the target feature vector is minG i -T, according to U j =minG i -TE j The transmission time of the target incremental data corresponding to the feature vector numbered j is determined.

6. A data synchronization transmission system for heterogeneous databases applied to the data synchronization transmission method for heterogeneous databases according to any one of claims 1 to 5, characterized in that: The system includes a temporary data storage area data storage module, a transmission similarity prediction module, a data synchronization transmission situation analysis module and a transmission time analysis module; The data storage module in the temporary data storage area is used to store the target incremental data obtained after processing, and the position information of each sub-incremental data in each target incremental data in the data to be changed in the temporary data storage area of ​​the corresponding sub-database; The transmission similarity prediction module is used to predict the transmission similarity between two target incremental data; The data synchronization transmission situation analysis module is used to analyze the data synchronization transmission situation of each sub-database; The transmission time analysis module is used to analyze the transmission time of each target incremental data.

7. A data synchronization transmission system for heterogeneous databases according to claim 6, characterized in that: The temporary data storage area data storage module includes a target incremental data determination unit and a transmission data determination unit; The target incremental data determination unit randomly selects a sub-database from the sub-databases of the heterogeneous database, compares the captured data to be synchronized with the data to be changed stored in the selected sub-database, deletes the data to be synchronized based on the comparison result, performs a secondary comparison on the deleted data to be synchronized with the data to be changed stored in the selected sub-database, and determines the target incremental data of the selected sub-database based on the secondary comparison result; The transmission data determination unit processes the target incremental data of the selected sub-database, and based on the processing result, stores the target incremental data of the selected sub-database and the number of bits of each sub-incremental data in the target incremental data of the selected sub-database in the data to be changed stored in the selected sub-database or the number of associated bits of each sub-incremental data in the temporary data storage area of ​​the selected sub-database, and determines the transmission data stored in the temporary data storage area.

8. A data synchronization transmission system for heterogeneous databases according to claim 7, characterized in that: The transmission similarity prediction module includes an adjustment unit, a feature vector acquisition unit and a transmission similarity prediction unit; The adjusting unit adjusts the storage position of each data stored in the temporary data storage area of ​​the sub-database corresponding to the target incremental data according to the number of bits of each sub-incremental data in the target incremental data in the data to be changed stored in the sub-database corresponding to the target incremental data and the number of bits associated with each sub-incremental data in the target incremental data; The feature vector acquisition unit receives the adjustment result transmitted by the adjustment unit, and determines the feature vector of each target incremental data in combination with the constructed feature vector model; The transmission similarity prediction unit predicts the transmission similarity between the target incremental data corresponding to the target feature vector and the target incremental data corresponding to other feature vectors through the constructed prediction model.

9. A data synchronization transmission system for heterogeneous databases according to claim 8, characterized in that: The data synchronization transmission situation analysis module includes a data transmission time calculation unit, a data transmission time difference calculation unit and a data synchronization transmission situation analysis unit; The data transmission time calculation unit calculates the time for transmitting the target incremental data corresponding to the target feature vector to the corresponding sub-database according to the constructed mathematical model; The data transmission time difference calculation unit calculates the transmission time difference between the target incremental data corresponding to the target feature vector and the target incremental data corresponding to other feature vectors; The data synchronous transmission situation analysis unit receives the transmission time difference value transmitted by the data transmission time difference calculation unit, compares the received data with 0, and determines the synchronous transmission situation of the target incremental data corresponding to other feature vectors according to the comparison result.

10. A data synchronization transmission system for heterogeneous databases according to claim 9, characterized in that: The transmission time analysis module includes a transmission time analysis unit 1 and a transmission time analysis unit 2; The transmission time analysis unit obtains the data synchronization boundary time of each sub-database, and determines the transmission time of the target incremental data corresponding to the target feature vector in combination with the calculation result transmitted by the data transmission time calculation unit; The second transmission time analysis unit receives the transmission time transmitted by the first transmission time analysis unit, and determines the transmission time of the target incremental data corresponding to other characteristic vectors in combination with the calculation result transmitted by the data transmission time difference calculation unit.

Images