A cross-platform heterogeneous real-time synchronization method for database data

Abstract

The present invention relates to the technical field of information technology integration, and in particular to a cross-platform heterogeneous real-time synchronization method for database data, comprising: step S1, extracting pre-stored data; step S2, converting and processing the pre-stored data; step S3, loading the converted data into a test database; step S4, preliminarily determining whether the heterogeneous synchronization is qualified, and determining the reason for the failure; step S5, heterogeneous synchronization, re-testing; step S6, completing the heterogeneous synchronization; the present invention determines whether the heterogeneous synchronization is qualified according to the similarity between the updated test data and the pre-stored data in the source database, and determines that the heterogeneous synchronization is qualified when the similarity is large, so as to ensure that the updated data in each test database are consistent, and whether the data conversion is qualified or not will affect the distribution of abnormal bytes, so whether the heterogeneous synchronization is qualified is secondary determined according to the distribution of the abnormal bytes, thereby improving the control precision of the data heterogeneous synchronization situation and improving the analysis accuracy.

Description

Technical Field

[0001] The present invention relates to the field of information technology integration technology, and in particular to a method for real-time synchronization of database data across platforms and heterogeneity. Background Art

[0002] Cross-platform heterogeneous real-time synchronization of data refers to the process of real-time transmission and synchronization of data between different platforms. The existing technology encodes information through measurement point parameter information, measurement point data information and communication parameters, and realizes the synchronization of heterogeneous real-time historical databases in different locations by extracting information from the message. However, when multiple nodes are updated, the data of each node is updated separately, and the differences are large, making it difficult to ensure the consistency of data in each database.

[0003] Chinese patent application number: CN201310322804.9 discloses a method and system for synchronizing heterogeneous real-time historical databases. The client encodes the configuration information of measurement point parameter information, measurement point data information and communication parameters, and adjusts the number of communication component threads in combination with the communication rules for data collection and transmission established during the data collection configuration and communication initialization process of the client communication node, thereby determining the best conditions for client communication. Then, the server receives data from each client node through the network, and after processing the information in the message, the data information is finally written into the real-time historical database to be synchronized, thereby realizing the synchronization of heterogeneous real-time historical databases in different locations. It can be seen that the method and system for synchronizing heterogeneous real-time historical databases have the following problems: the data of each node is updated separately, and the difference is large, making it difficult to ensure the consistency of the data in each database. Summary of the invention

[0004] To this end, the present invention provides a cross-platform heterogeneous real-time synchronization method for database data, which is used to overcome the problem in the prior art that each node data is updated separately and the differences are large, making it difficult to ensure data consistency in each database.

[0005] To achieve the above objectives, the present invention provides a method for real-time synchronization of database data across platforms and heterogeneity. The method comprises:

[0006] Step S1, extracting pre-stored data to be synchronized from the source database;

[0007] Step S2: converting and processing the extracted pre-stored data to adapt to the format and structure of the test database;

[0008] Step S3, loading the converted data into the test database;

[0009] Step S4, the data analysis module preliminarily determines whether the heterogeneous synchronization of the pre-stored data is qualified according to the similarity between the test data in the single test database and the pre-stored data in the source database, and determines the reason for the failure according to the similarity when the heterogeneous synchronization is preliminarily determined to be unqualified;

[0010] Step S5, the data analysis module adjusts the parameters in the data heterogeneous synchronization process to corresponding values ​​according to the analysis results, re-synchronizes the data heterogeneously, and the data analysis module re-detects;

[0011] Step S6: When the data analysis module determines that the heterogeneous synchronization of the pre-stored data is qualified, the data analysis module completes the heterogeneous synchronization of the pre-stored data.

[0012] Furthermore, the data analysis module determines whether the heterogeneous synchronization of the pre-stored data in the test database is qualified according to the similarity between the test data in the single test database and the pre-stored data in the source database.

[0013] When the data analysis module preliminarily determines that the heterogeneous synchronization of the pre-stored data in the test database is unqualified, it determines whether the data conversion in the heterogeneous synchronization process is qualified based on the distribution of different bytes in the test database that are not similar to those in the source database, or determines the reason for its failure based on the similarity.

[0014] Furthermore, the data analysis module determines whether the data conversion in the heterogeneous synchronization process is qualified according to the distribution of the heterogeneous bytes, and,

[0015] When determining that data conversion in the heterogeneous synchronization process is qualified, the data analysis module determines that the reason why the heterogeneous synchronization in the test database is unqualified is that data transmission is damaged;

[0016] The data analysis module determines the reason for the failure when determining that the data conversion in the heterogeneous synchronization process is unqualified.

[0017] Furthermore, when the data analysis module determines that the reason for the failure of heterogeneous synchronization in the test database is damaged data transmission, it provides several correction methods for bytes in the buffer according to the number of simultaneous transmission devices, and the correction amplitude of each correction method for bytes is different.

[0018] Further, when determining the reason for the failure according to the similarity, the data analysis module records the ratio of the similarity to the second preset similarity in the data analysis module as a primary ratio, and determines the reason for the failure of heterogeneous synchronization in the test database according to the primary ratio, the reason including data loading failure and data conversion failure;

[0019] When determining to mark the test database as an abnormal database, the data analysis module determines the reason why the heterogeneous synchronization in the test database is unqualified according to the proportion of the abnormal databases.

[0020] Furthermore, when the data analysis module determines that the reason for the failure of heterogeneous synchronization in the test database is that the data loading is unqualified, the data analysis module uses a compression algorithm to compress the data volume to optimize the connection settings and transmission protocols for the distributed data.

[0021] Furthermore, the data analysis module records the ratio of the number of abnormal databases to the total number of test databases as a secondary ratio, and determines the reasons for the failure of heterogeneous synchronization in the test database based on the secondary ratio, the reasons including: data conversion failure and data loading failure.

[0022] Furthermore, when determining that the data conversion is unqualified, the data analysis module is provided with a plurality of adjustment modes for the sampling rate according to the order of the filter, and the adjustment range of the sampling rate in each adjustment mode is different.

[0023] Compared with the prior art, the beneficial effect of the present invention lies in that the data analysis module in the present invention determines whether the heterogeneous synchronization of data is qualified according to the similarity between the updated test data in a single test database and the pre-stored data in the source database, and determines that the heterogeneous synchronization is qualified when the similarity is large, thereby ensuring that the updated data in each test database is consistent. Whether the data conversion is qualified or not will affect the distribution of abnormal bytes, so whether the heterogeneous synchronization is qualified is secondary determined according to the distribution of abnormal bytes, thereby improving the control accuracy of the data heterogeneous synchronization situation and improving the analysis accuracy.

[0024] Furthermore, the data analysis module in the present invention determines whether the data conversion is qualified based on the distribution of abnormal bytes. If the abnormal bytes in the original data are distributed relatively evenly, after conversion, the abnormal bytes will be concentrated in certain positions or ranges due to the influence of the format or encoding rules. Therefore, the data analysis module determines that the data conversion is unqualified when the distribution of the abnormal bytes is scattered, thereby further improving the analysis accuracy.

[0025] Furthermore, in the present invention, when there are a large number of simultaneous transmission devices, the data volume is large, the data transmission efficiency is slow, the data transmission flow is too large, and the system buffer is set too small, which will lead to data loss. The data analysis module selects the corresponding correction coefficient according to the number of simultaneous transmission devices to correct the size of the buffer to the corresponding value, thereby ensuring the quality of data transmission.

[0026] Furthermore, the data analysis module in the present invention determines the cause of the failure of heterogeneous synchronization based on the ratio of the similarity to the second preset similarity, and determines the cause of the failure of heterogeneous synchronization based on the proportion of abnormal databases when the ratio is between the first preset first-level ratio and the second preset first-level ratio, thereby further improving the control accuracy of the heterogeneous synchronization process. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of the cross-platform heterogeneous real-time synchronization method of database data of the present invention;

[0028] Figure 2 A flowchart for preliminarily determining whether the heterogeneous synchronization of pre-stored data in the test database is qualified;

[0029] Figure 3 A flowchart for determining whether data conversion in a heterogeneous synchronization process is qualified;

[0030] Figure 4 The present invention is a flowchart for determining the reasons why heterogeneous synchronization fails. DETAILED DESCRIPTION

[0031] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0032] It should be pointed out that the data in this embodiment are obtained by comprehensive analysis and evaluation of the historical data of the system of the present invention in the six months before this determination and the corresponding historical determination results. Before this detection, the system of the present invention comprehensively determines the values ​​of the preset parameter standards for this determination based on the evaluation values ​​of the 37,383 retrieval results detected cumulatively in the first three months. It can be understood by those skilled in the art that the system of the present invention can determine the single parameter mentioned above by selecting the value with the highest proportion as the preset standard parameter according to the data distribution, using weighted summation to use the obtained value as the preset standard parameter, substituting each historical data into a specific formula and using the value obtained by the formula as the preset standard parameter or other selection methods, as long as the system of the present invention can clearly define the different specific situations in the single determination process through the obtained values.

[0033] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0034] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0035] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0036] See also Figure 1 As shown, it is a flow chart of the cross-platform heterogeneous real-time synchronization method of database data of the present invention.

[0037] Step S1, extracting pre-stored data to be synchronized from the source database;

[0038] Step S2: converting and processing the extracted pre-stored data to adapt to the format and structure of the test database;

[0039] Step S3, loading the converted data into the test database;

[0040] Step S4, the data analysis module preliminarily determines whether the heterogeneous synchronization of the pre-stored data is qualified according to the similarity between the test data in the single test database and the pre-stored data in the source database, and determines the reason for the failure according to the similarity when the heterogeneous synchronization is preliminarily determined to be unqualified;

[0041] Step S5, the data analysis module adjusts the parameters in the data heterogeneous synchronization process to corresponding values ​​according to the analysis results, re-synchronizes the data heterogeneously, and the data analysis module re-detects;

[0042] Step S6: When the data analysis module determines that the heterogeneous synchronization of the pre-stored data is qualified, the data analysis module completes the heterogeneous synchronization of the pre-stored data.

[0043] See also Figure 2 As shown, it is a flowchart for preliminarily determining whether the heterogeneous synchronization of pre-stored data in the test database is qualified.

[0044] Specifically, the data analysis module determines whether the heterogeneous synchronization of the pre-stored data in the test database is qualified according to the similarity between the test data in the single test database and the pre-stored data in the source database, wherein:

[0045] The first preliminary determination method is that the data analysis module determines that the heterogeneous synchronization of the pre-stored data in the test database is qualified; the first preliminary determination method satisfies that the similarity is greater than or equal to a first preset similarity in the data analysis module;

[0046] The second preliminary determination method is that the data analysis module preliminarily determines that the heterogeneous synchronization of the pre-stored data in the test database is unqualified, and determines whether the data conversion in the heterogeneous synchronization process is qualified according to the distribution of different bytes in the test database that are not similar to those in the source database; the second preliminary determination method satisfies that the similarity is less than the first preset similarity and is greater than or equal to the second preset similarity in the data analysis module;

[0047] The third preliminary determination method is that the data analysis module determines that the heterogeneous synchronization of the pre-stored data in the test database is unqualified, and determines the reason for the unqualified synchronization based on the similarity; the third preliminary determination method satisfies that the similarity is less than the second preset similarity.

[0048] In the embodiment of the present invention, the first preset similarity is 0.95, and the second preset similarity is 0.96.

[0049] The data analysis module in the present invention determines whether the heterogeneous synchronization of data is qualified according to the similarity between the updated test data in a single test database and the pre-stored data in the source database, and determines that the heterogeneous synchronization is qualified when the similarity is large, thereby ensuring that the updated data in each test database is consistent. Whether the data conversion is qualified or not will affect the distribution of abnormal bytes, so whether the heterogeneous synchronization is qualified is secondary determined according to the distribution of abnormal bytes, thereby improving the control accuracy of the data heterogeneous synchronization situation and improving the analysis accuracy.

[0050] See also Figure 3 As shown, it is a flowchart for determining whether data conversion in a heterogeneous synchronization process is qualified.

[0051] Specifically, the data analysis module determines a determination method for whether data conversion in the heterogeneous synchronization process is qualified according to the distribution of the different bytes in the second preliminary determination method, wherein:

[0052] The first determination method is that the data analysis module determines that the data conversion in the heterogeneous synchronization process is qualified, and determines that the reason for the failure of the heterogeneous synchronization in the test database is that the data transmission is damaged; the first determination method satisfies the distribution concentration of the heterogeneous bytes;

[0053] The second determination method is that the data analysis module determines that the data conversion in the heterogeneous synchronization process is unqualified, and determines the reason for the unqualified data conversion; the second determination method satisfies the distribution dispersion of the heterogeneous bytes.

[0054] The data analysis module in the present invention determines whether the data conversion is qualified according to the distribution of abnormal bytes. If the abnormal bytes in the original data are distributed relatively evenly, after conversion, the abnormal bytes will be concentrated in certain positions or ranges due to the influence of the format or encoding rules. Therefore, the data analysis module determines that the data conversion is unqualified when the distribution of the abnormal bytes is scattered, thereby further improving the analysis accuracy.

[0055] Specifically, the data analysis module determines a byte correction method for the buffer according to the number of simultaneous transmission devices under the first determination method, wherein:

[0056] The first byte correction method is that the data analysis module selects the first byte correction coefficient β1 to correct the byte Z of the buffer to the corresponding value, and sets the corrected byte Z'=β1×Z0, where Z0 is the initial byte before correction; the first byte correction method satisfies that the number of simultaneous transmission devices is less than or equal to the preset number in the data analysis module;

[0057] The second byte correction method is that the data analysis module selects the second byte correction coefficient β2 to correct the byte Z in the buffer to the corresponding value, and sets the corrected byte Z'=β2×Z0; the second byte correction method satisfies that the number of simultaneous transmission devices is greater than the preset number.

[0058] In the embodiment of the present invention, the first byte correction coefficient is 1.24, the second byte correction coefficient is 1.32, and the preset number is 8.

[0059] In the present invention, when there are a large number of simultaneous transmission devices, the data volume is large, the data transmission efficiency is slow, the data transmission flow is too large, and the system buffer is set too small, which will lead to data loss. The data analysis module selects the corresponding correction coefficient according to the number of simultaneous transmission devices to correct the size of the buffer to the corresponding value, thereby ensuring the quality of data transmission.

[0060] See also Figure 4 As shown, it is a flowchart for determining the reasons why heterogeneous synchronization fails.

[0061] Specifically, the data analysis module records the ratio of the similarity to the second preset similarity as a primary ratio under the third preliminary determination method, and determines the cause determination method for the heterogeneous synchronization failure in the test database according to the primary ratio, wherein:

[0062] The first cause determination method is that the data analysis module determines that the cause of heterogeneous synchronization failure in the test database is data loading failure; the first cause determination method satisfies that the first-level ratio is greater than or equal to the first preset first-level ratio in the data analysis module;

[0063] The second cause determination method is that the data analysis module determines to mark the test database as an abnormal database, and determines the cause of heterogeneous synchronization failure in the test database according to the proportion of abnormal databases; the second cause determination method is that the first-level ratio is less than the first preset first-level ratio and is greater than or equal to the second preset first-level ratio in the data analysis module;

[0064] The third cause determination method is that the data analysis module determines that the cause of the failure of heterogeneous synchronization in the test database is failure of data conversion; the third cause determination method satisfies that the first-level ratio is less than the second preset first-level ratio.

[0065] In the embodiment of the present invention, the first preset primary ratio is 0.85, and the second preset primary ratio is 0.76.

[0066] The data analysis module in the present invention determines the cause of the failure of heterogeneous synchronization based on the ratio of the similarity to the second preset similarity, and determines the cause of the failure of heterogeneous synchronization based on the proportion of abnormal databases when the ratio is between the first preset first-level ratio and the second preset first-level ratio, thereby further improving the control accuracy of the heterogeneous synchronization process.

[0067] Specifically, the data analysis module uses a compression algorithm to compress the data volume under the first cause determination method to optimize the connection settings and transmission protocols for distributed data.

[0068] Specifically, the data analysis module records the ratio of the number of abnormal databases to the total number of test databases as a secondary ratio under the second cause determination method, and determines a cause determination method for heterogeneous synchronization failure in the test database according to the secondary ratio, wherein:

[0069] The first cause determination method is that the data analysis module determines that the cause of the heterogeneous synchronization failure in the test database is data conversion failure; the first cause determination method satisfies that the secondary ratio is less than or equal to the preset secondary ratio in the data analysis module;

[0070] The second cause determination method is that the data analysis module determines that the cause of the failure of heterogeneous synchronization in the test database is failure of data loading; the second cause determination method satisfies that the secondary ratio is greater than the preset secondary ratio.

[0071] In the embodiment of the present invention, the preset secondary ratio is 0.43.

[0072] Specifically, the data analysis module determines an adjustment method for the sampling rate according to the order of the filter under the first preset condition, wherein:

[0073] The first adjustment method is that the data analysis module selects the first adjustment coefficient α1 to adjust the sampling rate V to a corresponding value, and sets the adjusted sampling rate V'=α1×V0, where V0 is the initial sampling rate before adjustment; the first adjustment method satisfies that the order of the filter is less than or equal to the preset order in the data analysis module;

[0074] The second adjustment method is that the data analysis module selects the second adjustment coefficient α2 to adjust the sampling rate V to a corresponding value, and sets the adjusted sampling rate V'=α2×V0; the second adjustment method satisfies that the order of the filter is greater than the preset order;

[0075] The first preset condition is that the data analysis module determines that the data conversion is unqualified.

[0076] In the embodiment of the present invention, the first adjustment coefficient is 1.32, and the second adjustment coefficient is 1.54.

[0077] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0078] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for real-time synchronization of database data across platforms and heterogeneous structures, characterized in that: include: Step S1, extracting pre-stored data to be synchronized from the source database; Step S2: converting and processing the extracted pre-stored data to adapt to the format and structure of the test database; Step S3, loading the converted data into the test database; Step S4, the data analysis module preliminarily determines whether the heterogeneous synchronization of the pre-stored data is qualified according to the similarity between the test data in the single test database and the pre-stored data in the source database, and determines the reason for the failure according to the similarity when the heterogeneous synchronization is preliminarily determined to be unqualified; Step S5, the data analysis module adjusts the parameters in the data heterogeneous synchronization process to corresponding values ​​according to the analysis results, re-synchronizes the data heterogeneously, and the data analysis module re-detects; Step S6: When the data analysis module determines that the heterogeneous synchronization of the pre-stored data is qualified, the heterogeneous synchronization of the pre-stored data is completed; The data analysis module determines whether the heterogeneous synchronization of the pre-stored data in the test database is qualified according to the similarity between the test data in the single test database and the pre-stored data in the source database. When the data analysis module preliminarily determines that the heterogeneous synchronization of the pre-stored data in the test database is unqualified, it determines whether the data conversion in the heterogeneous synchronization process is qualified according to the distribution of different bytes in the test database that are not similar to those in the source database, or determines the reason for its failure according to the similarity; The data analysis module determines whether the data conversion in the heterogeneous synchronization process is qualified according to the distribution of the heterogeneous bytes, and When determining that data conversion in the heterogeneous synchronization process is qualified, the data analysis module determines that the reason why the heterogeneous synchronization in the test database is unqualified is that data transmission is damaged; The data analysis module determines the reason for the failure when determining that the data conversion in the heterogeneous synchronization process is unqualified; When the data analysis module determines that the reason for heterogeneous synchronization failure in the test database is damaged data transmission, it sets several correction methods for bytes in the buffer according to the number of simultaneous transmission devices, and the correction amplitude of each correction method for bytes is different.

2. The method for real-time synchronization of database data across platforms and heterogeneous structures according to claim 1, characterized in that: The data analysis module records the ratio of the similarity to the second preset similarity in the data analysis module as a primary ratio when determining the reason for the failure according to the similarity, and determines the reason for the failure of heterogeneous synchronization in the test database according to the primary ratio, the reason including data loading failure and data conversion failure; When determining to mark the test database as an abnormal database, the data analysis module determines the reason why the heterogeneous synchronization in the test database is unqualified according to the proportion of the abnormal databases.

3. The method for real-time synchronization of database data across platforms and heterogeneous structures according to claim 2 is characterized in that: When the data analysis module determines that the reason for the failure of heterogeneous synchronization in the test database is that the data loading is unqualified, the data analysis module uses a compression algorithm to compress the data volume to optimize the connection settings and transmission protocols for the distributed data.

4. The method for real-time synchronization of database data across platforms and heterogeneous structures according to claim 3 is characterized in that: The data analysis module records the ratio of the number of abnormal databases to the total number of test databases as a secondary ratio, and determines the reasons for the failure of heterogeneous synchronization in the test database according to the secondary ratio, the reasons including: data conversion failure and data loading failure.

5. The method for real-time synchronization of database data across platforms and heterogeneous structures according to claim 4 is characterized in that: When determining that the data conversion is unqualified, the data analysis module is provided with a plurality of adjustment modes for the sampling rate according to the order of the filter, and the adjustment range of the sampling rate in each adjustment mode is different.

Images