Data consistency determination method and device, electronic equipment and readable storage medium
By monitoring database change events and using change object information to detect master-slave database consistency, the problem of inefficient data consistency detection in the existing technology is solved, and efficient data consistency management is realized, suitable for high concurrency and large data volume scenarios .
Patent Information
- Application Number
- CN202411079753.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-22
AI Technical Summary
In the prior art, data consistency detection efficiency is low and it is difficult to meet the needs of high concurrency and large data volume application scenarios.
By monitoring the data change events of the main database, obtaining change information, and determining the data consistency between the master database and the slave database based on the change object information, avoiding comparison of all data, and performing exception processing operations to resolve inconsistencies.
It improves the efficiency of data consistency detection, and can accurately detect data consistency in high concurrency and large data volume scenarios to adapt to complex business needs.
Smart Images

Figure CN119003548A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of databases, and in particular to a method, device, electronic device and readable storage medium for determining data consistency. Background Art
[0002] With the increasing complexity and high concurrency requirements of Internet business, data access efficiency and consistency have become important issues in system design. In many application scenarios, it is often necessary to query data from multiple data sources, such as relational databases represented by MySQL and cache systems represented by Redis. In order to improve query efficiency, data is often synchronized from MySQL to Redis through MySQL IO; the setting of relevant configuration items determines whether to query data from MySQL or Redis; and when MySQL and Redis are synchronized, if there is a high delay, the corresponding data in MySQL and Redis may be inconsistent, which brings risks to the business. Therefore, it is necessary to detect the consistency of data in MySQL and Redis; in the prior art, consistency detection is usually performed by directly comparing the data in MySQL and Redis in sequence, which is inefficient and difficult to implement in high-concurrency and large-data-volume application scenarios. Summary of the invention
[0003] The main purpose of the present invention is to propose a data consistency determination method, device, electronic device and readable storage medium, aiming to solve the problem that the detection efficiency of data consistency in the prior art is low and it is difficult to meet the needs of high concurrency and large data volume application scenarios.
[0004] To achieve the above object, the present invention provides a method for determining data consistency, the method comprising the steps of:
[0005] Monitor data change events of the master database to obtain data change information corresponding to the data change events;
[0006] Acquire change object information corresponding to the data change information from the master database and a slave database corresponding to the master database;
[0007] The data consistency between the master database and the slave database is determined according to the change object information.
[0008] Optionally, determining the data consistency between the master database and the slave database according to the change object information includes:
[0009] Acquire first object information and second object information in the changed object information, wherein the first object information is the changed object information in the master database, and the second object information is the changed object information in the slave database;
[0010] Determining whether the first object information is consistent with the second object information;
[0011] If the first object information is inconsistent with the second object information, it is considered that the master database and the slave database have data consistency in the data change event.
[0012] Optionally, the determining whether the first object information is consistent with the second object information includes:
[0013] Obtaining a first data version number in the first object information and a second data version number in the second object information;
[0014] Determine whether the first data version number is consistent with the second data version number;
[0015] If the first data version number is consistent with the second data version number, it is considered that the first object information is consistent with the second object information.
[0016] Optionally, the determining whether the first object information is consistent with the second object information includes:
[0017] Obtaining a first data summary corresponding to the first object information and a second data summary in the second object information;
[0018] Determining whether the first data summary is consistent with the second data summary;
[0019] If the first data digest is consistent with the second data digest, it is considered that the first object information is consistent with the second object information.
[0020] Optionally, after determining the data consistency between the master database and the slave database according to the change object information, the method further includes:
[0021] If the master database and the slave database do not have data consistency in the data change event, determining different data information between the master database and the slave database;
[0022] An exception handling operation is performed according to the different data information.
[0023] Optionally, performing an exception handling operation according to the different data information includes:
[0024] Acquire a difference ratio in the different data information, wherein the difference ratio is a ratio of inconsistent data between the master database and the slave database to the total data;
[0025] Obtaining a preset ratio threshold, and determining whether the difference ratio reaches the preset ratio threshold;
[0026] If the difference ratio reaches the preset ratio threshold, an exception handling operation is performed according to the difference ratio.
[0027] Optionally, performing an exception handling operation according to the different data information includes:
[0028] Acquire a difference ratio in the different data information, wherein the difference ratio is a ratio of inconsistent data between the master database and the slave database to the total data;
[0029] The business query ratio is set according to the difference ratio, wherein the business query ratio includes a main database query ratio and a slave database query ratio, and the main database query ratio is positively correlated with the difference ratio.
[0030] To achieve the above object, the present invention further provides a data consistency determination device, the data consistency determination device comprising:
[0031] A first monitoring module is used to monitor data change events of the primary database and obtain data change information corresponding to the data change events;
[0032] A first acquisition module, configured to acquire change object information corresponding to the data change information from the master database and a slave database corresponding to the master database;
[0033] The first determining module is used to determine the data consistency between the master database and the slave database according to the change object information.
[0034] Optionally, the first determining module includes:
[0035] A first acquiring unit, configured to acquire first object information and second object information in the changed object information, wherein the first object information is the changed object information in the master database, and the second object information is the changed object information in the slave database;
[0036] A first judging unit, configured to judge whether the first object information is consistent with the second object information;
[0037] The first execution unit is configured to consider that the master database and the slave database have data consistency in the data change event if the first object information is inconsistent with the second object information.
[0038] Optionally, the first determining unit includes:
[0039] A first acquisition subunit, configured to acquire a first data version number in the first object information and a second data version number in the second object information;
[0040] A first judging subunit, used to judge whether the first data version number is consistent with the second data version number;
[0041] The first execution subunit is configured to consider that the first object information is consistent with the second object information if the first data version number is consistent with the second data version number.
[0042] Optionally, the first determining unit includes:
[0043] A second acquisition subunit is used to acquire a first data summary corresponding to the first object information and a second data summary in the second object information;
[0044] A second judgment subunit, used to judge whether the first data summary is consistent with the second data summary;
[0045] The second execution subunit is configured to consider that the first object information is consistent with the second object information if the first data summary is consistent with the second data summary.
[0046] Optionally, the device further comprises:
[0047] A second determination module, configured to determine different data information between the master database and the slave database if the master database and the slave database do not have data consistency in the data change event;
[0048] The first execution module is used to execute an exception handling operation according to the different data information.
[0049] Optionally, the first execution module includes:
[0050] A second acquisition unit is used to acquire a difference ratio in the different data information, wherein the difference ratio is a ratio of inconsistent data between the master database and the slave database to the total data;
[0051] A third acquisition unit, used to acquire a preset ratio threshold and determine whether the difference ratio reaches the preset ratio threshold;
[0052] The second execution unit is configured to execute an exception handling operation according to the difference ratio if the difference ratio reaches the preset ratio threshold.
[0053] Optionally, the first execution module includes:
[0054] a fourth acquisition unit, configured to acquire a difference ratio in the different data information, wherein the difference ratio is a ratio of inconsistent data between the master database and the slave database to the total data;
[0055] The first setting unit is used to set the business query ratio according to the difference ratio, wherein the business query ratio includes a main database query ratio and a slave database query ratio, and the main database query ratio is positively correlated with the difference ratio.
[0056] To achieve the above objectives, the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the data consistency determination method as described above when executed by the processor.
[0057] To achieve the above object, the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data consistency determination method described above are implemented.
[0058] The present invention proposes a method, device, electronic device and readable storage medium for determining data consistency, which monitors data change events of a master database to obtain data change information corresponding to the data change events; obtains change object information corresponding to the data change information from the master database and a slave database corresponding to the master database; and determines the data consistency between the master database and the slave database based on the change object information. By monitoring data change events, the changes in the data in the database can be determined in a timely manner, and the data consistency between the master database and the slave database is determined based on the change object information, so that there is no need to compare all the data in the database. At the same time, each data change event will perform data consistency detection, which improves the efficiency of data consistency detection on the basis of ensuring the accuracy of data consistency detection, and can adapt to high-concurrency and large-data volume application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0061] Figure 1 A schematic diagram of a flow chart of a first embodiment of a method for determining data consistency according to the present invention;
[0062] Figure 2 A system schematic diagram of the data consistency determination method of the present invention;
[0063] Figure 3 It is a schematic diagram of the module structure of the electronic device of the present invention. DETAILED DESCRIPTION
[0064] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only embodiments of a part of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.
[0065] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0066] The present invention provides a data consistency determination method, which is applied to a data consistency determination device. The data consistency determination device can be a real-time data comparison system. Figure 1 , Figure 1 This is a flow chart of a first embodiment of a method for determining data consistency according to the present invention, wherein the method comprises the following steps:
[0067] Step S10, monitoring the data change event of the master database to obtain data change information corresponding to the data change event;
[0068] The master database is the database that directly performs data writing, updating and other change operations;
[0069] The slave database is a database whose data is synchronized with the master database.
[0070] In this embodiment, the master database is a relational database, specifically MySQL, and the slave database is a cache system, specifically Redis, for example.
[0071] Data change events are used to indicate data changes in the master database; data changes include data writing, data deletion, and data updating.
[0072] The data change information indicates the specific content of the data change event; the data change information includes but is not limited to the data type, change type, data identifier, and timestamp.
[0073] The specific monitoring method for data change events can be set based on actual needs; for example, in this embodiment, a RocketMQ component is set to subscribe to data change events of business operations; when the business operation is executed, the system generates relevant data corresponding to the business operation, and updates the relevant data to the master database, the data change event is triggered, and data change information corresponding to the data change event is generated, and the data change information is sent to the subscribing RocketMQ component; after receiving the data change information, the RocketMQ component sends the data change information to the real-time data comparison system.
[0074] Step S20, acquiring change object information corresponding to the data change information from the master database and a slave database corresponding to the master database;
[0075] When a data change event is triggered, it indicates that data has been changed in the master database. In the case of normal synchronization, after the change is made in the master database, the master database synchronizes the changed information to the slave database through MySQL IO. The content of the data change based on the data change event is the same between the master database and the slave database.
[0076] When an exception occurs in synchronization, after the change is made in the master database, the changed information is not successfully synchronized to the slave database, and the content of the data change based on the data change event is different between the master database and the slave database; for example, when the data change event is the writing of new data, the newly written data exists in the master database, but does not exist in the slave database; for another example, when the data change event is a data update, the updated related data is stored in the master database, while the related data before the update is still stored in the slave database.
[0077] The change object information indicates the relevant information of the data corresponding to the data change event in the master database and the slave database; for example, in the case of normal synchronization, the change object information respectively indicates the content of the data change based on the data change event in the master database and the slave database; for another example, when an abnormality occurs in the synchronization, the change object information respectively indicates the information of the data whose change has been completed in the master database, and the information of the data whose change has not been completed in the slave database.
[0078] Step S30: determining the data consistency between the master database and the slave database according to the change object information.
[0079] The change object information indicates the relevant information of the data corresponding to the data change event in the master database and the slave database; therefore, through the change object information, it is possible to know whether the master database and the slave database have completed data synchronization in this data change event, thereby determining the data consistency between the master database and the slave database.
[0080] It can be understood that when the present embodiment begins to be applied, the master database and the slave database complete the synchronization of initial data. At this time, the data in the master database and the slave database remain consistent, or the data consistency relationship between the master database and the slave database is known. On this basis, each time a data change event is triggered, the data consistency between the master database and the slave database is determined, so that the impact of each data change event on the data consistency between the master database and the slave database can be known, and the overall data consistency of the master database and the slave database can be determined by comprehensively analyzing the change object information corresponding to each data change event. For example, when a data change event is triggered, it can be determined that all previous data change events have not caused data inconsistency between the master database and the slave database. Therefore, it can be determined that the data in the master database and the slave database are completely consistent. On this basis, if the change object information corresponding to this data change event still indicates data consistency, it is considered that the data in the master database and the slave database are still completely consistent. If the change object information corresponding to this data change event indicates data inconsistency, it is considered that there is data inconsistency between the master database and the slave database in this data change event. For another example, when a data change event is triggered, it is determined that there were certain data change events that caused data inconsistency between the master database and the slave database. On this basis, if the change object information corresponding to this data change event indicates data consistency, it is considered that the inconsistent data between the master database and the slave database is the same as before. If the change object information corresponding to this data change event indicates data inconsistency, it is considered that the degree of data inconsistency between the master database and the slave database is further aggravated.
[0081] This embodiment can timely determine the changes in data in the database by monitoring data change events, and determine the data consistency between the master database and the slave database based on the change object information, so that there is no need to compare all data in the database. At the same time, each data change event will perform data consistency detection. On the basis of ensuring the accuracy of data consistency detection, the efficiency of data consistency detection is improved, and it can adapt to application scenarios with high concurrency and large data volumes.
[0082] Further, see also Figure 2 In the second embodiment of the method for determining data consistency of the present invention proposed based on the first embodiment of the present invention, step S30 includes the steps of:
[0083] Step S31, acquiring first object information and second object information in the changed object information, wherein the first object information is the changed object information in the master database, and the second object information is the changed object information in the slave database;
[0084] Step S32, determining whether the first object information is consistent with the second object information;
[0085] Step S33: If the first object information is inconsistent with the second object information, it is considered that the master database and the slave database have data consistency in the data change event.
[0086] The first object information indicates relevant information of data in the master database that has been changed based on the data change event; the second object information indicates relevant information of data in the slave database that has been changed based on the data change event.
[0087] It can be understood that, when the master database and the slave database complete normal synchronization based on the data change event, as a result, the master database and the slave database complete the same data change, and therefore, the first object information and the second object information obtained thereby should be consistent; conversely, when the master database and the slave database have a synchronization exception based on the data change time, as a result, there is a difference between the data change situations of the master database and the slave database, and even the slave database has not changed the data, and therefore, the first object information and the second object information obtained thereby are inconsistent; therefore, the data consistency of the master database and the slave database in the data change event can be determined by the consistency of the first object information and the second object information; when the first object information is consistent with the second object information, it is considered that the master database and the slave database have data consistency in the data change event, and when the first object information is inconsistent with the second object information, it is considered that the master database and the slave database do not have data consistency in the data change event.
[0088] Furthermore, the step S32 includes the steps of:
[0089] Step S321, obtaining a first data version number in the first object information and a second data version number in the second object information;
[0090] Step S322, determining whether the first data version number is consistent with the second data version number;
[0091] Step S323: If the first data version number is consistent with the second data version number, it is considered that the first object information is consistent with the second object information.
[0092] The first data version number is the version number of the data in the master database corresponding to the data change event; the second data version number is the version number of the data in the slave data corresponding to the data change event.
[0093] It can be understood that when specific data in the database changes, its corresponding version number will be updated accordingly to indicate the latest status of the relevant data; when the master database and the slave database complete normal synchronization based on the data change event, before the data change event, the version numbers of the data corresponding to the data change event in the master database and the slave database are consistent. After the data in the master database is changed based on the data change event, the corresponding data updates the data version number to obtain the first data version number, and then generates the first object information. The master database synchronizes the corresponding data to the slave database. After the synchronization is completed, the corresponding data in the slave database updates the data version number to obtain the second data version number, and then generates the second object information. The first data version number is consistent with the second data version number, and then the first object information is consistent with the second object information.
[0094] In the case where a synchronization anomaly occurs between the master database and the slave database based on the data change time, after the data in the master database is changed based on the data change event, the data version number of the corresponding data is updated to obtain the first data version number, and then the first object information is generated; the data in the master database cannot be successfully synchronized to the slave database, and therefore, the version number of the corresponding data in the slave database is not updated, and therefore, the second data version number is still the version number before the data change, and the second object information is obtained based on the second data version number, the first data version number is inconsistent with the second data version number, and thus the first object information is inconsistent with the second object information.
[0095] Therefore, based on the comparison of whether the first data version number is consistent with the second data version number, it is possible to determine whether the first object information is consistent with the second object information, and further determine whether the master database and the slave database have data consistency in the data change event.
[0096] Furthermore, the step S32 includes the steps of:
[0097] Step S324, obtaining a first data summary corresponding to the first object information and a second data summary in the second object information;
[0098] Step S325, determining whether the first data summary is consistent with the second data summary;
[0099] Step S326: If the first data summary is consistent with the second data summary, it is considered that the first object information is consistent with the second object information.
[0100] Data summaries are used to indicate the characteristics of data;
[0101] The first data summary is a data summary of the data corresponding to the data change event in the master database; the second data summary is a data summary of the data corresponding to the data change event in the slave data.
[0102] It can be understood that when the content of specific data in the database changes, its corresponding data summary will be updated accordingly to indicate the latest characteristics of the relevant data; when the master database and the slave database complete normal synchronization based on the data change event, before the data change event, the data summary of the data corresponding to the data change event in the master database and the slave database is consistent, after the data of the master database is changed based on the data change event, the corresponding data is updated with the data summary to obtain the first data summary, and then the first object information is generated, the master database synchronizes the corresponding data to the slave database, and after the synchronization is completed, the corresponding data in the slave database is updated with the data summary to obtain the second data summary, and then the second object information is generated, the first data summary is consistent with the second data summary, and then the first object information is consistent with the second object information.
[0103] In the case where a synchronization anomaly occurs between the master database and the slave database based on the data change time, after the data in the master database is changed based on the data change event, the data summary of the corresponding data is updated to obtain the first data summary, and then the first object information is generated; the data in the master database cannot be successfully synchronized to the slave database, and therefore, the data summary of the corresponding data in the slave database is not updated, and therefore, the second data summary is still the data summary before the data change, and the second object information is obtained based on the second data summary, the first data summary is inconsistent with the second data summary, and thus the first object information is inconsistent with the second object information.
[0104] Therefore, based on comparing whether the first data digest is consistent with the second data digest, it is possible to determine whether the first object information is consistent with the second object information, and further determine whether the master database and the slave database have data consistency in the data change event.
[0105] It can be understood that the above consistency judgment methods based on data version number and data summary are all for judging whether the corresponding data in the master database and the slave database are consistent. In practical applications, other parameters and methods can be selected based on actual needs to compare and determine whether the data corresponding to the data change event in the master database and the slave database are consistent.
[0106] Further, in a third embodiment of the method for determining data consistency of the present invention proposed based on the first embodiment of the present invention, the step S30 includes the following steps:
[0107] Step S40, if the master database and the slave database do not have data consistency in the data change event, determining different data information between the master database and the slave database;
[0108] Step S50, performing an exception handling operation according to the different data information.
[0109] When the master database and the slave database do not have data consistency in a data change event, it means that the data corresponding to the data change event is inconsistent between the master database and the slave database; and when there is data inconsistency between the master database and the slave database, when querying the data in the slave database, wrong data may be queried, causing business anomalies. Therefore, exception handling operations are required to solve or avoid business anomalies as much as possible.
[0110] The divergent data information is used to indicate inconsistent data between the master database and the slave database; it can be understood that before the data change event, the divergent data information is certain, and when the data corresponding to the data change event does not have data consistency, the relevant information of the inconsistent data generated by this data change event is added to the original divergent data information to achieve the update of the divergent data information, that is, the divergent data information always indicates the overall data inconsistency between the current master database and the slave database.
[0111] Therefore, performing exception handling operations based on different data information can handle actual data inconsistencies to achieve the best response.
[0112] Furthermore, the step S50 includes the steps of:
[0113] Step S51, obtaining a difference ratio in the different data information, wherein the difference ratio is a ratio of inconsistent data between the master database and the slave database to the total data;
[0114] Step S52, obtaining a preset ratio threshold, and determining whether the difference ratio reaches the preset ratio threshold;
[0115] Step S53: if the difference ratio reaches the preset ratio threshold, an exception handling operation is performed according to the difference ratio.
[0116] The difference ratio indicates the degree of data inconsistency between the master database and the slave database. It can be understood that the larger the difference ratio is, the higher the degree of data inconsistency between the master database and the slave database is.
[0117] In actual applications, when the difference ratio is low, although there are inconsistencies in the data between the master database and the slave database, the impact on the actual business is small. Therefore, in order to ensure the optimal performance of the master database and the slave database, the exception handling operation may not be performed; when the difference ratio is high, it means that it will have a certain impact on the actual business. Therefore, it is necessary to perform an exception handling operation to avoid business anomalies as much as possible; based on this, a preset ratio threshold is set in this embodiment. When the difference ratio is less than the preset ratio threshold, it is considered that the impact on the actual business is small and the exception handling operation is not performed. When the difference ratio is greater than or equal to the preset ratio threshold, it is considered that there is a certain impact on the actual business and the exception handling operation is performed; the specific value of the preset ratio threshold can be set based on actual needs, such as 5%.
[0118] In other embodiments, the number of differences in inconsistent data between the master database and the slave database in the different data information can also be obtained, and a preset number threshold can be set; when the number of differences is less than the preset number threshold, no exception handling operation is performed, and when the number of differences is greater than or equal to the preset number threshold, an exception handling operation is performed.
[0119] Furthermore, the step S50 includes the steps of:
[0120] Step S54, obtaining a difference ratio in the different data information, wherein the difference ratio is a ratio of inconsistent data between the master database and the slave database to the total data;
[0121] Step S55, setting the business query ratio according to the difference ratio, wherein the business query ratio includes a main database query ratio and a slave database query ratio, and the main database query ratio is positively correlated with the difference ratio.
[0122] It is understandable that when performing business queries, the corresponding data can be obtained from the main database or the slave database. The business query ratio of the main database and the slave database can be set in advance in the configuration center. For example, when the business query ratio is 7:3, for every ten businesses, seven businesses query from the main database and three businesses query from the slave database.
[0123] When there is data inconsistency between the master database and the slave database, the query from the slave data may contain data errors, resulting in business anomalies. In this embodiment, in order to solve this problem, the business query ratio is set according to the difference ratio. Specifically, the master database query ratio is positively correlated with the difference ratio, that is, the higher the difference ratio, the higher the possibility of data errors in the slave database. Therefore, more queries are required from the master database. Therefore, the master data query ratio is increased, so that more businesses can query from the master database to ensure the correctness of the data.
[0124] For example, when the difference between the master database and the slave database is less than 5%, the business query ratio is 7:3. When the difference between the master database and the slave database reaches 5%, the business query ratio of the configuration center is adjusted to 9:1, that is, for every ten businesses, nine businesses query from the master database and one business queries from the slave database.
[0125] The specific method of exception handling operation can also be set based on actual needs, such as setting an alarm threshold. When the difference ratio is greater than the alarm threshold, an alarm operation is performed to remind relevant personnel to deal with data inconsistencies; or an automatic repair mechanism is set to automatically synchronize or repair different data.
[0126] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.
[0127] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0128] The present application also provides a data consistency determination device for implementing the above data consistency determination method, the data consistency determination device comprising:
[0129] A first monitoring module is used to monitor data change events of the primary database and obtain data change information corresponding to the data change events;
[0130] A first acquisition module, configured to acquire change object information corresponding to the data change information from the master database and a slave database corresponding to the master database;
[0131] The first determining module is used to determine the data consistency between the master database and the slave database according to the change object information.
[0132] This data consistency determination device can timely determine the changes in data in the database by monitoring data change events, and determine the data consistency between the master database and the slave database based on the change object information, so that there is no need to compare all the data in the database. At the same time, each data change event will perform data consistency detection. On the basis of ensuring the accuracy of data consistency detection, the efficiency of data consistency detection is improved, and it can adapt to application scenarios with high concurrency and large data volumes.
[0133] It should be noted that the first monitoring module in this embodiment can be used to execute step S10 in the embodiment of the present application, the first acquisition module in this embodiment can be used to execute step S20 in the embodiment of the present application, and the first determination module in this embodiment can be used to execute step S30 in the embodiment of the present application.
[0134] Furthermore, the first determining module includes:
[0135] A first acquiring unit, configured to acquire first object information and second object information in the changed object information, wherein the first object information is the changed object information in the master database, and the second object information is the changed object information in the slave database;
[0136] A first judging unit, configured to judge whether the first object information is consistent with the second object information;
[0137] The first execution unit is configured to consider that the master database and the slave database have data consistency in the data change event if the first object information is inconsistent with the second object information.
[0138] Furthermore, the first determining unit includes:
[0139] A first acquisition subunit, configured to acquire a first data version number in the first object information and a second data version number in the second object information;
[0140] A first judging subunit, used to judge whether the first data version number is consistent with the second data version number;
[0141] The first execution subunit is configured to consider that the first object information is consistent with the second object information if the first data version number is consistent with the second data version number.
[0142] Furthermore, the first determining unit includes:
[0143] A second acquisition subunit is used to acquire a first data summary corresponding to the first object information and a second data summary in the second object information;
[0144] A second judgment subunit, used to judge whether the first data summary is consistent with the second data summary;
[0145] The second execution subunit is configured to consider that the first object information is consistent with the second object information if the first data summary is consistent with the second data summary.
[0146] Furthermore, the device also includes:
[0147] A second determination module, configured to determine different data information between the master database and the slave database if the master database and the slave database do not have data consistency in the data change event;
[0148] The first execution module is used to execute an exception handling operation according to the different data information.
[0149] Furthermore, the first execution module includes:
[0150] A second acquisition unit is used to acquire a difference ratio in the different data information, wherein the difference ratio is a ratio of inconsistent data between the master database and the slave database to the total data;
[0151] A third acquisition unit, used to acquire a preset ratio threshold and determine whether the difference ratio reaches the preset ratio threshold;
[0152] The second execution unit is configured to execute an exception handling operation according to the difference ratio if the difference ratio reaches the preset ratio threshold.
[0153] Furthermore, the first execution module includes:
[0154] a fourth acquisition unit, configured to acquire a difference ratio in the different data information, wherein the difference ratio is a ratio of inconsistent data between the master database and the slave database to the total data;
[0155] The first setting unit is used to set the business query ratio according to the difference ratio, wherein the business query ratio includes a main database query ratio and a slave database query ratio, and the main database query ratio is positively correlated with the difference ratio.
[0156] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.
[0157] Reference Figure 3 In terms of hardware structure, the electronic device may include components such as a communication module 10, a memory 20, and a processor 30. In the electronic device, the processor 30 is connected to the memory 20 and the communication module 10 respectively, and a computer program is stored in the memory 20. The computer program is executed by the processor 30 at the same time, and the steps of the above method embodiment are implemented when the computer program is executed.
[0158] The communication module 10 can be connected to an external communication device through a network. The communication module 10 can receive requests from an external communication device, and can also send requests, instructions and information to the external communication device, which can be other electronic devices, servers or Internet of Things devices, such as televisions, etc.
[0159] The memory 20 can be used to store software programs and various data. The memory 20 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function (such as monitoring data change events of a master database), etc.; the data storage area can include a database, and the data storage area can store data or information created according to the use of the system, etc. In addition, the memory 20 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0160] The processor 30 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 20, and calling data stored in the memory 20, so as to monitor the electronic device as a whole. The processor 30 may include one or more processing units; optionally, the processor 30 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 30.
[0161] although Figure 3 Although not shown, the electronic device may further include a circuit control module, which is used to connect to a power source to ensure the normal operation of other components. Figure 3 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.
[0162] The present invention also provides a computer-readable storage medium on which a computer program is stored. The computer-readable storage medium may be Figure 3 The memory 20 in the electronic device may also be at least one of a ROM (Read-Only Memory) / RAM (Random Access Memory), a magnetic disk, and an optical disk. The computer-readable storage medium includes a number of instructions for enabling a terminal device with a processor (which may be a television, a car, a mobile phone, a computer, a server, a terminal, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.
[0163] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.
[0164] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.
[0165] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.
[0166] In the present invention, the terms "first", "second", "third", "fourth" and "fifth" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0167] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0168] Although the embodiments of the present invention have been shown and described above, the scope of protection of the present invention is not limited thereto. It is understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art can change, modify and replace the above embodiments within the scope of the present invention, and these changes, modifications and replacements should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.
Claims
1. A method for determining data consistency, characterized in that: The data consistency determination method comprises: Monitor data change events of the master database to obtain data change information corresponding to the data change events; Acquire change object information corresponding to the data change information from the master database and a slave database corresponding to the master database; The data consistency between the master database and the slave database is determined according to the change object information.
2. The data consistency determination method according to claim 1, characterized in that: Determining the data consistency between the master database and the slave database according to the change object information includes: Acquire first object information and second object information in the changed object information, wherein the first object information is the changed object information in the master database, and the second object information is the changed object information in the slave database; Determining whether the first object information is consistent with the second object information; If the first object information is inconsistent with the second object information, it is considered that the master database and the slave database have data consistency in the data change event.
3. The data consistency determination method according to claim 2, characterized in that: The determining whether the first object information is consistent with the second object information includes: Obtaining a first data version number in the first object information and a second data version number in the second object information; Determine whether the first data version number is consistent with the second data version number; If the first data version number is consistent with the second data version number, it is considered that the first object information is consistent with the second object information.
4. The data consistency determination method according to claim 2, characterized in that: The determining whether the first object information is consistent with the second object information includes: Obtaining a first data summary corresponding to the first object information and a second data summary in the second object information; Determining whether the first data summary is consistent with the second data summary; If the first data digest is consistent with the second data digest, it is considered that the first object information is consistent with the second object information.
5. The data consistency determination method according to claim 1, characterized in that: After determining the data consistency between the master database and the slave database according to the change object information, the method further includes: If the master database and the slave database do not have data consistency in the data change event, determining different data information between the master database and the slave database; An exception handling operation is performed according to the different data information.
6. The data consistency determination method according to claim 5, characterized in that: The performing of the exception handling operation according to the different data information comprises: Acquire a difference ratio in the different data information, wherein the difference ratio is a ratio of inconsistent data between the master database and the slave database to the total data; Obtaining a preset ratio threshold, and determining whether the difference ratio reaches the preset ratio threshold; If the difference ratio reaches the preset ratio threshold, an exception handling operation is performed according to the difference ratio.
7. The data consistency determination method according to claim 5, characterized in that: The performing of the exception handling operation according to the different data information comprises: Acquire a difference ratio in the different data information, wherein the difference ratio is a ratio of inconsistent data between the master database and the slave database to the total data; The business query ratio is set according to the difference ratio, wherein the business query ratio includes a main database query ratio and a slave database query ratio, and the main database query ratio is positively correlated with the difference ratio.
8. A data consistency determination device, characterized in that: The data consistency determination device comprises: A first monitoring module is used to monitor data change events of the primary database and obtain data change information corresponding to the data change events; A first acquisition module, configured to acquire change object information corresponding to the data change information from the master database and a slave database corresponding to the master database; The first determining module is used to determine the data consistency between the master database and the slave database according to the change object information.
9. An electronic device, characterized in that: The electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the data consistency determination method according to any one of claims 1 to 7 when executed by the processor.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data consistency determination method according to any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Dynamic check method and device for consistency of main and salve databases
CN104252481A
Cache health degree detection method and device
CN105573838A
Data processing method and device and electronic equipment
CN112925808A
Database migration method and device, equipment and storage medium
CN113190531A
Data consistency verification method and device, equipment and storage medium
CN114610735A
