Data consistency rate correction method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of databases, and discloses a data consistency rate correction method and device, computer equipment and a storage medium, and one scheme comprises the steps that a data calculation list table is established, the data calculation list table comprises a data lake and table names of all data tables in business databases, and the data lake is associated with one or more business databases; searching a business database value of the data table corresponding to the table name from a business database result table of the business database; retrieving a data lake value of the data table corresponding to the table name from a data lake result table of the data lake; according to the business database value and the data lake value, the consistency rate of corresponding data tables in the data lake and the business database is calculated; judging whether the consistency rate is within a preset range or not; and if the consistency rate is not within the preset range, correcting the consistency rate according to the database operation log. And the calculation accuracy of the data consistency rate is improved.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a method, device, computer equipment and storage medium for correcting data consistency rate based on a database operation log. Background Art

[0002] A business database is a database specifically used to store and manage enterprise business data, while a data lake is a platform for centralized storage and processing of large amounts of data, usually used for offline access to data from various business databases. In financial companies, two methods are usually used to synchronize data from business databases to data lakes. The first method is full synchronization, which means synchronizing all data in the business database to the data lake at a fixed time point every day. The second method is incremental synchronization, which means synchronizing only the data that changed the previous day to the data lake every day and merging it with the existing data.

[0003] Regardless of the synchronization method, since one side of the synchronization is real-time data and the other side is offline data, it is difficult to directly verify whether the data in the business database and the data lake are completely consistent after the synchronization is completed. However, the existing consistency rate calculation method often simply compares the number of data in the business database and the data lake, and the accuracy of the consistency rate calculated by this direct comparison is low and cannot meet business requirements. Therefore, how to improve the calculation accuracy of the consistency rate of the corresponding data table after the database and the data lake are synchronized is an urgent problem to be solved. Summary of the invention

[0004] The embodiments of the present invention provide a data consistency rate correction method, device, computer equipment and storage medium to solve the problem of low accuracy of the existing consistency rate calculation method.

[0005] In a first aspect, a method for correcting a data consistency rate based on a database operation log is provided, comprising: Establishing a data calculation list table, wherein the data calculation list table includes table names of various data tables in the data lake and the business database, wherein the data lake is associated with one or more of the business databases; Retrieving a business database value of a data table corresponding to the table name from a business database result table of the business database, wherein the business database value is a calculated value of the data table corresponding to the table name in the business database; Retrieving a data lake value of a data table corresponding to the table name from a data lake result table of the data lake, wherein the data lake value is a calculated value of the data table corresponding to the table name in the data lake; Calculating the consistency rate of the corresponding data tables in the data lake and the business database according to the business database value and the data lake value; Determining whether the consistency rate is within a preset range; If the consistency rate is not within the preset range, the consistency rate is corrected according to the database operation log.

[0006] In a second aspect, a data consistency rate correction device based on a database operation log is provided, comprising: An establishment module, used to establish a data calculation list table, wherein the data calculation list table includes table names of various data tables in the data lake and the business database, and the data lake is associated with one or more of the business databases; A first retrieval module, configured to retrieve a business database value of a data table corresponding to the table name from a business database result table of the business database, wherein the business database value is a calculated value of the data table corresponding to the table name in the business database; A second retrieval module is used to retrieve a data lake value of a data table corresponding to the table name from a data lake result table of the data lake, wherein the data lake value is a calculated value of the data table corresponding to the table name in the data lake; A calculation module, used to calculate the consistency rate of the data lake and the corresponding data table in the business database according to the business database value and the data lake value; A judging module, used to judge whether the consistency rate is within a preset range; The correction module is used to correct the consistency rate according to the database operation log if the consistency rate is not within the preset range.

[0007] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the data consistency rate correction method based on the database operation log described in the first aspect are implemented.

[0008] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data consistency rate correction method based on the database operation log described in the first aspect are implemented.

[0009] In one solution implemented by the above-mentioned data consistency rate correction method, device, computer equipment and storage medium, a data calculation list table is established to record the table names of each data table in the data lake and the business database; and the business database value of the data table corresponding to the table name is retrieved from the business database by using the table name, and the data lake value of the data table corresponding to the table name is retrieved from the data lake, thereby calculating the consistency rate of the corresponding data table in the data lake and the business database; by judging whether the consistency rate is within the preset range, and when the consistency rate does not reach the preset standard, automatic correction is performed based on the database operation log to accurately calculate the consistency rate of the corresponding data table in the data lake and the business database. This process improves the calculation accuracy of the data consistency rate through a precise data comparison and correction mechanism, and provides solid technical support for enterprise data analysis and decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0011] Figure 1 is a flow chart of a method for correcting data consistency rate based on a database operation log in one embodiment of the present invention; Figure 2 is another flow chart of a method for correcting data consistency rate based on a database operation log in one embodiment of the present invention; Figure 3 is another flow chart of a method for correcting data consistency rate based on a database operation log in one embodiment of the present invention; Figure 4 is another flow chart of a method for correcting data consistency rate based on a database operation log in one embodiment of the present invention; Figure 5 is another flow chart of a method for correcting data consistency rate based on a database operation log in one embodiment of the present invention; Figure 6 is another flow chart of a method for correcting data consistency rate based on a database operation log in one embodiment of the present invention; Figure 7 is a schematic diagram of a data consistency rate correction device based on a database operation log in one embodiment of the present invention; Figure 8 is a schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0013] A method for correcting the data consistency rate based on a database operation log provided by an embodiment of the present invention can be widely used in multiple fields, including but not limited to: the financial services field, the medical health field, and the scientific research and big data analysis field. As an example, in the financial services field, such as banks, securities and insurance institutions, the method can effectively monitor the data differences between the business database and the data lake. In the medical health field, the method can promptly discover the inconsistency of data between the business database and the data lake of the hospital's medical information system, providing a basis for data analysis for building smart medical care. In the field of scientific research and big data analysis, the method can effectively monitor the consistency of data and ensure the accuracy and reliability of scientific research data. Specifically, the method for correcting the data consistency rate based on the database operation log can be applied to the client. Among them, the client includes but is not limited to various personal computers, laptops, smart phones and tablet computers. The following will describe in detail a method for correcting the data consistency rate based on a database operation log provided by an embodiment of the present invention.

[0014] In one embodiment, if Figure 1 As shown, a method for correcting data consistency rate based on database operation log is provided, comprising the following steps: S10. Create a data calculation list table, where the data calculation list table includes the table names of various data tables in the data lake and the business database, and the data lake is associated with one or more of the business databases.

[0015] In this embodiment, the establishment of the data calculation list table is intended to monitor the relevant information between the data lake and each data table in multiple business databases, so that the consistency rate of the corresponding data table can be calculated based on the data list table after the data lake synchronizes data from the business database. Specifically, the data calculation list table includes the table names of all relevant data tables in the data lake and the business database, and also includes the monitoring information corresponding to each data table. Among them, the monitoring information includes key fields, delineated fields and their calculation ranges. Among them, the data tables in the data lake can be associated with the data tables in each business database by table name, so that when performing data synchronization and consistency verification, the relevant data sources can be quickly identified, and they can also be associated by other means, without specific limitation.

[0016] In some embodiments, the establishment of a data list table can be established based on table granularity, that is, the monitoring information is set based on a single data table as the basic unit. As an example, first, determine the data tables that need to be monitored, such as all data tables that need to be compared in the business database and the data lake, such as "policy table" and "transaction record table". Next, define monitoring information for each data table, including key fields, delineated fields, and calculation ranges of delineated fields. Key fields are fields used to uniquely identify data records, such as "insured amount" or "transaction amount", while delineated fields are used to define the monitoring time of data changes, such as "transaction time" or "update time". Then, set the calculation range of each delineated field, for example, select "October 1, 2024 to October 31, 2024" as the monitoring time period, so as to compare the consistency rate of data within this time. Then, organize this information to obtain a data list table. By establishing a data calculation list table, we can effectively carry out data governance and quality management, improve the reliability of data use, and provide a solid foundation for subsequent data analysis and decision-making.

[0017] S20. Retrieve a business database value of a data table corresponding to the table name from a business database result table of the business database, wherein the business database value is a calculated value of the data table corresponding to the table name in the business database.

[0018] In this embodiment, the business database result table is a table for storing the calculation result information of all data tables in the business database. Specifically, the business database result table may include a table name and a business database value, and the business database value may include a first number of data items and a first aggregate value. Among them, the first number of data items refers to the number of records in the data table within the calculation scope, and the first aggregate value refers to the aggregate value of the key field in the data table within the calculation scope. Among them, the aggregate value of the key field may include the sum, average value, maximum value and minimum value of the key field within the calculation scope, for example, the total insured amount, average value, maximum value and minimum value of the insured amount within the calculation scope of the "insurance policy table".

[0019] In some implementations, a query can be performed in the business database result table of the business database by using a preset table name to determine the business database value of the corresponding data table. This process can be implemented by SQL query language. For example, if the table name is "policy table", a SQL statement similar to "SELECT * FROM business database result table WHERE table name = 'policy table'" will be executed.

[0020] After executing the query, the business database value corresponding to the table name will be returned, and the business database value includes information such as the number of first data items and the first aggregate value of the key field. These business database values ​​reflect the real-time data status of the data table in the business database within a specific time range. For example, the returned results include the number of insurance policies and the total insured amount. It should be noted that the above is only an example and does not constitute a limitation of the present invention.

[0021] S30. Retrieve a data lake value of the data table corresponding to the table name from a data lake result table of the data lake, where the data lake value is a calculated value of the data table corresponding to the table name in the data lake.

[0022] In this embodiment, the data lake result table is used to store the calculation result information synchronized from each business database to the data table in the data lake, similar to the business database result table. Specifically, the data lake result table may include a table name and a data lake value. The data lake value includes a second number of data items and a second aggregate value, wherein the definition of the second number of data items and the second aggregate value can refer to the first number of data items and the first aggregate value, and will not be repeated here.

[0023] It should be understood that the corresponding table name retrieved in this embodiment corresponds to the table name of the above-mentioned retrieval business database, that is, the data lake value of the data table corresponding to the table name corresponding to the business database is retrieved from the data lake result table of the data lake.

[0024] In some implementations, the data lake result table can be queried by the table name corresponding to the query business database to find the data lake value corresponding to the table name in the data lake result table. This process can be implemented by SQL query or data processing framework (such as Apache Spark or Hive) to ensure the efficiency and accuracy of data retrieval. For example, execute the following query statement: "SELECT * FROM data lake result table WHERE table name = 'policy table'".

[0025] After executing the query, the data lake value of the data table corresponding to the "policy table" in the business database will be returned. The data lake value includes information such as the second number of data items and the second aggregate value of the key field. The data lake value represents the status of the data table corresponding to the table name in the data lake. Among them, the returned results also include the number of insurance policies and the total insured amount. It should be noted that the above is only an example and does not constitute a limitation of the present invention.

[0026] The above steps S20 and S30 provide the necessary data basis for subsequent data consistency comparison by extracting the business database value and the data lake value into the memory.

[0027] S40. Calculate the consistency rate of the data lake and the corresponding data table in the business database according to the business database value and the data lake value.

[0028] In some implementations, after the business database value and the data lake value are obtained through query, the following formula can be used to further obtain the business database value and the data lake value: The consistency rate of the corresponding data tables in the data lake and the business database is calculated. Through this process, potential data differences can be effectively identified, providing a reliable basis for subsequent data analysis and decision-making.

[0029] As an example, assume that the business database value retrieved from the business database is B (including the first number of data items and the first aggregate value), and the data lake value retrieved from the data lake is D (including the second number of data items and the second aggregate value). On this basis, the calculation of the consistency rate can be divided into two aspects: the consistency rate of the number of data items and the consistency rate of the aggregate value.

[0030] Calculation of data consistency rate: in, Indicates the first data entry number of the corresponding data table in the business database. Indicates the number of second data items in the corresponding data table in the data lake. This calculation can directly reflect the consistency rate of the number of records between the two.

[0031] Calculation of the consistency rate of aggregated values: in, Represents the first aggregate value of the key field of the corresponding data table in the business database, and Indicates the second aggregate value of the corresponding data table in the data lake. This calculation reflects the consistency rate between the two at the numerical calculation level.

[0032] The following is a specific example that illustrates how to calculate the consistency rate based on the business database value and the data lake value. Assume that there is an "insurance policy table" with the key field being the insured amount. The corresponding data of the "insurance policy table" in the business database and the data lake are as follows: Business database value: the first number of data items ( ) is 1000; the first aggregate value of the insured amount (total insured amount, ( )): 500,000 yuan.

[0033] Data lake value: the number of second data items ( ): 950 items; the second aggregate value of the insured amount (total insured amount, ( ): 475,000 yuan.

[0034] Data consistency rate: 05 Aggregation value consistency rate: 05 It should be understood that the calculated results of the data item consistency rate and the aggregate value consistency rate are greater than 100%, indicating that the number of data items in the business database is greater than the number of data items in the data lake, which means that there are records in the business database but have not been synchronized to the data lake. For example, after the data lake synchronizes data from the business data lake, new data is added to the business database.

[0035] S50: Determine whether the consistency rate is within a preset range.

[0036] In some implementations, after calculating the consistency rate of the corresponding data tables of the data lake and the business database, it is further determined whether the consistency rate is within a preset range, where the preset range can be 95% to 100%. Specifically, when determining whether the consistency rate is within the preset range, it can be determined whether the consistency rate of the number of data items and / or the consistency rate of the aggregate value are within the preset range to determine whether the consistency rate of the current data meets the business requirements.

[0037] S60: If the consistency rate is not within the preset range, correct the consistency rate according to the database operation log.

[0038] In some embodiments, if the consistency rate is not within a preset range, the consistency rate is corrected according to the database operation log. Specifically, it includes: extracting the database operation log related to the business database, and the database operation log includes the operation records related to the addition, deletion and update of the corresponding data table in the business database after the database synchronizes data from the business database, and its corresponding timestamp and operator information. Then, by analyzing the operation record, the data offset can be obtained, and then the corrected difference amount that needs to be corrected is obtained according to the data offset. Finally, based on the corrected difference amount, the previously calculated consistency rate is corrected to obtain an accurate data consistency rate, so as to effectively deal with the situation where the calculation accuracy of the data consistency rate is low due to data changes.

[0039] In summary, in a solution provided by an embodiment of the present invention, a data calculation list table is established to record the table names of each data table in the data lake and the business database; and the business database value of the data table corresponding to the table name is retrieved from the business database by using the table name, and the data lake value of the data table corresponding to the table name is retrieved from the data lake, thereby calculating the consistency rate of the corresponding data table in the data lake and the business database; by judging whether the consistency rate is within the preset range, and when the consistency rate does not reach the preset standard, automatic correction is performed based on the database operation log to accurately calculate the consistency rate of the corresponding data table in the data lake and the business database. This process improves the calculation accuracy of the data consistency rate through a precise data comparison and correction mechanism, and provides solid technical support for enterprise data analysis and decision-making.

[0040] In one embodiment, if Figure 2 As shown, that is, after step S10, the data calculation list table also includes monitoring information of each data table in the data lake and the business database, and the monitoring information includes key fields, delimited fields and calculation ranges of delimited fields, and the delimited fields are pre-selected calculation time fields; That is, after the data calculation list table is established, the following steps are included: S70, calculating the number of data items in the corresponding data table in the business database according to the table name, the delimited field, and the calculation range, to obtain a first number of data items; S80, calculating the aggregate value of the key field in the corresponding data table in the business database according to the table name, the delimited field and the calculation range, to obtain a first aggregate value; S90. Write the first number of data items and the first aggregation value as the business database value into the business database result table.

[0041] In this embodiment, the data calculation list table includes not only the table names of each data table in the data lake and the business database, but also monitoring information. The monitoring information includes key fields, delimited fields, and delimited field calculation ranges, wherein the delimited fields are pre-selected calculation time fields used to define the time period for which data needs to be analyzed.

[0042] In some implementations, first, the records of the corresponding data table in the business database are queried based on the table name, and the number of data items that meet the conditions is calculated based on the conditions of the defined fields to obtain the first number of data items. Next, the first aggregate value of the key field in the corresponding data table is calculated based on the same table name, defined fields and their calculation range. The first aggregate value includes the sum, average, maximum and minimum values, etc. After completing the above steps, the first number of data items and the first aggregate value are written into the business database result table as the business database values. This process provides an important basis for subsequent data analysis and consistency calculations.

[0043] For example, take the "policy table" as an example, which includes multiple fields, such as "policy number", "policy amount", "insured person's name" and "insurance time". Among them, the key field is "policy amount", the delimited field is "insurance time", and the calculation range is set to "October 1, 2024 to October 31, 2024".

[0044] First, query the business database based on the table name "policy table". This process involves writing query statements, such as "SELECT COUNT(*) FROM business database.policy table WHERE insurance time BETWEEN '2024-10-01' AND '2024-10-31'", to calculate the number of policy data within the time range. For example, within the specified time period, there are 150 records in the policy table that meet the conditions, so the first data number is 150.

[0045] Next, the key field "policy amount" will be aggregated and calculated. The calculation SQL statement involved in this process is such as "SELECT SUM(insured amount) FROM policy table WHERE insurance date BETWEEN '2024-10-01'AND '2024-10-31'". Through the query, the sum of the insured amounts within the preset calculation range will be obtained. For example, assuming that the calculation result is that the total insured amount is RMB 5 million, then the first aggregate value is RMB 5,000,000.

[0046] Finally, the first data number 150 and the first aggregation value 5,000,000 yuan are written into the "Business Database Result Table" to form a record.

[0047] It should be understood that the above is only an example, and specifically, the average value, maximum value and / or minimum value of the insured amount may also be calculated as the first aggregate value of the key field, which is not limited here.

[0048] In one embodiment, if Figure 3 As shown, after step S10, that is, after the data calculation list table is established, the following steps are also included: S100, calculating the number of data items in the corresponding data table in the data lake according to the table name, the delimited field, and the calculation range of the delimited field to obtain a second number of data items; S110, calculating the aggregate value of the key field in the corresponding data table in the data lake according to the table name, the delimited field, and the calculation range of the delimited field to obtain a second aggregate value; S120. Write the second number of data items and the second aggregate value as the data lake value into the data lake result table.

[0049] In some implementations, first, the records of the corresponding data table in the data lake are queried based on the table name, and the number of data items that meet the conditions is calculated based on the conditions of the delimited fields to obtain the second number of data items. Then, based on the same table name, delimited fields and their calculation ranges, the second aggregate value of the key field in the corresponding data table is calculated, and the aggregate value includes the sum, average, maximum and minimum values, etc. After completing the above steps, the second number of data items and the second aggregate value are written into the data lake result table as the data lake value. This process also provides an important basis for subsequent data analysis and consistency calculations.

[0050] For example, the previously established data calculation list table is continued to be used to monitor the insurance policy table in the data lake. Similar to the business database, the key fields, delimited fields, and calculation scope are consistent with the above.

[0051] First, query the "policy table" in the data calculation list table to obtain the corresponding records in the data lake. Execute SQL query commands, such as "SELECT COUNT(*) FROM data lake.policy table WHERE insurance time BETWEEN'2024-10-01' AND '2024-10-31'", to calculate the number of insurance policy data that meet the conditions in the data lake within the above time range. Assume that the query results show that there are 140 insurance policy records that meet the conditions in the data lake, so the second data number is 140.

[0052] Next, the key field "policy amount" will be aggregated and calculated. The calculation SQL statement involved in this process is such as "SELECT SUM(insured amount) FROM policy table WHERE insurance date BETWEEN '2024-10-01'AND '2024-10-31'". Through the query, the sum of the insured amounts within the preset calculation range will be obtained. For example, assuming that the calculation result is that the total insured amount is RMB 4.8 million, then the first aggregate value is RMB 4,800,000.

[0053] Finally, the second data number 140 and the second aggregation value 4,800,000 yuan are written into the "Data Lake Result Table" to form a record.

[0054] It should be understood that the above is only an example, and specifically, the average value, maximum value and / or minimum value of the insured amount may also be calculated as the second aggregate value of the key field, which is not limited here.

[0055] In the above steps S70 to S120, the business database value and the data lake value are obtained by calculation, which provides data support for subsequent data comparison and consistency check.

[0056] In one embodiment, if Figure 4 As shown, in step S60, that is, if the consistency rate is not within the preset range, the consistency rate is corrected according to the database operation log, including the following steps: S61. Obtain a preset synchronization strategy, where the synchronization strategy includes a first synchronization strategy and a second synchronization strategy, where the first synchronization strategy includes synchronization date and synchronization period data when the data lake synchronizes data from the business database, and the second synchronization strategy does not include synchronization date and synchronization period data when the data lake synchronizes data from the business database; S62: If the synchronization strategy is the first synchronization strategy, obtain a first database operation log during synchronization of the business database; S63, correcting the consistency rate according to the first database operation log; S64: If the synchronization strategy is the second synchronization strategy, obtain a second database operation log on the synchronization day of the business database; S65. Correct the consistency rate according to the second database operation log.

[0057] In some embodiments, if it is determined that the consistency rate is not within the preset range, it indicates that the consistency rate at this time cannot meet the business needs. At this time, it is necessary to obtain a pre-set synchronization strategy, and obtain the corresponding database operation day according to different synchronization strategies to correct the consistency rate. Specifically, the synchronization strategy includes a first synchronization strategy and a second synchronization strategy. Among them, the first synchronization strategy is defined as including the data on the synchronization day and the synchronization period when the data lake synchronizes data from the business database; and the second synchronization strategy is defined as excluding the data on the synchronization day and the synchronization period when the data lake synchronizes data from the business database. For example, on T+1, the data lake obtains the data of T day from the business database. At this time, assuming that the synchronization start time is 2 o'clock on T+1, the time required for synchronization is 1 hour, that is, 2 o'clock to 3 o'clock, and the first synchronization strategy is characterized as including the data from 0 o'clock to 3 o'clock on T+1 when synchronizing data on T+1, and the second synchronization strategy is characterized as excluding the data from 0 o'clock to 3 o'clock on T+1 when synchronizing data on T+1.

[0058] When the synchronization strategy is the first synchronization strategy, the first database operation log during the synchronization period will be obtained from the business database. The first database operation log records the data operations related to the business database during the period when the data lake synchronizes data from the business database, such as the addition, deletion, and update of data. When the synchronization strategy is the second synchronization strategy, the second database operation log on the synchronization day will be obtained from the business database. The second database operation log records the data operations related to the business database on the day when the data lake synchronizes data from the business database, such as the addition, deletion, and update of data.

[0059] Next, the consistency rate is corrected according to the database operation logs corresponding to different synchronization strategies, thereby ensuring that the calculation accuracy of the consistency rate can be improved regardless of the synchronization strategy. And through the setting of this flexible synchronization strategy, the most suitable data synchronization method can be selected according to different business scenarios to optimize the real-time and consistency of data.

[0060] It should be understood that when starting data synchronization, the existing data needs to be packaged and "sealed", and if there is data added, deleted or updated during the synchronization period, the data at this time cannot be synchronized to the data lake. When using the first synchronization strategy for data synchronization, it is necessary to obtain the operation log during the business data synchronization period, and when using the second synchronization strategy for data synchronization, it is necessary to obtain the operation log of the business data synchronization day.

[0061] For example, take the "insurance policy table" as an example. The synchronization strategy of the business database and the data lake for the "insurance policy table" is set to the first synchronization strategy. During the synchronization process, it was found that 100 records were added to the insurance policy table and 50 records were updated. In this process, the first database operation log during the synchronization period was obtained. The first database operation log records all the new and updated operations of the business database during the synchronization period. Through the first database operation log, the consistency rate is recalculated to obtain a new data consistency rate indicator, which may be revised to 97%, for example.

[0062] In contrast, if the synchronization strategy is the second synchronization strategy, 200 records are added on the synchronization day. In this process, the second database operation log of the synchronization day is obtained, and the second database operation log records all the new additions and updates of the business database on the synchronization day. Through the second database operation log, the consistency rate is recalculated to obtain a new data consistency rate indicator, which may be revised to 98%, for example.

[0063] It should be noted that the above is only an example and does not constitute a limitation of the present invention.

[0064] In one embodiment, if Figure 5As shown, step S63, i.e., correcting the consistency rate according to the first database operation log, includes the following steps: S631, parsing the first database operation log to obtain all first data operation records in the business database during the synchronization period; S632. Calculate a first data offset of the business database and the data lake according to the first data operation record; S633, using the first data offset as a first corrected difference; S634: Correct the consistency rate according to the first corrected difference amount.

[0065] In some embodiments, the goal of parsing the first data operation log is to obtain all data operation records that occurred in the business database during the period when the data lake synchronizes data from the business database. These operation records include all data modifications made in the business database during the synchronization period, including but not limited to operations such as addition, update, and deletion. Then, based on the parsed data operation records, the first data offset between the business database and the data lake is calculated, and the calculated first data offset is used as the first corrected difference. For example, assuming that the first data offset of the business database is 20, and the data lake fails to update these 20 records during the synchronization period, then these 20 records will be regarded as the first corrected difference. Finally, based on the first corrected difference, the previously calculated consistency rate is corrected.

[0066] Specifically, the following formula can be used to correct the consistency rate under the first synchronization strategy: The first corrected difference represents the data offset during the synchronization of the business database and the data lake.

[0067] For example, assume that the first number of data items in the business database value is 5,000, and the first aggregate value is 5,000,000 yuan; the second number of data items in the data lake value is 4,800, and the second aggregate value is 4,000,000 yuan; the difference in the first number of data items in the first correction difference is 200, and the difference in the first aggregate value is 1,000,000 yuan. The above formula can be used to calculate the consistency rate of the number of data items correction: The aggregate value correction consistency rate is: It should be noted that the above is only an example and the present invention is not limited thereto.

[0068] In one embodiment, if Figure 6As shown, step S65, i.e., correcting the consistency rate according to the second database operation log, includes the following steps: S651, parsing the second database operation log to obtain all second data operation records in the business database on the synchronization day and during the synchronization period; S652. Calculate a second data offset of the business database and the data lake according to the second data operation record; S653, using the second data offset as a second corrected difference; S654: Correct the consistency rate according to the second corrected difference.

[0069] In some embodiments, the step of parsing the second data operation log to obtain the second data offset of the data lake can refer to the description of steps S631 to S634, the only difference being that the parsed database operation logs are different, that is, the first database operation log includes the operation log during the synchronization period, while the second database operation log includes the operation log on the synchronization day. Then, based on the parsed data operation records, the second data offset between the business database and the data lake is calculated, and the calculated second data offset is used as the second corrected difference. For example, assuming that the second data offset of the business database is 30, and the data lake does not need to update these 30 records during the synchronization period, then these 30 records will be regarded as the second corrected difference. Finally, based on the second corrected difference, the previously calculated consistency rate is corrected.

[0070] Specifically, the following formula can be used to correct the consistency rate under the second synchronization strategy: The second corrected difference represents the data offset of the business database synchronization day.

[0071] For example, assume that the second number of data items in the business database value is 4800, and the first aggregate value is 4,000,000 yuan; the second number of data items in the data lake value is 5,000, and the second aggregate value is 5,000,000 yuan; the second number of data items difference in the second correction difference is 200, and the second aggregate value difference is 1,000,000 yuan. The above formula can be used to calculate the consistency rate of data item correction: The aggregate value correction consistency rate is: It should be noted that the above is only an example and the present invention is not limited thereto.

[0072] The above process can produce a more accurate consistency rate indicator, which not only provides an important reference for subsequent data governance, but also ensures the reliability of data when enterprises use data lakes for decision-making.

[0073] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0074] In one embodiment, a data consistency rate correction device based on a database operation log is provided, and the data consistency rate correction device based on a database operation log corresponds one-to-one to the data consistency rate correction method of the database operation log in the above embodiment. Figure 7 As shown, the data consistency rate correction device of the database operation log includes a creation module 101, a first search module 102, a second search module 103, a first calculation module 104, a judgment module 105 and a correction module 106. The detailed description of each functional module is as follows: Establishing module 101, used to establish a data calculation list table, the data calculation list table includes the table name of each data table in the data lake and the business database, the data lake is associated with one or more of the business databases; A first retrieval module 102 is used to retrieve a business database value of a data table corresponding to the table name from a business database result table of the business database, wherein the business database value is a calculated value of the data table corresponding to the table name in the business database; A second retrieval module 103 is used to retrieve a data lake value of a data table corresponding to the table name from a data lake result table of the data lake, wherein the data lake value is a calculated value of the data table corresponding to the table name in the data lake; A first calculation module 104, configured to calculate the consistency rate between the data lake and the corresponding data table in the business database according to the business database value and the data lake value; A determination module 105, configured to determine whether the consistency rate is within the preset range; The correction module 106 is configured to correct the consistency rate according to the database operation log if the consistency rate is not within a preset range.

[0075] In one embodiment, the data calculation list table also includes monitoring information of each data table in the data lake and the business database, the monitoring information includes key fields, delimited fields and calculation ranges of delimited fields, the delimited fields are pre-selected calculation time fields; the data consistency rate correction device of the database operation log also includes: A second calculation module, used to calculate the number of data items in the corresponding data table in the business database according to the table name, the delimited field and the calculation range, to obtain a first number of data items; A third calculation module calculates the aggregate value of the key field in the corresponding data table in the business database according to the table name, the delimited field and the calculation range to obtain a first aggregate value; The first writing module is used to write the first number of data items and the first aggregation value as the business database value into the business database result table.

[0076] In one embodiment, the data consistency rate correction device of the database operation log further includes: a fourth calculation module, configured to calculate the number of data items in the corresponding data table in the data lake according to the table name, the delimited field, and the calculation range, to obtain a second number of data items; a fifth calculation module, calculating, according to the table name, the delimited field, and the calculation scope, an aggregate value of a key field in a corresponding data table in the data lake to obtain a second aggregate value; The second writing module is used to write the second number of data items and the second aggregation value into the data lake result table as the data lake value.

[0077] In one embodiment, the correction module 106 is specifically configured to: Obtaining a preset synchronization strategy, the synchronization strategy including a first synchronization strategy and a second synchronization strategy, the first synchronization strategy is that when the data lake synchronizes data from the business database, synchronization day data is included, and the second synchronization strategy is that when the data lake synchronizes data from the business database, synchronization day data is not included; If the synchronization strategy is the first synchronization strategy, obtaining a first database operation log during synchronization of the business database; Correcting the consistency rate according to the first database operation log; If the synchronization strategy is the second synchronization strategy, obtaining the second database operation log of the business database synchronization day; The consistency rate is corrected according to the second database operation log.

[0078] In one embodiment, the correction module 106 is further configured to: Parsing the first database operation log to obtain all first data operation records in the business database during the synchronization period; Calculating a first data offset of the business database and the data lake according to the first data operation record; Using the first data offset as a first corrected difference; The consistency rate is corrected according to the first corrected difference amount.

[0079] In one embodiment, the correction module 106 is further configured to: Parsing the second database operation log to obtain all second data operation records in the business database on the synchronization day; Calculating a second data offset of the business database and the data lake according to the second data operation record; Using the second data offset as a second corrected difference; The consistency rate is corrected according to the second corrected difference amount.

[0080] In one embodiment, the data operation record includes insertion, update and deletion of data.

[0081] The specific definition of the data consistency rate correction device for the database operation log can be found in the definition of the data consistency rate correction method for the database operation log in the above text, which will not be repeated here. Each module in the above-mentioned data consistency rate correction device for the database operation log can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0082] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to execute the data required for the data consistency rate correction method based on the database operation log. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data consistency rate correction method for a database operation log is implemented.

[0083] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: Establishing a data calculation list table, wherein the data calculation list table includes table names of various data tables in the data lake and the business database, wherein the data lake is associated with one or more of the business databases; Retrieving a business database value of a data table corresponding to the table name from a business database result table of the business database, wherein the business database value is a calculated value of the data table corresponding to the table name in the business database; Retrieving a data lake value of a data table corresponding to the table name from a data lake result table of the data lake, wherein the data lake value is a calculated value of the data table corresponding to the table name in the data lake; Calculating the consistency rate of the corresponding data tables in the data lake and the business database according to the business database value and the data lake value; Determining whether the consistency rate is within a preset range; If the consistency rate is not within the preset range, the consistency rate is corrected according to the database operation log.

[0084] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: Establishing a data calculation list table, wherein the data calculation list table includes table names of various data tables in the data lake and the business database, wherein the data lake is associated with one or more of the business databases; Retrieving a business database value of a data table corresponding to the table name from a business database result table of the business database, wherein the business database value is a calculated value of the data table corresponding to the table name in the business database; Retrieving a data lake value of a data table corresponding to the table name from a data lake result table of the data lake, wherein the data lake value is a calculated value of the data table corresponding to the table name in the data lake; Calculating the consistency rate of the corresponding data tables in the data lake and the business database according to the business database value and the data lake value; Determining whether the consistency rate is within a preset range; If the consistency rate is not within the preset range, the consistency rate is corrected according to the database operation log.

[0085] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0086] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0087] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for correcting data consistency rate based on database operation log, characterized in that: include: Establishing a data calculation list table, wherein the data calculation list table includes table names of various data tables in the data lake and the business database, wherein the data lake is associated with one or more of the business databases; Retrieving a business database value of a data table corresponding to the table name from a business database result table of the business database, wherein the business database value is a calculated value of the data table corresponding to the table name in the business database; Retrieving a data lake value of a data table corresponding to the table name from a data lake result table of the data lake, wherein the data lake value is a calculated value of the data table corresponding to the table name in the data lake; Calculating the consistency rate of the corresponding data tables in the data lake and the business database according to the business database value and the data lake value; Determining whether the consistency rate is within a preset range; If the consistency rate is not within the preset range, the consistency rate is corrected according to the database operation log.

2. The method for correcting data consistency rate based on database operation log according to claim 1, characterized in that: The data calculation list table also includes monitoring information of each data table in the data lake and the business database, and the monitoring information includes key fields, delimited fields and calculation ranges of delimited fields, and the delimited fields are pre-selected calculation time fields; After the data calculation list table is established, it includes: Calculate the number of data items in the corresponding data table in the business database according to the table name, the delimited field, and the calculation range to obtain a first number of data items; Calculate the aggregate value of the key field in the corresponding data table in the business database according to the table name, the delimited field and the calculation range to obtain a first aggregate value; The first number of data items and the first aggregation value are used as the business database values ​​and written into the business database result table.

3. The method for correcting data consistency rate based on database operation log according to claim 2, characterized in that: After the data calculation list table is established, the following steps are also included: Calculate the number of data items in the corresponding data table in the data lake according to the table name, the delimited field, and the calculation range to obtain a second number of data items; Calculate the aggregate value of the key field in the corresponding data table in the data lake according to the table name, the delimited field, and the calculation scope to obtain a second aggregate value; The second number of data items and the second aggregate value are used as the data lake value and written into the data lake result table.

4. The method for correcting data consistency rate based on database operation log according to claim 1, characterized in that: If the consistency rate is not within the preset range, the consistency rate is corrected according to the database operation log, including: Obtain a preset synchronization strategy, the synchronization strategy including a first synchronization strategy and a second synchronization strategy, the first synchronization strategy is that when the data lake synchronizes data from the business database, the synchronization date and synchronization period data are included, and the second synchronization strategy is that when the data lake synchronizes data from the business database, the synchronization date and synchronization period data are not included; If the synchronization strategy is the first synchronization strategy, obtaining a first database operation log during synchronization of the business database; Correcting the consistency rate according to the first database operation log; If the synchronization strategy is the second synchronization strategy, obtaining the second database operation log of the business database synchronization day; The consistency rate is corrected according to the second database operation log.

5. The method for correcting data consistency rate based on database operation log according to claim 4, characterized in that: The correcting the consistency rate according to the first database operation log includes: Parsing the first database operation log to obtain all first data operation records in the business database during the synchronization period; Calculating a first data offset of the business database and the data lake according to the first data operation record; Using the first data offset as a first corrected difference; The consistency rate is corrected according to the first corrected difference amount.

6. The method for correcting data consistency rate based on database operation log according to claim 4, characterized in that: The correcting the consistency rate according to the second database operation log includes: Parsing the second database operation log to obtain all second data operation records in the business database on the synchronization day; Calculating a second data offset of the business database and the data lake according to the second data operation record; Using the second data offset as a second corrected difference; The consistency rate is corrected according to the second corrected difference amount.

7. The method for correcting data consistency rate based on database operation log according to claim 5 or 6, characterized in that: The data operation record includes the insertion, update and deletion of data.

8. A data consistency rate correction device based on database operation log, characterized in that: include: An establishment module, used to establish a data calculation list table, wherein the data calculation list table includes table names of various data tables in the data lake and the business database, and the data lake is associated with one or more of the business databases; A first retrieval module, configured to retrieve a business database value of a data table corresponding to the table name from a business database result table of the business database, wherein the business database value is a calculated value of the data table corresponding to the table name in the business database; A second retrieval module is used to retrieve a data lake value of a data table corresponding to the table name from a data lake result table of the data lake, wherein the data lake value is a calculated value of the data table corresponding to the table name in the data lake; A first calculation module, configured to calculate the consistency rate of the data lake and the corresponding data table in the business database according to the business database value and the data lake value; A judgment module, used to judge whether the consistency rate is within a preset range; The correction module is used to correct the consistency rate according to the database operation log if the consistency rate is not within the preset range.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the data consistency rate correction method based on the database operation log as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the data consistency rate correction method based on the database operation log as described in any one of claims 1 to 7 are implemented.

Images