Data processing method and device, computer, storage medium and program product

Abstract

The invention discloses a data processing method and device, a computer, a storage medium and a program product. The data processing method comprises the steps of obtaining first flow table data at a first moment and statistical summary data corresponding to the first flow table data; determining a first result and a second result according to the first flow table data and the statistical summary data; if it is determined that the first flow table data is incomplete according to the first result and the second result, determining lost data according to second flow table data at the current moment in the data bin and flow table deletion data corresponding to the second flow table data; and obtaining complete flow table data according to the lost data and the first flow table data. By acquiring the statistical summary data corresponding to the flow table and then determining whether the flow table data is missing or not according to the first result and the second result obtained according to the quantitative indexes in the statistical summary data, whether the flow table data is missing or not can be quickly and accurately determined, and the judgment efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a data processing method, device, computer, storage medium and program product. Background Art

[0002] As an important data table in the financial field, transaction flow tables are widely used in different business scenarios. In actual use, offline tasks such as financial accounting, regulatory reporting, and risk control applications usually require full flow records to support data processing. Therefore, it is crucial to ensure the integrity of the flow table data. In the actual production environment, the two links of extracting flow table data from the business system to the data warehouse and storing it for a long time or permanently may lead to incomplete data and abnormal loss. Summary of the invention

[0003] The embodiments of the present application provide a data processing method, device, computer, storage medium and program product, which do not require one-to-one comparison of the water flow meter data at the first moment with the water flow meter data at the latest moment to determine whether the water flow meter data is missing, which is conducive to quickly and accurately determining whether the water flow meter data is missing and improving the determination efficiency.

[0004] In a first aspect, an embodiment of the present application provides a data processing method, including:

[0005] Obtain the first water flow meter data at the first moment in the data warehouse and the statistical summary data corresponding to the first water flow meter data, the statistical summary data including amount summary data, record quantity summary data, deleted amount summary data and deleted record quantity summary data; determine a first result according to the first water flow meter data, the record quantity summary data and the deleted record quantity summary data, the first result is used to determine the data quantity integrity of the first water flow meter data; determine a second result according to the first water flow meter data, the amount summary data and the deleted amount summary data, the second result is used to determine the amount data integrity of the first water flow meter data; if it is determined that the first water flow meter data is incomplete according to the first result and the second result, determine the missing data according to the second water flow meter data at the current moment in the data warehouse and the water flow table deletion data corresponding to the second water flow meter data; obtain the third water flow meter data according to the missing data and the first water flow meter data, the third water flow meter data is the complete water flow meter data at the first moment.

[0006] In a second aspect, an embodiment of the present application provides a data processing device, including:

[0007] an acquisition module, used to acquire first water flow meter data at a first moment in a data warehouse and statistical summary data corresponding to the first water flow meter data, wherein the statistical summary data includes amount summary data, record quantity summary data, deleted amount summary data, and deleted record quantity summary data;

[0008] A processing module, used to determine a first result based on the first water flow meter data, record quantity summary data and the deleted record quantity summary data, the first result being used to determine the data quantity integrity of the first water flow meter data; and used to determine a second result based on the first water flow meter data, amount summary data and the deleted amount summary data, the second result being used to determine the amount data integrity of the first water flow meter data; and used to determine the missing data based on the second water flow meter data at the current moment in the data warehouse and the water flow meter deletion data corresponding to the second water flow meter data if it is determined that the first water flow meter data is incomplete based on the first result and the second result; and used to obtain third water flow meter data based on the missing data and the first water flow meter data, the third water flow meter data being the complete water flow meter data at the first moment.

[0009] In a third aspect, an embodiment of the present application provides a data processing device, the device comprising:

[0010] A memory, a processor, and a data processing program stored in the memory and executable on the processor, wherein the data processing program is configured to implement part or all of the steps described in any method in the first aspect.

[0011] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a data processing program is stored. When the data processing program is executed by a processor, some or all of the steps described in any method in the first aspect are implemented.

[0012] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in any method of the first aspect of the embodiment of the present application. The computer program product can be a software installation package.

[0013] By implementing the embodiments of the present application, firstly, the first water flow meter data at the first moment in the data warehouse and the statistical summary data corresponding to the first water flow meter data are obtained, the statistical summary data including amount summary data, record quantity summary data, deleted amount summary data and deleted record quantity summary data; a first result is determined according to the first water flow meter data, the record quantity summary data and the deleted record quantity summary data, and the first result is used to determine the data quantity integrity of the first water flow meter data; a second result is determined according to the first water flow meter data, the amount summary data and the deleted amount summary data, and the second result is used to determine the amount data integrity of the first water flow meter data; if it is determined that the first water flow meter data is incomplete according to the first result and the second result, the missing data is determined according to the second water flow meter data at the current moment in the data warehouse and the water flow meter deletion data corresponding to the second water flow meter data; the third water flow meter data is obtained according to the missing data and the first water flow meter data, and the third water flow meter data is the complete water flow meter data at the first moment. By acquiring statistical summary data corresponding to the water meter at the first moment when the integrity check of the water meter data is required, and then obtaining a first result for determining the data quantity integrity of the water meter data and a second result for determining the amount data integrity of the water meter data according to various quantitative indicators in the statistical summary data, finally directly combining the first result and the second result to determine whether the water meter data is missing, there is no need to compare the water meter data at the first moment with the water meter data at the latest moment one by one to determine whether the water meter data is missing, which is conducive to quickly and accurately determining whether the water meter data is missing and improving the determination efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0015] Figure 1 It is a schematic diagram of the architecture of a data processing system provided in an embodiment of the present application;

[0016] Figure 2 is a flow chart of a data processing method provided in an embodiment of the present application;

[0017] Figure 3 It is a table schematic diagram of water flow meter data provided by an embodiment of the present application;

[0018] Figure 4 It is a schematic diagram of a data verification process provided by an embodiment of the present application;

[0019] Figure 5 is a flow chart of a method for data supplementation provided in an embodiment of the present application;

[0020] Figure 6This is a data schematic diagram of an auxiliary statistical module provided in an embodiment of the present application;

[0021] Figure 7 is a schematic diagram of first water flow meter data provided by an embodiment of the present application;

[0022] Figure 8 is a schematic diagram of a second water flow meter data provided by an embodiment of the present application;

[0023] Fig. 9 This is a schematic diagram of deleting data from a flow table provided in an embodiment of the present application;

[0024] Fig.10 is a schematic diagram of missing water flow meter data provided by an embodiment of the present application;

[0025] Fig.11 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0026] Fig.12 It is a structural diagram of a computer provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. According to the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0028] The terms "first", "second", "third", etc. in the specification, claims and drawings of this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0029] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0030] As an important data table in the financial field, transaction flow tables are widely used in different business scenarios. In actual use, offline tasks such as financial accounting, regulatory reporting and risk control applications usually require full flow records to support data processing. Therefore, it is crucial to ensure the integrity of the flow table data.

[0031] In an actual production environment, both the extraction of flow meter data from the business system to the data warehouse and long-term or permanent storage may lead to incomplete data and abnormal data loss.

[0032] Specifically, the existing solutions for determining whether abnormal loss of water meter data occurs in an actual production environment mainly include the following three methods:

[0033] Method 1: Checksum-based method: Generate a checksum for each record or a group of records and store it in the database. When data integrity needs to be checked, recalculate the checksum of the current data and compare it with the previously saved value to determine whether there is any missing or damaged data.

[0034] Method 2: Partition audit method: partition the entire flow table according to time, type and other dimensions, and perform audit operations on each partition regularly. Detect abnormal loss by comparing the data volume, amount and other key indicators between different time periods.

[0035] Method three, log audit method: use system logs (such as database operation logs or application logs) to track data changes in flow tables, and determine whether there are abnormal deletion or modification operations by analyzing log information, thereby inferring potential data loss.

[0036] It can be seen that the above-mentioned common data processing methods can determine whether abnormal loss of water meter data occurs in an actual production environment. However, when the scale of water meter data is large, a large amount of data needs to be traversed, which slows down the overall process and reduces the determination efficiency.

[0037] In response to the above problems, an embodiment of the present application provides a data processing method, device, computer, storage medium and program product, first obtaining the first water flow meter data at the first moment in the data warehouse and the statistical summary data corresponding to the first water flow meter data, the statistical summary data including amount summary data, record quantity summary data, deleted amount summary data and deleted record quantity summary data; determining a first result based on the first water flow meter data, the record quantity summary data and the deleted record quantity summary data, the first result is used to determine the data quantity integrity of the first water flow meter data; determining a second result based on the first water flow meter data, the amount summary data and the deleted amount summary data, the second result is used to determine the amount data integrity of the first water flow meter data; if it is determined that the first water flow meter data is incomplete based on the first result and the second result, determining the missing data based on the second water flow meter data at the current moment in the data warehouse and the water flow table deletion data corresponding to the second water flow meter data; obtaining the third water flow meter data based on the missing data and the first water flow meter data, the third water flow meter data is the complete water flow meter data at the first moment. By acquiring statistical summary data corresponding to the water meter at the first moment when the integrity check of the water meter data is required, and then obtaining a first result for determining the data quantity integrity of the water meter data and a second result for determining the amount data integrity of the water meter data according to various quantitative indicators in the statistical summary data, finally directly combining the first result and the second result to determine whether the water meter data is missing, there is no need to compare the water meter data at the first moment with the water meter data at the latest moment one by one to determine whether the water meter data is missing, which is conducive to quickly and accurately determining whether the water meter data is missing and improving the determination efficiency.

[0038] The data processing method, device, computer, storage medium and program product provided in the embodiments of the present application can be applied to Figure 1 In the data processing system shown, see Figure 1 , Figure 1 1 is a schematic diagram of the architecture of a data processing system 100 provided in an embodiment of the present application. The data processing system 100 includes a client 101 and a server 102. The client 101 can communicate with the server 102 through a network. The client 101 refers to a device used by a user, such as a smart phone, a computer, etc. The user can interact with the system server 102 through the client 101. In this solution, the client 101 is mainly responsible for sending the flow table file and other data required for data processing to the server 102, and sending requests to the server 102, including operations such as importing a new flow table into a database, requesting to obtain data under specific conditions, and obtaining the results of integrity checks.

[0039] The server 102 refers to a remote computer used to process a large number of computing tasks and store data, and is responsible for data import, data storage, and data processing in the data processing system 100. In this solution, the server 102 receives the flow table file from the client and performs a series of operations on the flow table, including performing extraction, conversion, loading, and integrity check operations.

[0040] Based on this, the present application provides a data processing method, device, computer, storage medium and program product, and the present application is described in detail below in conjunction with the accompanying drawings.

[0041] See also Figure 2 , Figure 2 is a flow chart of a data processing method provided in an embodiment of the present application, such as Figure 2 As shown, the method comprises the following steps:

[0042] S201, obtaining first water flow meter data at a first moment in a data warehouse and statistical summary data corresponding to the first water flow meter data.

[0043] Among them, the first moment can be the current moment or a certain moment in history, and the statistical summary data includes amount summary data, record quantity summary data, deleted amount summary data and deleted record quantity summary data. For example, when the first moment is 20200131, the first water flow meter data of 20200131 in the data warehouse and the statistical summary data of 20200131 corresponding to the first water flow meter data are obtained.

[0044] Among them, the server of the data processing system is provided with an auxiliary statistical module and a deletion record module, wherein the auxiliary statistical module includes an amount accumulator, a record counter, a deletion accumulator, a deletion counter and an update time recorder. Specifically, the amount accumulator, the record counter, the deletion accumulator and the deletion counter are essentially accumulators. The amount accumulator is used to accumulate the amount information of all transaction records, and adopts an incremental accumulation method, that is, for each additional transaction record, the amount accumulator accumulates the incremental transaction record amount; the record counter is used to accumulate the number information of all transaction records, and also adopts an incremental accumulation method. The deletion accumulator is used to accumulate the amount information of all transaction records deleted by the system or manually, using the incremental accumulation method, that is, each time a transaction record is deleted, the deletion accumulator accumulates the amount of the deleted transaction records; the deletion counter is used to accumulate the number of transaction records deleted by the system or manually, using the incremental accumulation method, that is, each time a transaction record is deleted, the deletion counter increases the number of records by 1; the update time recorder is used to record the time information of the most recent update of the auxiliary module.

[0045] For example, see Figure 3 , Figure 3 is a table diagram of water flow meter data provided by an embodiment of the present application, such as Figure 3 As shown, there are 5 transaction records in the transaction table A of section 20240131, of which d and e are mistakenly generated transaction records. At this time, the amount accumulator in the auxiliary statistical module uses an incremental accumulation method to obtain the amount summary data of transaction table A, which is 10+15+60+50+15=190, and the record quantity summary data of transaction table A is 1+1+1+1+1=5. Subsequently, d and e mistakenly generated in transaction table A are deleted in section 20240201. At this time, the deletion accumulator in the auxiliary statistical module uses an incremental accumulation method to obtain the deletion amount summary data of transaction table A, which is 50+15=65, and the deletion record quantity summary data of transaction table A is 1+1=2.

[0046] S202: Determine a first result according to the first flow table data, the record quantity summary data, and the deleted record quantity summary data.

[0047] Before determining the first result according to the first water flow meter data, the record quantity summary data and the deleted record quantity summary data, a preprocessing operation may be performed on the first water flow meter data, the record quantity summary data and the deleted record quantity summary data. The preprocessing operation includes data cleaning, data conversion, time series processing, feature engineering, data integration and format consistency check, etc. Specifically, data cleaning includes deduplication, missing value processing and outlier detection, data conversion includes type conversion and normalization, time series processing includes time formatting and frequency aggregation, feature engineering includes creating derivative variables and grouping aggregation, and data integration includes connection operations.

[0048] Exemplarily, after obtaining the first water flow meter data, the server of the data processing system can use the interquartile range (IQR) method to detect and process the outliers of the first water flow meter data, and then convert the date string in the data water flow table into a date type, convert the amount field into a floating point number, and finally detect whether the naming rules and formats of the same attributes in the first water flow table are consistent, for example, whether the units of the amount data are consistent.

[0049] In a possible implementation, determining a first result according to the first water flow meter data, the record quantity summary data, and the deleted record quantity summary data includes:

[0050] Obtain a first difference between the record quantity summary data and the deleted record quantity summary data; when the first difference is equal to the total number of water flow meter records in the first water flow meter data, determine that the first result is that the data quantity of the first water flow meter data is complete; when the first difference is not equal to the total number of water flow meter records in the first water flow meter data, determine that the first result is that the data quantity of the first water flow meter data is incomplete.

[0051] Among them, see Figure 4 , Figure 4 is a schematic diagram of a data verification process provided by an embodiment of the present application, such as Figure 4 As shown, the server of the data processing system first extracts the water flow meter data and the auxiliary statistical module data and the deletion record module data corresponding to the water flow meter data into the data warehouse, wherein the extraction of the water flow meter data uses the current date as the time information and includes the historical extracted time information. The data of the auxiliary statistical module is similar to the water flow meter, and the historical data of the auxiliary statistical module is retained. For the daily extracted flow table data, first perform data integrity check. Specifically, count the total number of flow table records in the flow table extracted into the data warehouse on that day, record as A1, count the record number summary data of the record counter in the auxiliary statistical module, record as B2, count the number of deleted records of the deletion counter in the auxiliary statistical module, record as B4, and judge whether A1 is equal to the difference between B2 and B4. The essence of this step is to judge whether the number of records in the flow table extracted from the latest data warehouse and the difference between the auxiliary module record counter and the deletion counter recorded by the business system are equal. If A1 is equal to the difference between B2 and B4, it proves that the number of flow table data is complete in the process of extracting the flow table data to the data warehouse.

[0052] Exemplarily, the server of the data processing system extracts the flow table B of 20240131 into the data warehouse, and the server performs a data integrity check on the extracted flow table data, and obtains that the total number of flow table records in the flow table extracted into the data warehouse on that day is 10, the record quantity summary data of the record counter in the auxiliary statistical module is 15, and the deletion record quantity summary data of the deletion counter in the auxiliary statistical module is 5. At this time, 10=15-5, so the first result is that the data quantity of flow table B is complete.

[0053] It can be seen that in this example, by comparing the total number of water flow meter records in the water flow table, the difference between the obtained record quantity summary data and the deleted record quantity summary data, it is possible to accurately determine whether the data of the first water flow meter is complete, and realize automatic detection of the integrity of the water flow meter data, which is conducive to improving the efficiency of determining whether the water flow meter data is abnormal, and facilitates subsequent timely measures to repair or supplement missing data.

[0054] S203: Determine a second result according to the first flow table data, the amount summary data and the deleted amount summary data.

[0055] Before determining the second result according to the first water flow meter data, the amount summary data and the deleted amount summary data, the first water flow meter data, the amount summary data and the deleted amount summary data can be preprocessed, and the preprocessing operation includes data cleaning, data conversion, time series processing, feature engineering, data integration and format consistency check, etc. Specifically, data cleaning includes deduplication, missing value processing and outlier detection, data conversion includes type conversion and normalization, time series processing includes time formatting and frequency aggregation, feature engineering includes creating derivative variables and grouping aggregation, and data integration includes connection operations.

[0056] In a possible example, determining the second result according to the first flow table data, the amount summary data, and the deleted amount summary data includes:

[0057] Obtain a second difference between the amount summary data and the deleted amount summary data; when the first difference is equal to the total amount in the first water flow table data, determine that the second result is that the amount data of the first water flow table data is complete; when the first difference is not equal to the total amount in the first water flow table data, determine that the second result is that the amount data of the first water flow table data is incomplete.

[0058] Please refer again to Figure 4 ,like Figure 4 As shown, after the data integrity check is performed on the flow table data extracted every day, the consistency check of the amount of the flow table data is performed at the same time. Specifically, the total amount of the flow table in the flow table extracted into the data warehouse on that day is counted and recorded as A2, the amount summary data of the amount accumulator in the auxiliary statistical module is counted and recorded as B1, and the deletion amount summary data of the deletion accumulator in the auxiliary statistical module is counted and recorded as B3. It is judged whether A2 is equal to the difference between B1 and B3. The essence of this step is to judge whether the total amount of the flow extracted from the latest data warehouse and the difference between the auxiliary module amount accumulator and the deletion accumulator recorded by the business system are equal. If A2 is equal to the difference between B1 and B3, it proves that the amount data in the flow table data is consistent in the process of extracting the flow table data to the data warehouse.

[0059] Exemplarily, the server of the data processing system extracts the transaction table C of 20240201 into the data warehouse, and the server performs an amount consistency check on the extracted transaction table data, and obtains the total transaction table amount in the transaction table extracted into the data warehouse on that day as 1050, the amount summary data of the amount accumulator in the statistical auxiliary statistical module is 1200, and the deletion amount summary data of the deletion accumulator in the statistical auxiliary statistical module is 50. At this time, 1050 is not equal to 1200-50, so the first result is that the amount data of transaction table B during the extraction process is inconsistent.

[0060] It can be seen that in this example, by comparing the difference between the total amount in the flow table, the obtained amount summary data and the deleted amount summary data, it is possible to accurately determine whether the amount of the first flow table remains consistent during the extraction process, and realize automatic detection of the amount consistency of the flow table data, which is conducive to improving the efficiency of determining whether the flow table data is abnormal, and facilitates subsequent timely measures to repair or supplement missing data.

[0061] Among them, the data verification of the flow table data can also include time dimension verification, user dimension verification and integrity constraint verification. Specifically, the time dimension check verifies the date and time fields in the flow table to ensure that there are no logical errors, such as the end time is earlier than the start time; the user dimension check verifies the user ID of each record to confirm that the user exists in the system and is in normal status; the integrity constraint check ensures that all necessary fields have been filled in and conform to the set data format, such as email address format, mobile phone number format, etc.

[0062] Exemplarily, the transaction flow table includes a transaction ID field, a user ID field, a transaction amount field, and a transaction time field. When using the time dimension in the multi-dimensional test to check whether the data in the transaction flow table is missing, check whether the transaction time field is within a reasonable range; when using the user dimension in the multi-dimensional test to check whether the data in the transaction flow table is missing, check whether the user ID of each record exists in the user table; when using the integrity constraint to check whether the data in the transaction flow table is missing, determine whether the transaction ID field, user ID field, transaction amount field, and transaction time field contain empty fields.

[0063] S204: If it is determined that the first water flow table data is incomplete according to the first result and the second result, the missing data is determined according to the second water flow table data at the current moment in the data warehouse and the water flow table deletion data corresponding to the second water flow table data.

[0064] Specifically, in a possible implementation manner, determining that the first water flow meter data is incomplete according to the first result and the second result includes:

[0065] When the first result is that the data quantity of the first water flow meter data is incomplete, and / or the second result is that the amount data of the first water flow meter data is incomplete, it is determined that the first water flow meter data is incomplete.

[0066] Among them, after determining that the data of the first water flow meter is incomplete, the water flow meter extraction operation can be re-performed, and data integrity verification and amount consistency verification can be performed after the extraction operation until the first result and the second result are both that the first water flow meter data is complete.

[0067] In a possible implementation, determining lost data according to the second water flow table data at the current moment in the data warehouse and the water flow table deletion data corresponding to the second water flow table data includes:

[0068] Delete the data whose transaction time is after the first moment in the second water meter data at the current moment in the data warehouse to obtain a first data set; delete the data whose transaction time is after the first moment in the water meter deletion data to obtain a second data set; obtain the data whose transaction time is before the first moment in the second data set to obtain a third data set; determine the lost data according to the first data set and the third data set.

[0069] The transaction time refers to the time when each transaction in the transaction table data occurs, usually expressed in the form of date and time. For example, please refer to Figure 3 , Figure 3 The transaction time of a is 20240101; the deletion transaction time refers to the time when the data is deleted in the deletion data of the transaction table. For example, the transaction time of a transaction in the transaction table is 20240131, but the transaction is deleted on 20240201. At this time, the deletion transaction time corresponding to the transaction in the deletion record of the transaction table is 20240201.

[0070] Among them, the total number of flow meter records in the first flow meter data is counted, recorded as A1, the record quantity summary data of the record counter in the auxiliary statistical module is counted, recorded as B2, and the deletion record quantity summary data of the deletion counter in the auxiliary statistical module is counted, recorded as B4. By calculating the value of B2-B4-A1, the number of missing records in the first flow meter data can be determined.

[0071] For example, assuming that the total number of water flow meter records in the first water flow meter data of section 20200131 is 60, the value of the record quantity summary data of the record counter in the auxiliary statistical module is 70, and the value of the deleted record quantity summary data of the deletion counter in the auxiliary statistical module is 5, then there should be 70-5=65 records in section 20200131, but there are actually only 60 records. Therefore, the number of missing records in the first water flow meter data of this section is 65-60=5.

[0072] The deletion data of the water flow table is obtained by the server from the deletion record module. The deletion data of the water flow table records the deletion records of the latest state and the full amount, including the deletion records before and after a specific historical section. Specifically, the deletion data of the water flow table is used to record the water flow information deleted by the system or manually. In addition to the full amount of information such as the transaction time, transaction amount, type, etc. of the original water flow record, the deletion data of the water flow table also includes the deletion transaction time of the corresponding water flow record.

[0073] Specifically, see Figure 5 , Figure 5 is a flow chart of a method for data supplementation provided in an embodiment of the present application, such as Figure 5 As shown, after obtaining the first result based on the data integrity check and the second result based on the amount consistency check, it is determined that the first water flow meter data is incomplete. At this time, the missing data is further determined based on the first data set and the third data set, and the first water flow meter data is made complete by filling the missing data back into the first water flow meter data.

[0074] Specifically, part of the missing data in the first moment's flow table needs to be restored from the latest section, and the latest section contains the new flow data generated after the first moment, so the flow data range needs to be limited first. Assuming that the full flow data of the latest section, that is, the second flow table data is P, through the limiting condition: the flow transaction time in the second flow table data is <= the first moment, the new flow data generated after the first moment is eliminated, and only the flow records generated at this moment and before are retained. The filtered data is recorded as P_part, that is, the first data set.

[0075] At the same time, it is necessary to restore the deleted data of the water flow table to a specific historical section, that is, the state of the first moment. Assuming that the latest and full amount of deleted data of the water flow table is recorded as Q, we need to restore it to the state of the first moment, and we need to limit two conditions at the same time: Condition 1, the water flow transaction time in the deleted data of the water flow table <= the first moment, and the second data set can be obtained by condition 1; Condition 2, the deletion transaction time in the deleted data of the water flow table > the first moment, that is, to ensure that neither the water flow records deleted before the first moment nor the water flow records that will occur in the future are included, and the deleted records of the second data set after being filtered by condition 2 are recorded as Q_part, that is, the third data set.

[0076] For example, the current time is 20240331, and the first time is 20240131. At this time, the data recorded by the auxiliary statistics module at the 20240131 section is obtained. Figure 6 , Figure 6 is a data schematic diagram of an auxiliary statistical module provided in an embodiment of the present application, such as Figure 6 As shown, at this time the value of the amount accumulator is 379, the value of the record counter is 9, the value of the deletion counter is 4, the value of the deletion accumulator is 188, and the value of the update time recorder is 2024-01-31.

[0077] See also Figure 7 , Figure 7 is a schematic diagram of a first water flow meter data provided by an embodiment of the present application, such as Figure 7 As shown, the complete set of the first water flow meter data is Y, the record quantity summary data of the first water flow meter is 1+1+1=3, the amount summary data is 10+15+16=41, and the difference between the record counter and the deletion counter of the auxiliary statistical module is 9-4=5, which is not equal to the record quantity summary data of the first water flow meter, indicating that the number of missing records in the first water flow meter is 5-3=2, and the amount of missing records in the first water flow meter is 379-188-41=150.

[0078] See also Figure 8 , Figure 8 is a schematic diagram of a second water flow meter data provided in an embodiment of the present application, such as Figure 8 As shown, the second water flow table data set is P, and the water flow transaction time in the second water flow table data is limited to <= 20240131, and P_part, that is, the first data set, is obtained; please refer to Fig. 9 , Fig. 9 Schematic diagram of deleting data from a flow table provided in an embodiment of the present application. Fig. 9 As shown, the complete set of deleted data in the water flow table is Q. At the same time, the water flow transaction time in the deleted data of the water flow table is restricted to be <= 20240131, and the deletion operation time is > 20240131, and Q_part, that is, the third data set, is obtained; the complete set U is obtained by obtaining the union of P_part and Q_part, and then the complete set U is subtracted from the first water flow table data Y to obtain the missing records R of the first water flow table, please refer to Fig.10 , Fig.10 is a schematic diagram of missing water flow meter data provided by an embodiment of the present application, such as Fig.10 As shown, the number of R at this time is 1+1=2, corresponding to the number of records missing in the first water flow meter, and the record amount of R is 100+50=150, corresponding to the record amount missing in the first water flow meter. The complete version data is obtained by filling R back into the first water flow meter data Y.

[0079] In a possible implementation manner, determining the missing data according to the first data set and the third data set includes:

[0080] Determine the union of the first data set and the third data set to obtain a fourth data set; obtain the difference between the fourth data set and the first water flow meter data to obtain a fifth data set, where the fifth data set is the missing data.

[0081] Among them, the union of the first data set and the third data set is the complete version data set of the first water flow table at the first moment, which is recorded as set U. The difference set of the full set U and the details of the first water flow table at the first moment can be obtained to obtain the number of missing records, which is recorded as set R. The difference set R, that is, the missing data of the first water flow table at the first moment, is added back to the first water flow table to obtain the complete data set at the first moment.

[0082] For example, assuming that the latest time is 20240331, we are standing at the time point of 20240331. To use the data of the historical section 20240131 for application processing, we must first verify the section data. X is the complete version data, which is the target data that meets the final expectation of verification; Y is the current situation, that is, the missing version data; Z is the statistical value recorded by the auxiliary statistical module in the 20240131 section. First, the number of records in the statistical set Y is 3, and the cumulative value of the transaction amount is: 10+15+16=41. The difference between the counter value of the auxiliary statistical module Z and the value of the deletion counter is 9-4=5, which is not equal to the statistical result of the Y set 3, indicating that the number of missing records = 5-3=2; at the same time, the amount summary data of the amount accumulator in the auxiliary statistical module Z is 379, and the deletion amount summary data of the deletion accumulator in the auxiliary statistical module Z is 188, indicating that the amount of missing records = 379-188-41=150. At this time, the deletion recorder Q is restricted to transaction time <= 20240131 and deletion time > 20240131, and the Q_part data, i.e., the third data set, is obtained. Then, for the latest time flow table data P, the transaction time is restricted to <= 20240131, and the P_part data, i.e., the first data set, is obtained. The union of Q_part and P_part is obtained to obtain the full set U. The full set U is subtracted from the missing version data Y to obtain the missing record R.

[0083] It can be seen that in this example, after obtaining the full set at the first moment, the set is not used to overwrite the first flow table data. Because there are more than one billion flow records and the missing data is relatively small, supplementing the missing data based on the original records saves more system resources than directly overwriting the original records.

[0084] S205: Obtain third water flow meter data according to the lost data and the first water flow meter data.

[0085] It can be seen that in this example, by obtaining the statistical summary data corresponding to the water meter at the first moment when the integrity check of the water meter data is required, and then obtaining the first result for determining the data quantity integrity of the water meter data and the second result for determining the amount data integrity of the water meter data according to the various quantitative indicators in the statistical summary data, finally, directly combining the first result and the second result to determine whether the water meter data is missing, there is no need to compare the water meter data at the first moment with the water meter data at the latest moment one by one to determine whether the water meter data is missing, which is conducive to quickly and accurately determining whether the water meter data is missing and improving the determination efficiency.

[0086] In a possible implementation manner, after obtaining the third water flow meter data according to the lost data and the first water flow meter data, the method further includes:

[0087] A third result is determined based on the third water flow meter data, the record quantity summary data and the deleted record quantity summary data; a fourth result is determined based on the third water flow meter data, the amount summary data and the deleted amount summary data; when the third result is that the data quantity of the third water flow meter data is incomplete, and / or the fourth result is that the amount data of the third water flow meter data is incomplete, it is determined that the third water flow meter data is incomplete, and an abnormal prompt information is sent, and the abnormal prompt information is used to prompt the user that the second water flow meter data is abnormal, and / or the water flow meter deleted data is abnormal.

[0088] Among them, after repeating the data verification operation to ensure that the supplemented data is complete in magnitude and consistent in amount with the original complete version, if the repaired data statistical results are still inconsistent with the data in the auxiliary statistical module at the first moment, it can be concluded that the flow table data at the latest moment is abnormal or the data in the deletion record module is abnormal. At this time, an abnormal prompt message is sent to prompt the user to cooperate with the upstream business system to verify the problem and restore it.

[0089] It can be seen that in this example, data verification is performed again after the water meter data is supplemented. If the data verification fails, an abnormal prompt message is sent in time to remind the user to deal with the abnormal data. This is conducive to establishing a sound data supervision mechanism, improving data reliability, and facilitating subsequent analysis based on water meter data.

[0090] See also Fig.11 , Fig.11 is a structural diagram of a data processing device provided in an embodiment of the present application, such as Fig.11 As shown, the data processing device 1100 includes:

[0091] An acquisition module 1101 is used to acquire first water flow meter data at a first moment in a data warehouse and statistical summary data corresponding to the first water flow meter data, wherein the statistical summary data includes amount summary data, record quantity summary data, deleted amount summary data, and deleted record quantity summary data;

[0092] Processing module 1102 is used to determine a first result based on the first water flow meter data, the record quantity summary data and the deleted record quantity summary data, the first result is used to determine the data quantity integrity of the first water flow meter data; and to determine a second result based on the first water flow meter data, the amount summary data and the deleted amount summary data, the second result is used to determine the amount data integrity of the first water flow meter data; and to determine the missing data based on the second water flow meter data at the current moment in the data warehouse and the water flow table deletion data corresponding to the second water flow meter data if it is determined that the first water flow meter data is incomplete based on the first result and the second result; and to obtain third water flow meter data based on the missing data and the first water flow meter data, the third water flow meter data being the complete water flow meter data at the first moment.

[0093] In one possible implementation, in determining the first result based on the first water flow meter data, the record quantity summary data and the deleted record quantity summary data, the processing module 1102 is specifically used to: obtain a first difference between the record quantity summary data and the deleted record quantity summary data; when the first difference is equal to the total number of water flow meter records in the first water flow meter data, determine that the first result is that the data quantity of the first water flow meter data is complete; when the first difference is not equal to the total number of water flow meter records in the first water flow meter data, determine that the first result is that the data quantity of the first water flow meter data is incomplete.

[0094] In one possible implementation, in determining the second result based on the first water flow meter data, the amount summary data and the deleted amount summary data, the processing module 1102 is specifically used to: obtain a second difference between the amount summary data and the deleted amount summary data; when the first difference is equal to the total amount in the first water flow meter data, determine that the second result is that the amount data of the first water flow meter data is complete; when the first difference is not equal to the total amount in the first water flow meter data, determine that the second result is that the amount data of the first water flow meter data is incomplete.

[0095] In a possible implementation, in terms of determining that the first water flow meter data is incomplete based on the first result and the second result, the processing module 1102 is specifically used to determine that the first water flow meter data is incomplete when the first result is that the data quantity of the first water flow meter data is incomplete, and / or the second result is that the amount data of the first water flow meter data is incomplete.

[0096] In a possible implementation, in terms of determining the missing data based on the second water flow table data at the current moment in the data warehouse and the water flow table deletion data corresponding to the second water flow table data, the processing module 1102 is specifically used to: delete the data in the second water flow table data at the current moment in the data warehouse whose water flow transaction time is after the first moment to obtain a first data set; delete the data in the water flow table deletion data whose water flow transaction time is after the first moment to obtain a second data set; obtain the data in the second data set whose deletion transaction time is before the first moment to obtain a third data set; and determine the missing data based on the first data set and the third data set.

[0097] In one possible implementation, in terms of determining the missing data based on the first data set and the third data set, the processing module 1102 is specifically used to: determine the union of the first data set and the third data set to obtain a fourth data set; obtain the difference between the fourth data set and the first water flow meter data to obtain a fifth data set, and the fifth data set is the missing data.

[0098] In a possible implementation, after obtaining the third water flow meter data based on the lost data and the first water flow meter data, the processing module 1102 is also used to: determine a third result based on the third water flow meter data, the record quantity summary data and the deleted record quantity summary data; determine a fourth result based on the third water flow meter data, the amount summary data and the deleted amount summary data; when the third result is that the data quantity of the third water flow meter data is incomplete, and / or the fourth result is that the amount data of the third water flow meter data is incomplete, it is determined that the third water flow meter data is incomplete, and an abnormal prompt information is sent, and the abnormal prompt information is used to prompt the user that the second water flow meter data is abnormal, and / or the water flow meter deletion data is abnormal.

[0099] It is worth noting that the specific functional implementation of the data processing device 1100 is shown in the above Figure 2The description of the data processing method shown in the figure, for example, the acquisition module 1101 is used to implement the relevant content of executing S201, and the processing module 1102 is used to implement the relevant content of executing S202-S205. Each unit or module in the data processing device 1100 can be separately or completely merged into one or more other units or modules to constitute, or one (some) of the units or modules can also be split into multiple smaller units or modules in function to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present invention. The above-mentioned units or modules are divided according to logical functions. In practical applications, the function of one unit (or module) is realized by multiple units (or modules), or the function of multiple units (or modules) is realized by one unit (or module).

[0100] According to the description of the above method embodiment and related device embodiment, please refer to Fig.12 , Fig.12 It is a structural diagram of a computer 1200 provided in an embodiment of the present application. Fig.12 The computer 1200 shown includes a processor 1201, a memory 1202, a communication interface 1203 and a bus 1204. The processor 1201, the memory 1202 and the communication interface 1203 are connected to each other through the bus 1204.

[0101] Optionally, the memory 1202 is a ROM, a static storage device, a dynamic storage device or a RAM.

[0102] The memory 1202 can store executable program codes. When the executable program codes stored in the memory 1202 are executed by the processor 1201, the processor 1201 and the communication interface 1203 are used to execute the executable program codes. Figure 2 The various steps of the data processing method of the illustrated embodiment.

[0103] The processor 1201 adopts a general-purpose CPU, a microprocessor, an application-specific integrated circuit ASIC, a GPU or one or more integrated circuits to execute relevant programs to perform the data processing method of the method embodiment of the present application.

[0104] The processor 1201 can also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the data processing method of the present application can be completed by the hardware integrated logic circuit or software instructions in the processor 1201. Optionally, the processor 1201 is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component. The processor can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor is a microprocessor or the processor is any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor are combined and executed. The optional software module is located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register or other mature storage media in the art. The storage medium is located in the memory 1202, and the processor 1201 reads the information in the memory 1202, and combines its hardware to complete the functions required to be executed by a module included in a data processing device 1100 of an embodiment of the present application, or executes the data processing method of the method embodiment of the present application.

[0105] The communication interface 1203 uses a transceiver or other related device such as, but not limited to, a transceiver.

[0106] The bus 1204 may include a path for transmitting information between various components of the computer 1200 (eg, the memory 1202 , the processor 1201 , and the communication interface 1203 ).

[0107] It should be noted that although Fig.12 The computer 1200 shown only shows a memory, a processor, and a communication interface, but in the specific implementation process, those skilled in the art should understand that the computer 1200 also includes other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the computer 1200 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the computer 1200 may also include only the devices necessary for implementing the embodiments of the present application, and does not necessarily include Fig.12 All devices shown in .

[0108] An embodiment of the present application provides a computer-readable storage medium, in which a computer program for electronic data exchange is stored. The computer program includes execution instructions, and the execution instructions are used to execute part or all of the steps of any one of the data processing methods recorded in the above-mentioned data processing method embodiments. The above-mentioned computer includes an electronic client device.

[0109] An embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, and the computer program can be operated to enable a computer to perform part or all of the steps of any data processing method recorded in the above method embodiments. The computer program product can be a software installation package.

[0110] It should be noted that, for the sake of simplicity, the embodiments of any of the aforementioned data processing methods are described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the present application.

[0111] The above is a detailed introduction to the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of a data processing method, device, computer, storage medium and program product of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for a person skilled in the art, according to the idea of ​​a data processing method, device, computer, storage medium and program product of the present application, there may be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0112] The present application is described with reference to the flowcharts and / or block diagrams of the methods, hardware products, and computer program products of the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0113] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1The memory may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0114] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "one" does not exclude multiple situations. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0115] A person skilled in the art can understand that all or part of the steps in the various methods of the method embodiments of any of the above-mentioned data processing methods can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, and the memory can include: a flash drive, a read-only memory (English: Read-Only Memory, abbreviated as: ROM), a random access memory (English: Random Access Memory, abbreviated as: RAM), a disk or an optical disk, etc.

[0116] It can be understood that any product that is controlled or configured to execute the processing method of the flowchart described in an embodiment of a data processing method of the present application, such as the device and computer program product of the above flowchart, falls within the scope of the related products described in the present application.

[0117] Obviously, those skilled in the art can make various changes and modifications to the data processing method, device, computer, storage medium and program product provided by the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire first water meter data at a first moment in the data warehouse and statistical summary data corresponding to the first water meter data, wherein the statistical summary data includes amount summary data, record quantity summary data, deleted amount summary data, and deleted record quantity summary data; Determine a first result according to the first water flow meter data, the record quantity summary data and the deleted record quantity summary data, wherein the first result is used to determine the data quantity integrity of the first water flow meter data; Determine a second result according to the first water flow meter data, the amount summary data and the deleted amount summary data, wherein the second result is used to determine the amount data integrity of the first water flow meter data; If it is determined that the first water flow table data is incomplete according to the first result and the second result, determining the missing data according to the second water flow table data at the current moment in the data warehouse and the water flow table deletion data corresponding to the second water flow table data; The third water flow meter data is obtained according to the lost data and the first water flow meter data, and the third water flow meter data is the complete water flow meter data at the first moment.

2. The method according to claim 1, characterized in that The determining of the first result according to the first flow meter data, the record quantity summary data and the deleted record quantity summary data includes: Obtaining a first difference between the summary data of the number of records and the summary data of the number of deleted records; When the first difference is equal to the total number of water flow meter records in the first water flow meter data, the first result is determined to be that the data quantity of the first water flow meter data is complete; when the first difference is not equal to the total number of water flow meter records in the first water flow meter data, the first result is determined to be that the data quantity of the first water flow meter data is incomplete.

3. The method according to claim 1, characterized in that The determining of the second result according to the first flow table data, the amount summary data and the deleted amount summary data includes: Obtaining a second difference between the amount summary data and the deleted amount summary data; When the first difference is equal to the total amount in the first water flow meter data, the second result is determined to be that the amount data of the first water flow meter data is complete; when the first difference is not equal to the total amount in the first water flow meter data, the second result is determined to be that the amount data of the first water flow meter data is incomplete.

4. The method according to claim 1, characterized in that The determining, according to the first result and the second result, that the first water flow meter data is incomplete includes: When the first result is that the data quantity of the first water flow meter data is incomplete, and / or the second result is that the amount data of the first water flow meter data is incomplete, it is determined that the first water flow meter data is incomplete.

5. The method according to any one of claims 1 to 4, characterized in that: The determining of lost data according to the second water flow table data at the current moment in the data warehouse and the water flow table deletion data corresponding to the second water flow table data includes: Deleting the data whose transaction time is after the first moment in the second transaction table data at the current moment in the data warehouse to obtain a first data set; Deleting the data in the transaction table whose transaction time is after the first moment to obtain a second data set; Obtain data in the second data set whose transaction time is before the first time, and delete the data to obtain a third data set; Missing data is determined based on the first data set and the third data set.

6. The method according to claim 5, characterized in that The determining the missing data according to the first data set and the third data set includes: Determine a union of the first data set and the third data set to obtain a fourth data set; A difference set between the fourth data set and the first water flow meter data is obtained to obtain a fifth data set, where the fifth data set is the lost data.

7. The method according to claim 1, characterized in that After obtaining the third water flow table data according to the lost data and the first water flow table data, the method further includes: Determine a third result according to the third water flow meter data, the record quantity summary data and the deleted record quantity summary data; Determine a fourth result according to the third flow table data, the amount summary data and the deleted amount summary data; When the third result is that the data quantity of the third water meter data is incomplete, and / or the fourth result is that the amount data of the third water meter data is incomplete, the third water meter data is determined to be incomplete, and an abnormal prompt message is sent. The abnormal prompt message is used to prompt the user that the second water meter data is abnormal, and / or the water meter deletion data is abnormal.

8. A computer, characterized in that: include: A memory, a processor, and an executable program code stored in the memory and executable on the processor, wherein the processor executes the steps of the data processing method according to any one of claims 1 to 7 when executing the executable program code.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores executable program code, wherein the executable program code includes execution instructions, and the execution instructions are used to execute the steps of the data processing method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The computer program product comprises a computer program, and the computer program is used to make a computer execute the steps of the data processing method according to any one of claims 1 to 7.

Images