Data consistency determination method and device, electronic equipment and program product

Abstract

The invention provides a data consistency determination method and device, electronic equipment and a program product, and relates to the field of internet storage, and the method comprises the following steps: extracting target data of a specified time slice from a pre-established target database; packaging the extracted target data of the specified time slice into a query request corresponding to the target data; sending a query request corresponding to the target data to the target APIs of the plurality of machine rooms, wherein the query request corresponding to the target data is used for performing data query based on the target APIs; receiving query results returned by the underlying libraries of the plurality of machine rooms; under the condition that the query results are consistent, determining that the data of the plurality of machine rooms are consistent; and under the condition that the query results are inconsistent, determining that the at least one machine room has data exception. According to the method and the device, the technical problem of poor service stability caused by the fact that the influence of service change on a system layer due to data inconsistency cannot be effectively found in the prior art is solved, and the technical effect of improving the stability of a service system is achieved.

Description

Technical Field

[0001] The present disclosure belongs to the field of Internet storage technology, and in particular, relates to a data consistency determination method and device, an electronic device, and a program product. Background Art

[0002] As the business continues to develop, the complexity of online business is increasing. Any changes to the business may have cascading effects. For example, component upgrades, multi-machine room deployment, data governance and other operations will have an impact on the entire system. The impact of these operations is sometimes difficult to accurately assess, and some risks can only be triggered under certain conditions. This can easily lead to them being ignored during business testing, but discovered by users after the business goes online, resulting in problems with business operations and poor user experience.

[0003] Currently, no effective solution has been proposed to the above problems.

[0004] The content of this background technology description is only for facilitating understanding of the relevant technology in this field and is not regarded as an admission of the prior art. Summary of the invention

[0005] The present disclosure aims to provide a method and device for determining data consistency, an electronic device and a program product, which can timely discover data inconsistencies to reduce the impact of business changes on the system level.

[0006] The present disclosure provides a data consistency determination method and device, electronic device, and program product which are implemented as follows:

[0007] A method for determining data consistency, comprising:

[0008] Extract target data of a specified time slice from a pre-established target database;

[0009] The extracted target data of the specified time slice is encapsulated into a query request corresponding to the target data according to the data query conditions corresponding to the target API;

[0010] Sending a query request corresponding to the target data to target APIs of multiple computer rooms, wherein the query request corresponding to the target data is used to perform data query based on the target API;

[0011] Receive query results returned by the underlying databases of multiple computer rooms;

[0012] When the query results are consistent, determining that the data of the multiple computer rooms are consistent;

[0013] When the query results are inconsistent, it is determined that data anomalies exist in at least one computer room.

[0014] In one embodiment, before extracting the target data of the specified time slice from the pre-established target database, the method further includes:

[0015] Get the data generated in real time by the business corresponding to the target API;

[0016] Write the data generated by the business in real time into the data table corresponding to the data warehouse, where there are one or more data tables in the data warehouse, and one business corresponds to one or more data tables;

[0017] Perform data sampling on the data in the data table corresponding to the data warehouse;

[0018] The sampled data is stored in the target database in time slices according to preset format conditions.

[0019] In one embodiment, data sampling is performed on the data in the data table corresponding to the data warehouse, including at least one of the following:

[0020] Perform data sampling on the real-time data in the data table corresponding to the data warehouse;

[0021] Randomly extract a specified number of incremental data that is updated daily in the data table corresponding to the data warehouse;

[0022] Randomly extract a specified number of incremental data that is updated monthly in the data table corresponding to the data warehouse.

[0023] In one embodiment, after the data generated by the business in real time is written into the data table corresponding to the data warehouse, the method further includes:

[0024] The data in the data table corresponding to the data warehouse is stored in the underlying database of the multiple computer rooms.

[0025] In one embodiment, the multiple computer rooms include: a first computer room and a second computer room, wherein the underlying database of the first computer room includes: an hbase database, an es database, and a mysql database, and the underlying database of the second computer room includes: an hbase database, an es database, and a mysql database.

[0026] In one embodiment, extracting data of a specified time slice from a pre-established target database includes:

[0027] Extracting data of a specified time slice from the target database according to a preset period, the specified time slice being the preset period; or,

[0028] When a trigger instruction is received, data of a specified time slice is extracted from the target database.

[0029] In one embodiment, after determining that there is a data anomaly, the method further includes:

[0030] Generate data exception records;

[0031] The data anomaly record is pushed to a preset processing node, and the preset processing node is used to determine the cause of the problem based on the data anomaly record.

[0032] A data consistency determination device, comprising:

[0033] An extraction module, used for extracting target data of a specified time slice from a pre-established target database;

[0034] The encapsulation module is used to encapsulate the extracted target data of the specified time slice into a query request corresponding to the target data according to the data query conditions corresponding to the target API;

[0035] A sending module, used for sending a query request corresponding to the target data to target APIs of multiple computer rooms, wherein the query request corresponding to the target data is used for performing data query based on the target API;

[0036] A receiving module is used to receive query results returned by the underlying libraries of multiple computer rooms;

[0037] A first determination module, configured to determine that the data of the plurality of computer rooms are consistent when the query results are consistent;

[0038] The second determination module is used to determine that data anomalies exist in at least one computer room when the query results are inconsistent.

[0039] An electronic device comprises a processor and a memory for storing instructions executable by the processor, wherein the steps of the above method are implemented when the processor executes the instructions.

[0040] A program product comprises a computer program, wherein the steps of the above method are implemented when the computer program is executed by a processor.

[0041] The data consistency determination method provided by the present disclosure extracts the target data of the specified time slice from the pre-established target database; encapsulates the extracted data of the specified time slice into a query request corresponding to the target data according to the data query conditions corresponding to the target API; transmits the query request to the target API of multiple computer rooms for data query; determines whether the query results returned by the underlying libraries of multiple computer rooms are consistent; if the results are consistent, determines that the data are consistent; if the results are inconsistent, determines that there is a data anomaly. That is, the data consistency determination method provided by the present disclosure compares the consistency of the data returned by the underlying libraries of multiple computer rooms based on the time slice data to determine whether there is an anomaly in the data. The present disclosure solves the technical problem of poor business stability caused by the inability to effectively discover the impact of business changes caused by data inconsistency on the system layer through the above method, and achieves the technical effect of improving the stability of the business system.

[0042] Other optional features and technical effects of the embodiments of the present disclosure are partially described below, and partially can be understood by reading this document. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. The elements shown are not limited by the proportions shown in the accompanying drawings. The same or similar reference numerals in the accompanying drawings represent the same or similar elements, wherein:

[0044] Figure 1 A schematic diagram of the architecture of a business system provided by an embodiment of the present disclosure is shown;

[0045] Figure 2 An exemplary flow chart of a method for determining data consistency provided by an embodiment of the present disclosure is shown;

[0046] Figure 3 A logical schematic diagram of determining data consistency in multiple computer rooms provided by an embodiment of the present disclosure is shown;

[0047] Figure 4 A schematic diagram showing the module structure of the data consistency determination device provided by the present disclosure is shown; and

[0048] Figure 5 An exemplary hardware structure block diagram of an electronic device capable of implementing the data consistency determination method provided by the present disclosure is shown. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with specific implementation methods and drawings. Here, the illustrative implementation methods and descriptions of the present disclosure are used to explain the present disclosure, but are not intended to limit the present disclosure.

[0050] For existing business systems, there are generally multiple computer rooms, each of which can support multiple businesses, and each of which is equipped with multiple underlying libraries. Figure 1 As shown, the business system may include: a first computer room and a second computer room. The first computer room is provided with: an API layer, a data center and an underlying library, wherein the API layer is connected to multiple application programming interfaces, such as: a first API, a second API...Nth API, and the API layer interacts with the underlying library through the data center, wherein the underlying library includes multiple storage units, such as: an hbase database, an es database and a mysql database. The second computer room is also provided with: an API layer, a data center and an underlying library, wherein the API layer is connected to multiple application programming interfaces, such as: a first API, a second API...Nth API, and the API layer interacts with the underlying library through the data center, wherein the underlying library includes multiple storage units, such as: an hbase database, an es database and a mysql database.

[0051] In actual scenarios, multiple computer rooms serve as primary and backup computer rooms for each other, and the data in multiple computer rooms needs to be consistent. However, when the business changes, sometimes some changes or failures may cause inconsistencies in the data in the underlying databases of multiple computer rooms.

[0052] To this end, a method for determining data consistency is provided in this example, wherein the extracted target data of a specified time slice is encapsulated into a query request corresponding to the target data according to the data query conditions corresponding to the target API, and the query request corresponding to the target data is transmitted to the target APIs of multiple computer rooms for data query, so as to compare the data returned by multiple underlying libraries, thereby achieving the determination of data consistency. In the embodiment of the present disclosure, the API corresponds to the business, and different APIs correspond to different businesses. Through the scheme provided by the present disclosure, the target API corresponding to a certain business can be used to access or query the data of the business stored in different computer rooms, and the results of the access or query can be used to determine whether the business data corresponding to the target API in each computer room is consistent.

[0053] Figure 2It is a method flow chart of an embodiment of the data consistency determination method provided by the present disclosure. Although the present disclosure provides method operation steps or device structures as shown in the following embodiments or drawings, more or fewer operation steps or module units may be included in the method or device based on routine or without creative labor. In the steps or structures where there is no necessary causal relationship logically, the execution order of these steps or the module structure of the device is not limited to the execution order or module structure described in the embodiments of the present disclosure and shown in the drawings. When the method or module structure described is applied to an actual device or terminal product, it can be connected according to the method or module structure shown in the embodiments or drawings for sequential execution or parallel execution (for example, a parallel processor or a multi-threaded processing environment, or even a distributed processing environment).

[0054] Specifically, Figure 2 As shown, the above data consistency determination method at least includes the following steps S201-S205.

[0055] S201: extracting target data of a specified time slice from a pre-established target database.

[0056] That is, a database can be pre-set or selected as the target database. For example, the MySQL database in the underlying library can be selected as the target database. The target database can be used to store the data generated by each business, and it can be used as the basic judgment data for sampling. The data generated by these businesses can also be stored in the database of the underlying library of each computer room.

[0057] The data in the target database can be stored according to time, that is, the stored data can be marked with time. If it is necessary to extract data of a certain time slice (such as a specified time slice), the data within the specified time slice range can be extracted according to the time mark. Exemplarily, the specified time slice is a preset time length with the current date as the endpoint of the time period and earlier than the current date. The preset time length is set as needed, such as 1 day, 7 days or 15 days. For example, the specified time slice is a time period within 7 days before the current date.

[0058] S202: Encapsulate the extracted target data of the specified time slice into a query request corresponding to the target data according to the data query condition corresponding to the target API.

[0059] For example, the API layer has access to the first API, the second API, etc. Different APIs correspond to different data query requirements. After obtaining the data, the API to which the data corresponds can be determined, and then encapsulated into a query request for the corresponding API to trigger a data query for the corresponding API.

[0060] Specifically, when matching the corresponding API, it may be the indicator information of the data to be obtained. The indicator information may include: business products, company name, identity information, etc. Based on this information and the API tag, it is matched to determine the API business corresponding to the data. Different APIs correspond to different data request parameters. Therefore, the obtained data can be encapsulated into a query request corresponding to the target data based on the matched data request parameters, so as to facilitate the corresponding target API call.

[0061] For example, if the indicator information is user identity information, then the user identity information can be matched with the labels of each API to determine which one or more APIs can provide user identity information queries, and then it can be encapsulated as a user identity information query request for the matched one or more APIs.

[0062] In other embodiments, it is also possible to select which computer rooms have consistent data, so that when it is determined that the data in some computer rooms are consistent, the computer rooms whose data consistency has not been determined can be selected to confirm the data consistency. In one example, when the corresponding API is matched, one or more APIs can be selected as the target API from the matched APIs, and an information query request can be encapsulated for the target API. Specifically, the computer room information of the computer room that has been confirmed for consistency can be recorded, and the API corresponding to the computer room information that has not been recorded can be selected from the matched APIs as the target API. The method of the disclosed embodiment can avoid repeated confirmation of computer room data with known data consistency, thereby improving the efficiency of data consistency determination.

[0063] For example, the API tag may include computer room information, which is used to describe the information of the computer room storing the API business data, and the computer room information includes but is not limited to one or more computer room addresses to which the API belongs. In one example, when API-1 of business 1 is successfully matched, the API tag of API-1 includes three computer room information - computer room address 1, computer room address 2, and computer room address 3, wherein, for data A in time slice A, the computer room data corresponding to computer room address 1 and computer room address 2 have been determined to be consistent, and the consistency result has been recorded. At this time, only the query request corresponding to the target data can be encapsulated for computer room address 3, and it should be understood that the time period represented by the time slice corresponding to the target data should be consistent with time slice A.

[0064] In specific implementation, when a certain API is successfully matched, the data center information of the API tag can be displayed in different forms depending on whether the data is consistent.

[0065] In another specific implementation, when a certain API is successfully matched and before a query request is generated, a prompt message may pop up, "The computer room corresponding to computer room address 1 and the computer room corresponding to computer room address 2 have been confirmed to be consistent. Do you want to generate a query request only for computer room address 3?" If "Yes" is selected, a query request is generated only for computer room address 3. If "No" is selected, query requests are generated for computer room addresses 1, 2, and 3.

[0066] In other implementations, when a certain API is successfully matched and before a query request is generated, a list of computer room information may pop up, and a preset label may be displayed after each computer room information. The preset label is used to indicate whether data consistency is determined, such as "consistency determined" or "not consistent", and the computer rooms for which query requests are generated may be determined by checking different computer room information. For example: the preset labels of computer room addresses 1 and 2 are "consistency determined", and the preset label of computer room address 3 is "not consistent", then computer room address 3 may be checked, and the corresponding query request may be generated only for computer room address 3; computer room addresses 1, 2, and 3 may also be checked, and query requests may be generated for computer room addresses 1, 2, and 3.

[0067] In the above embodiment, the time period represented by time slice A can be specifically set as needed, for example, 1 day ago, 1 week ago, etc. For example, if the time period represented by time slice A is one day ago, then the time period represented by time slice B is the same as time slice A, which is also one day before the time corresponding to data A. For example, today is 8:00:00 am on January 2, 2024, then the time range represented by time slice A is 8:00:00 am on January 1, 2024-7:59:59 am on January 2, 2024, and the time range represented by time slice B is 8:00:00 am on January 1, 2024-7:59:59 am on January 2, 2024.

[0068] S203: Sending a query request corresponding to the target data to target APIs of multiple computer rooms, wherein the query request corresponding to the target data is used to perform data query based on the target API.

[0069] S204: receiving query results returned by the underlying libraries of multiple computer rooms.

[0070] S205: When the query results are consistent, determining that the data in the multiple computer rooms are consistent;

[0071] S206: When the query results are inconsistent, it is determined that data anomalies exist in at least one computer room.

[0072] Specifically, it is necessary to establish the above-mentioned target database to ensure that there is enough data in it for extraction. To this end, before extracting the data of the specified time slice from the pre-established target database, it also includes: obtaining the data generated in real time by the business corresponding to the target API; writing the data generated in real time by the business into the data table corresponding to the data warehouse, wherein there are one or more data tables in the data warehouse, and one business corresponds to one or more data tables; sampling the data in the data table corresponding to the data warehouse; and storing the sampled data in the target database in time slices according to a preset format. In one example, the above-mentioned data warehouse can be a Hive data warehouse, but the present disclosure is not limited to this. For the sampled data, because of the different data types, for example, some are business data and some are user data, different data types have different corresponding access formats, that is, different table headers or different fields in the table, etc., which can be considered as preset format conditions.

[0073] The above-mentioned time-slice storage can be understood as storing data according to different cycles or time slices. For example, some data needs to be stored every day, so these data need to be extracted once a day; some data (such as data from cooperation with partners) is updated once a week, so these data only need to be extracted once the next day after the data comes, that is, extracted once a week; some data is updated monthly, so these data are extracted once a month. Because of the different extraction frequencies, they correspond to different time slices. In some embodiments, data of different time slices can be stored by time partitioning. Specifically, time partitioning is to distribute data in different storage partitions according to the time range, which helps to process time series data and control the time range covered by each storage partition. The time column can be selected as the partition key, such as date, year or month. When implementing it specifically, the time range of the partition can be set according to business needs, such as one partition per day or one partition per month. The storage method of time slices can improve query efficiency, simplify data management, and improve overall system performance.

[0074] That is, the real-time data of each business line can be written into the Hive table through the corresponding API interface. The data in the Hive table can be sampled and put into the target database on the one hand, and on the other hand, it can be saved in the underlying database of each computer room.

[0075] ‌Among them, the data warehouse is a subject-oriented, integrated, relatively stable data set that reflects historical changes. The data warehouse can be used to assist decision support and provide current and historical data for business intelligence analysis and decision support.‌ Through multidimensional data analysis in the data warehouse, data can be organized for analysis actions such as slicing, dicing, drilling, and rotating, so as to deeply understand the information in the data, which helps to generate relevant business indicators, ensure monitoring quality, and avoid missing sample data.

[0076] The data in the data table corresponding to the data warehouse is sampled, which may include but is not limited to at least one of the following:

[0077] 1) Perform data sampling on the real-time data in the data table corresponding to the data warehouse;

[0078] 2) Randomly extract a specified number of incremental data that is updated daily in the data table corresponding to the data warehouse;

[0079] 3) Randomly extract a specified number of incremental data that is updated monthly in the data table corresponding to the data warehouse.

[0080] In some embodiments, after the data generated by the business in real time is written into the data table of the data warehouse, the data in the data table of the data warehouse can be stored in the underlying libraries of the multiple computer rooms. For example, in the data warehouse, the data is stored in the form of tables, and the data in the form of tables in the Hive table of the data warehouse can be loaded and written into the underlying library of the computer room to achieve storage.

[0081] Among them, the multiple computer rooms may include: a first computer room and a second computer room, wherein the underlying library of the first computer room may include: an hbase database, an es database, and a mysql database, and the underlying library of the second computer room may include: an hbase database, an es database, and a mysql database.

[0082] The execution subject of the above data consistency determination can be: a processor for data processing, that is, a processor for data processing can be introduced, and data can be extracted from the target database through the processor to realize data consistency judgment. The processor for data processing can be set in one of the computer rooms, or can be set separately, or can be set in a cloud server, etc. The specific setting method can be selected according to actual needs, and this disclosure does not limit this.

[0083] The method provided by the embodiments of the present disclosure may be executed automatically or after being triggered. In some examples, extracting data of a specified time slice from a pre-established target database may include: automatically extracting data with a time slice of the day from the target database every day; or manually triggering the extraction of data of a selected time slice from the target database.

[0084] Considering that after the data consistency judgment is performed, if the data is consistent, it can be determined that the judgment is passed and there is no problem with the data. If the data is determined to be inconsistent, it can be determined that there is a problem in the data flow or other operation process of the business. After the storage problem is determined, the cause of the problem can be automatically analyzed, or the problem can be sent to the processing personnel for investigation and resolution. To this end, after determining that there is a data anomaly, it can also include: generating a data anomaly record; pushing the data anomaly record to a preset processing node, and the preset processing result intervenes to determine the cause of the problem based on the abnormal data. In this way, problems in data flow or business changes can be discovered in a timely manner, ensuring the stability of the business.

[0085] Specifically, when data inconsistency is determined, an operation log or analysis report can be automatically generated, and the database where the inconsistent data is located, the storage location in the database, and the specific data content can be recorded in the operation log or analysis report, and then the database where the inconsistent data is located, the storage location in the database, and the specific data content can be recorded so that the problem can be quickly located later. In the case of detected data inconsistency, data resynchronization can be triggered to ensure data consistency between the two computer rooms. Alternatively, an event-driven architecture can be used to synchronize data through event logs. Specifically, an event log can be generated for each database operation, and the databases of the two computer rooms are synchronized through event logs. When inconsistencies occur, repairs are made by replaying the event logs. Alternatively, log recording and comparison technology can be used. When synchronization fails or data is inconsistent, an operation log is automatically generated, listing the abnormal fields and corresponding databases, and providing a detailed analysis report for reference by the processing personnel.

[0086] To troubleshoot data inconsistency, you can check the data update frequency from the data dimension. If the update frequency is consistent, check whether the data in the underlying database is consistent. If it is consistent, check whether there is any online operation recently from the business side. If there is an online operation, you can check the code of the online application to find out whether there is a problem. The problems involved may be logic processing exceptions, network exceptions, etc. For example, sometimes the route is set to call data A, but data B is actually called, resulting in data inconsistency.

[0087] The storage and extraction of the above data can be achieved through offline ETL (Extract, Transform, Load). The offline ETL process is a bridge between the data source and the data application target. Through offline ETL, a specified number of updated incremental data such as t+1 (data updated daily, t represents day) / m+1 (data updated monthly, m represents month) can be randomly extracted. Furthermore, after randomly extracting a specified number of data, it is determined whether each data source has extracted hit data. If some data sources have not extracted hit data, data is extracted from the data source to determine that each data source can extract hit data and encapsulate it into corresponding data query conditions.

[0088] Specifically, when extracting data, you can sample data from the data tables table_1...table_n in the Hive data warehouse every day. Because each table has different update characteristics, different rules are used for sample extraction according to different tables (t+0, t+1, and m+1). That is, different tables have different sampling intervals, and different sampling intervals are set according to the needs of days or months. The t+0 real-time table stores real-time sample data, the t+1 table stores sample data from the previous day that is updated every day, and the m+1 table stores sample data from the previous month that is updated every month.

[0089] Furthermore, after the extracted target data of the specified time slice is encapsulated into a query request corresponding to the target data according to the data query conditions corresponding to the target API, the data rules can be combined according to the time of the day of extraction, and sorted into or generated business request parameters required by the API. Specifically, the business line corresponding to the corresponding data product can be identified, and then the data rules are combined according to the time slice, data API label, data request parameters, etc., and then inserted into the corresponding underlying library. The above operations can be repeated every day to ensure that the data can be updated in a rolling manner. In some embodiments, the data rules can be various calculations for different data request parameters, or various calculations for data with different API labels. The data rules can be specifically set according to actual needs and are not limited here. In a specific example, for the transaction data evaluation API, when a transaction data sample of 2000-01-01 is extracted from the target database on 2000-01-02, including transaction ID, transaction timestamp, transaction type, transaction amount, etc., the system recognizes that these data belong to the financial business line, and determines that the corresponding API is the transaction data evaluation API, and then combines the parameters according to the data rule combination corresponding to the transaction data evaluation API to obtain the transaction ID, transaction date, and daily total transaction amount required by the transaction data evaluation API, where the daily total transaction amount is a parameter or indicator obtained based on the transaction timestamp, transaction type, and transaction amount. Then, the system inserts the sorted data into the predefined underlying MySQL database, and this process is repeated every day to achieve rolling updates of the data.

[0090] When implementing, the comparison platform can obtain the full amount of data for comparison every day, so as to achieve effective data monitoring and avoid missing sample data. Furthermore, when there is an online situation, the impact of the online operation on the data can also be identified in real time. For example, in actual use, for scenarios with frequent online operations, for example, there are 8 online requirements in a month, and data anomalies caused by 2 processes are found.

[0091] The above method is described below in conjunction with a specific embodiment. However, it should be noted that this specific embodiment is only for better illustrating the present disclosure and does not constitute an improper limitation on the present disclosure.

[0092] At present, for business testing, the common scenario in the industry is that each component performs consistency verification on its own dual computer rooms. For example, hbase performs consistency verification and comparison tools on itself; elasticsearch performs consistency verification and comparison tools on itself. However, because each component only performs consistency verification on itself, it is impossible to perform full-process monitoring on the entire business product line. In other scenarios, there is another way to deal with it, which is to use a batch of the same samples for continuous monitoring or to perform consistency comparisons at different time periods every day. This method can usually detect some problems, such as: problems caused by changes in some components, changes in software or technical architecture in multiple computer rooms, etc. However, this method also has some problems, such as:

[0093] Data fragmentation: It is impossible to link with the underlying ETL to compare the underlying data changes with the online performance;

[0094] Poor timeliness: Data has its own uniqueness and time validity, and cannot be used in all business scenarios. It needs to be triggered by specific scenarios.

[0095] Late problem discovery: When there are ETL data problems, the problem perception cycle will be relatively long, which will inevitably cause customers to perceive the problem first, seriously affecting the user experience.

[0096] Based on this, this example provides a method for determining data consistency in multiple computer rooms to ensure that changes at the system or component level can be perceived in a timely manner and can be linked with the underlying ETL process. Figure 3 The figure shows a logical diagram for determining the consistency of data in multiple computer rooms, where ZW computer room and YZ computer room represent the numbers of the two computer rooms, t+0 represents real-time data, t+1 represents data updated daily, m+1 represents data updated monthly, ETL represents the data processing flow, the API layer includes services of multiple business lines, hbase, es, and mysql are data storage components in the underlying library, and the data center is a unified data abstraction layer used to provide queries for various online data.

[0097] Specifically, the above data consistency comparison method may include the following steps:

[0098] S1: Offline data extraction:

[0099] The offline ETL (Extract, Transform, Load) process is a bridge between the data source and the data application target. Through offline ETL, a specified number of incremental data updated by t+1 (data updated daily) / m+1 (data updated monthly) are randomly extracted. During the data extraction process, try to ensure that each data source can extract a hit data source, and encapsulate the extracted data into the corresponding data query conditions;

[0100] S2: Real-time data extraction:

[0101] Extract incremental data from real-time updated data in a t+1 cycle, randomly extract a specified number of incremental data, and ensure that each data source can extract a hit data source and encapsulate it into the corresponding data query condition;

[0102] S3: Store the extracted data in the underlying database:

[0103] The extracted data source is stored in the specified MySQL in time slices according to the specified format conditions. Then, steps S1 and S2 are repeated periodically every day to ensure the extraction of incremental data.

[0104] S4: Data consistency comparison:

[0105] First, extract the data of the specified time slice in the specified MySQL, and then encapsulate the extracted data into the query request corresponding to the target data according to the data query conditions corresponding to the API, so as to compare the consistency of the data in multiple computer rooms / multiple environments. If the results are consistent, it means there is no problem. If the results are inconsistent, it means that there is an anomaly in the data and the problem needs to be investigated. In this way, the data results of multiple computer rooms can be guaranteed to be consistent.

[0106] For the above data consistency comparison method, assuming that there are two computer rooms, ZW computer room and YZ computer room, the data consistency comparison can be performed as follows:

[0107] a) ETL warehousing (offline data written to the database):

[0108] The ZW computer room processes the offline updated data, stores the data tables (table_1 to table_n) in the Hive data warehouse into the ZW computer room according to the format of the respective tables of the online database, and synchronously stores the data into the database of the YZ computer room. This process can ensure the consistency of the data in the database of the ZW computer room and the database of the YZ computer room. Among them, one API corresponds to one or more data tables (table), and one data table (table) corresponds to one API.

[0109] b) ETL data extraction (real-time data written into Hive data warehouse):

[0110] The data of t+0 (real-time data) is extracted, and the data generated by the business in real time is written to the Hive data warehouse in real time through the consumer end and other businesses to ensure that the Hive data warehouse has real-time online business data.

[0111] c) ETL sample extraction:

[0112] Among them, a) and b) are the prerequisites for extracting sample data. That is, offline data and real-time data must be stored in the warehouse before sample extraction can be triggered. Specifically, when extracting sample data, ensure that the corresponding table data is stored in the warehouse in process a before extracting data samples from the corresponding table, and ensure that the data is written into the data warehouse in process b before extracting the corresponding data source.

[0113] Specifically, data can be sampled daily from the data tables table_1...table_n in the Hive data warehouse. Because each table has different update characteristics, different rules for sample extraction are used for different tables. The t+0 real-time table stores real-time sample data, the t+1 table stores sample data from the previous day that is updated every day, and the m+1 table stores sample data from the previous month that is updated every month.

[0114] Specifically, the processor for performing data processing may perform data update comparison in the following manner:

[0115] First: Automatic update. By querying the relevant data in MySQL at the same time every day, the corresponding indicator information (for example, business products, company name, identity information, etc.) is obtained, and the data API tag is used for identification. Different data tags call different API business lines, and the stored data request parameters are encapsulated into the specified API interface for calling. Initiate call requests to APIs in different computer rooms and different environments in the same computer room, and compare the returned results for consistency. If the data is consistent, there is no problem; if the data is inconsistent, it means that there are differences in the data, and there are problems in the data flow or other operations of the business, which requires manual intervention to troubleshoot the operation.

[0116] Second: Manual update. Manual update can specify data shards for data backtracking comparison. The process is the same as automatic update, except that the data samples obtained at the corresponding time are different. Manual update comparison is applicable to consistency regression of change operations of system components, software components, etc.

[0117] In the above example, corresponding business indicators are generated based on specific data to ensure monitoring quality and avoid missing sample data. Through manual or automatic data updates and comparisons, the purpose of adaptive rolling monitoring can be achieved. By combining ETL, the need for data consistency comparison in multiple computer rooms can be met.

[0118] The above solution solves the problem that failures caused by existing business changes cannot be discovered in time during actual testing scenarios, resulting in customers discovering them earlier than business personnel. Failures can be discovered in time through data consistency comparison, and periodic data comparison can ensure that the data generated by the business can be continuously monitored, thereby ensuring the accuracy of the data in each computer room.

[0119] The method embodiments provided in the above embodiments of the present disclosure can also be executed in a mobile terminal, a computer terminal or a similar computing device.

[0120] The present disclosure also provides a data consistency determination device 400. In a preferred embodiment, the data scoring device 400 can be implemented based on software. Figure 4 As shown, including:

[0121] Extraction module 401, used to extract target data of a specified time slice from a pre-established target database;

[0122] The encapsulation module 402 is used to encapsulate the extracted target data of the specified time slice into a query request corresponding to the target data according to the data query condition corresponding to the target API;

[0123] A sending module 403, used to send a query request corresponding to the target data to target APIs of multiple computer rooms, where the query request corresponding to the target data is used to perform data query based on the target API;

[0124] The receiving module 404 is used to receive the query results returned by the underlying libraries of multiple computer rooms;

[0125] A first determination module 405 is used to determine that the data of the multiple computer rooms are consistent when the query results are consistent;

[0126] The second determination module 406 is used to determine that data anomalies exist in at least one computer room when the query results are inconsistent.

[0127] In one embodiment, the above-mentioned data consistency determination device 400 can also obtain the data generated in real time by the business corresponding to the target API before extracting the target data of the specified time slice from the pre-established target database; write the data generated in real time by the business into the data table corresponding to the data warehouse, wherein there are one or more data tables in the data warehouse, and one business corresponds to one or more data tables; sample the data in the data table corresponding to the data warehouse; and store the sampled data in the target database in time slices according to preset format conditions.

[0128] In one embodiment, data sampling is performed on the data in the data table corresponding to the data warehouse, and may include but is not limited to at least one of the following:

[0129] Perform data sampling on the real-time data in the data table corresponding to the data warehouse;

[0130] Randomly extract a specified number of incremental data that is updated daily in the data table corresponding to the data warehouse;

[0131] Randomly extract a specified number of incremental data that is updated monthly in the data table corresponding to the data warehouse.

[0132] In one embodiment, after writing the data generated by the business in real time into the data table corresponding to the data warehouse, the data consistency determination device 400 may also store the data in the data table corresponding to the data warehouse into the underlying libraries of the multiple computer rooms.

[0133] In one embodiment, the above-mentioned multiple computer rooms may include: a first computer room and a second computer room, wherein the underlying database of the first computer room includes: an hbase database, an es database, and a mysql database, and the underlying database of the second computer room includes: an hbase database, an es database, and a mysql database.

[0134] In one embodiment, extracting data of a specified time slice from a pre-established target database may include: extracting data of a specified time slice from the target database according to a preset period, the specified time slice being the preset period; or, extracting data of a specified time slice from the target database upon receiving a trigger instruction.

[0135] In one embodiment, after determining that there is a data anomaly, the data consistency determination device 400 may also generate a data anomaly record; and push the data anomaly record to a preset processing node, where the preset processing node is used to determine the cause of the problem based on the data anomaly record.

[0136] In an embodiment of the present disclosure, an electronic device is provided, including: a processor and a memory storing a computer program, wherein the processor is configured to execute any data consistency determination method of the embodiment of the present disclosure when running the computer program.

[0137] Figure 5 A schematic diagram of an electronic device 500 that can implement the method of the disclosed embodiment or implement the disclosed embodiment is shown, and in some embodiments, more or fewer electronic devices may be included than shown. In some embodiments, it can be implemented using a single or multiple electronic devices. In some embodiments, it can be implemented using cloud or distributed electronic devices.

[0138] like Figure 5 As shown, the electronic device 500 includes a central processing unit (CPU) 501, which can perform various appropriate operations and processes according to the programs and / or data stored in the read-only memory (ROM) 502 or the programs and / or data loaded from the storage part 508 to the random access memory (RAM) 503. CPU 501 can be a multi-core processor or include multiple processors. In some embodiments, CPU 501 can include a general main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. In RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. CPU 501, ROM 502 and RAM 503 are connected to each other via a bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0139] The processor and the memory are used together to execute a program stored in the memory. When the program is executed by a computer, the steps or functions of the data consistency determination method described in the above embodiments can be implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0140] Step 1: Extract data of a specified time slice from a pre-established target database;

[0141] Step 2: Encapsulate the extracted data of the specified time slice into a target query request according to the data query conditions corresponding to the target API;

[0142] Step 3: transmitting the target query request to the target APIs of multiple computer rooms for data query;

[0143] Step 4: Determine whether the query results returned by the underlying databases of multiple computer rooms are consistent;

[0144] Step 5: If the results are consistent, determine that the data is consistent; if the results are inconsistent, determine that there is a data anomaly.

[0145] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, a touch screen, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed. Figure 5 Only some components are schematically shown in the figure, which does not mean that the computer system 500 only includes Figure 5 Components shown.

[0146] In some embodiments, the electronic device 500 refers to a mobile terminal or a computer, including a mobile phone, a car terminal, a smart TV, etc. Taking a mobile phone as an example, the electronic device 500 also includes a display screen with a touch function, an external speaker, a gyroscope, a camera, a 4G / 5G antenna and other device modules.

[0147] The systems, devices, modules or units described in the above embodiments may be implemented by a computer or its associated components. The computer may be, for example, a mobile terminal, a smart phone, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a personal digital assistant, a media player, a navigation device, a game console, a tablet computer, a wearable device, a smart TV, an Internet of Things system, a smart home, an industrial computer, a server or a combination thereof.

[0148] Although not shown, in an embodiment of the present disclosure, a program product is provided, the program product including a computer program, the computer program being configured to perform any method of the embodiments of the present disclosure when executed.

[0149] Although not shown, in an embodiment of the present disclosure, a storage medium is provided, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, all steps of the data consistency determination method in the above embodiment are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0150] Step 1: Extract target data of a specified time slice from a pre-established target database;

[0151] Step 2: Encapsulate the extracted target data of the specified time slice into a query request corresponding to the target data according to the data query conditions corresponding to the target API;

[0152] Step 3: Sending a query request corresponding to the target data to target APIs of multiple computer rooms, where the query request corresponding to the target data is used to perform data query based on the target API;

[0153] Step 4: Receive query results returned by the underlying databases of multiple computer rooms;

[0154] Step 5: When the query results are consistent, determining that the data of the multiple computer rooms are consistent;

[0155] Step 6: If the query results are inconsistent, determine that at least one computer room has data anomalies.

[0156] From the above description, it can be seen that the embodiment of the present disclosure extracts the target data of the specified time slice from the pre-established target database; encapsulates the extracted data of the specified time slice into a query request corresponding to the target data according to the data query conditions corresponding to the target API; transmits the target query request to the target API of multiple computer rooms for data query; determines whether the query results returned by the underlying libraries of multiple computer rooms are consistent; if the results are consistent, determines that the data are consistent; if the results are inconsistent, determines that there is a data anomaly. That is, based on the time slice data, the data returned by the underlying libraries of multiple computer rooms are compared for consistency to determine whether there is an anomaly in the data. The above method solves the existing technical problem of poor business stability caused by the inability to effectively discover the impact of business changes caused by data inconsistency on the system layer, and achieves the technical effect of improving the stability of the business system.

[0157] The information, data and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data comply with the relevant laws, regulations and standards of relevant countries and regions.

[0158] The storage media in the embodiments of the present disclosure include permanent and non-permanent, removable and non-removable items that can be implemented by any method or technology to store information. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0159] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the hardware + program embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0160] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0161] Although the embodiments of this specification provide method operation steps as described in the embodiments or flow charts, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements.

[0162] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing the embodiments of this specification, the functions of each module can be implemented in the same or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0163] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules for implementing the method and structures within the hardware component.

[0164] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0165] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, the embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0166] The present specification embodiments may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present specification embodiments may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0167] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In the description of this specification, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, in the absence of contradiction, a person skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0168] The above is only an example of the embodiment of the present specification and is not intended to limit the embodiment of the present specification. For those skilled in the art, the embodiment of the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiment of the present specification shall be included in the scope of the claims of the embodiment of the present specification.

Claims

1. A method for determining data consistency, characterized in that: include: Extract target data of a specified time slice from a pre-established target database; The extracted target data of the specified time slice is encapsulated into a query request corresponding to the target data according to the data query conditions corresponding to the target API; Sending a query request corresponding to the target data to target APIs of multiple computer rooms, wherein the query request corresponding to the target data is used to perform data query based on the target API; Receive query results returned by the underlying databases of multiple computer rooms; When the query results are consistent, determining that the data of the multiple computer rooms are consistent; When the query results are inconsistent, it is determined that data anomalies exist in at least one computer room.

2. The method according to claim 1, characterized in that Before extracting the target data of the specified time slice from the pre-established target database, it also includes: Get the data generated in real time by the business corresponding to the target API; Write the data generated by the business in real time into the data table corresponding to the data warehouse, where there are one or more data tables in the data warehouse, and one business corresponds to one or more data tables; Perform data sampling on the data in the data table corresponding to the data warehouse; The sampled data is stored in the target database in time slices according to preset format conditions.

3. The method according to claim 2, characterized in that Data sampling is performed on the data in the data table corresponding to the data warehouse, including at least one of the following: Perform data sampling on the real-time data in the data table corresponding to the data warehouse; Randomly extract a specified number of incremental data that is updated daily in the data table corresponding to the data warehouse; Randomly extract a specified number of incremental data that is updated monthly in the data table corresponding to the data warehouse.

4. The method according to claim 2, characterized in that: After the data generated by the business in real time is written into the corresponding data table of the data warehouse, it also includes: The data in the data table corresponding to the data warehouse is stored in the underlying database of the multiple computer rooms.

5. The method according to claim 4, characterized in that The multiple computer rooms include: a first computer room and a second computer room, wherein the underlying database of the first computer room includes: an hbase database, an es database, and a mysql database, and the underlying database of the second computer room includes: an hbase database, an es database, and a mysql database.

6. The method according to any one of claims 1 to 5, characterized in that Extract data of a specified time slice from a pre-established target database, including: Extracting data of a specified time slice from the target database according to a preset period, the specified time slice being the preset period; or, When a trigger instruction is received, data of a specified time slice is extracted from the target database.

7. The method according to any one of claims 1 to 5, characterized in that After determining that there are data anomalies, it also includes: Generate data exception records; The data anomaly record is pushed to a preset processing node, and the preset processing node is used to determine the cause of the problem based on the data anomaly record.

8. A data consistency determination device, characterized in that: include: An extraction module, used for extracting target data of a specified time slice from a pre-established target database; The encapsulation module is used to encapsulate the extracted target data of the specified time slice into a query request corresponding to the target data according to the data query conditions corresponding to the target API; A sending module, used for sending a query request corresponding to the target data to target APIs of multiple computer rooms, wherein the query request corresponding to the target data is used for performing data query based on the target API; A receiving module is used to receive query results returned by the underlying libraries of multiple computer rooms; A first determination module, configured to determine that the data of the plurality of computer rooms are consistent when the query results are consistent; The second determination module is used to determine that data anomalies exist in at least one computer room when the query results are inconsistent.

9. An electronic device comprising a processor and a memory for storing instructions executable by the processor, characterized in that: When the processor executes the instructions, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A program product comprising a computer program, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images