Method and apparatus for maintaining data availability of a data system

Abstract

The embodiment of the application discloses a data availability maintenance method and device of a data system, the method comprising: obtaining platform data generated by a resource platform; performing offline calculation on the platform data according to a first preset time interval to obtain offline data indexes corresponding to each first preset time interval; performing real-time calculation on the platform data to obtain real-time data indexes; for any first preset time interval, comparing the offline data indexes corresponding to the first preset time interval with the real-time data indexes generated within the first preset time interval, and judging whether there is abnormal data in the data system within the first preset time interval according to the comparison result; if there is abnormal data, performing data repair processing on the abnormal data. The application can monitor the data availability in the data system, can discover abnormal data in time, and can start a data repair process in the case of abnormal data, thereby reducing the influence of abnormal data on online business.

Description

Technical Field

[0001] This application relates to the field of Internet technology, specifically to a method and apparatus for maintaining the data availability of a data system. Background Technology

[0002] Current technologies primarily address data system availability by implementing redundant backups of data system services. However, these methods are ineffective at promptly detecting and processing abnormal data, lacking the capability to monitor and repair anomalies. Consequently, the presence of abnormal data in the system can significantly impact online business operations. Summary of the Invention

[0003] In view of the above problems, this application proposes a data availability maintenance method, apparatus, computing device and computer storage medium for a data system, to solve the following problems: existing methods cannot detect abnormal data in a timely manner and cannot effectively process abnormal data.

[0004] According to one aspect of the embodiments of this application, a data availability maintenance method for a data system is provided, comprising:

[0005] Acquire platform data generated by the resource platform;

[0006] The platform data is calculated offline according to the first preset time interval to obtain the offline data indicators corresponding to each first preset time interval.

[0007] Real-time calculations are performed on platform data to obtain real-time data metrics;

[0008] For any first preset time interval, the offline data indicators corresponding to the first preset time interval are compared with the real-time data indicators generated within the first preset time interval, and the data system is judged to have any abnormal data within the first preset time interval based on the comparison results.

[0009] If abnormal data is found, data repair processing will be performed on the abnormal data.

[0010] Furthermore, the offline calculation of platform data according to the first preset time interval, to obtain the offline data indicators corresponding to each first preset time interval, further includes:

[0011] Save platform data to an offline data warehouse;

[0012] Based on the preset data indicator calculation logic, the platform data in the offline data warehouse is calculated offline according to the second preset time interval to obtain the offline incremental data indicators corresponding to each second preset time interval, and the offline incremental data indicators corresponding to each second preset time interval are saved to the offline data warehouse.

[0013] Based on the offline incremental data indicators corresponding to each second preset time interval, the offline data indicators corresponding to each first preset time interval are determined; wherein, the second preset time interval is less than the first preset time interval.

[0014] Furthermore, real-time calculations are performed on the platform data to obtain real-time data metrics, including:

[0015] Based on the preset data indicator calculation logic, the platform data is calculated in real time to obtain real-time data indicators, which are then saved to the database to form a fact table.

[0016] Furthermore, for any first preset time interval, comparing the offline data indicator corresponding to the first preset time interval with the real-time data indicator generated within the first preset time interval, and determining whether there is abnormal data in the data system within the first preset time interval based on the comparison result further includes:

[0017] For any first preset time interval, calculate the index difference between the offline data index corresponding to the first preset time interval and the real-time data index generated within the first preset time interval, and obtain a comparison result containing the index difference.

[0018] Determine whether the difference in indicators exceeds a preset threshold;

[0019] If the difference in indicators exceeds a preset threshold, it is determined that there is abnormal data in the data system within the first preset time interval; if the difference in indicators does not exceed the preset threshold, it is determined that there is no abnormal data in the data system within the first preset time interval.

[0020] Furthermore, the method also includes:

[0021] Back up platform data to an offline data warehouse;

[0022] Real-time data metrics are backed up incrementally to an offline data warehouse.

[0023] Furthermore, data repair processing for abnormal data further includes:

[0024] Create a temporary table;

[0025] Retrieve the target real-time data metrics generated before the first preset time interval from the real-time data metrics that have been backed up in the offline data warehouse, and recall the target real-time data metrics to a temporary table;

[0026] Retrieve the target platform data generated within the first preset time interval from the platform data backed up in the offline data warehouse, perform data playback and calculation on the target platform data to obtain playback data indicators, and save the playback data indicators to a temporary table;

[0027] Replace the fact table in the database that contains abnormal data with a temporary table.

[0028] Furthermore, retrieving the target real-time data metrics generated before the first preset time interval from the real-time data metrics already backed up in the offline data warehouse further includes:

[0029] The target real-time data indicator is obtained from the real-time data indicators that have been backed up in the offline data warehouse and generated within a preset number of time intervals before the first preset time interval.

[0030] Furthermore, the method also includes:

[0031] If abnormal data is found, an alarm will be sent to the system management terminal.

[0032] According to another aspect of the embodiments of this application, a data availability maintenance apparatus for a data system is provided, comprising:

[0033] The data acquisition module is used to acquire platform data generated by the resource platform;

[0034] The offline calculation module is used to perform offline calculations on platform data according to a first preset time interval to obtain offline data indicators corresponding to each first preset time interval.

[0035] The real-time computing module is used to perform real-time calculations on platform data to obtain real-time data metrics.

[0036] The comparison module is used to compare the offline data indicators corresponding to any first preset time interval with the real-time data indicators generated within the first preset time interval, and to determine whether there is any abnormal data in the data system within the first preset time interval based on the comparison results.

[0037] The processing module is used to repair abnormal data if any is found.

[0038] According to another aspect of the embodiments of this application, a computing device is provided, including: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus;

[0039] The memory is used to store at least one executable instruction that causes the processor to perform an operation corresponding to a data availability maintenance method for a data system.

[0040] According to another aspect of the embodiments of this application, a computer storage medium is provided, wherein at least one executable instruction is stored in the storage medium, and the executable instruction causes a processor to perform an operation corresponding to the data availability maintenance method of the data system described above.

[0041] According to an embodiment of this application, a data availability maintenance method and apparatus for a data system is provided. This method involves acquiring platform data generated by a resource platform; performing offline calculations on the platform data according to a first preset time interval to obtain offline data indicators corresponding to each first preset time interval; performing real-time calculations on the platform data to obtain real-time data indicators; comparing the offline data indicators corresponding to any first preset time interval with the real-time data indicators generated within that first preset time interval, and determining whether abnormal data exists in the data system within that first preset time interval based on the comparison results; and performing data repair processing on the abnormal data if abnormal data is found. This application can monitor data availability in a data system using a first preset time interval as the time granularity, enabling timely detection of abnormal data. In the event of abnormal data in the data system, it can also issue alarm prompts via instant messaging tools to promptly notify users on the system management side and initiate a data repair process to complete lossless data repair, thus effectively reducing the impact of abnormal data on online business. Based on the data partitioning characteristics of OLAP databases, this application creates an offline data warehouse to store offline computational data and uses a real-time computing link to calculate real-time data metrics, thereby facilitating data comparison and determining the existence of abnormal data. Platform data and real-time data metrics are backed up to the offline data warehouse, facilitating subsequent retrieval of usable data and timely repair of abnormal data. After abnormal data is repaired using a temporary table, it can be directly replaced with the fact table containing abnormal data, thus achieving fast and convenient data repair.

[0042] The above description is merely an overview of the technical solutions of the embodiments of this application. In order to better understand the technical means of the embodiments of this application and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of this application more obvious and understandable, specific implementation methods of the embodiments of this application are described below. Attached Figure Description

[0043] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0044] Figure 1 A flowchart illustrating a data availability maintenance method for a data system according to an embodiment of this application is shown.

[0045] Figure 2a A flowchart illustrating a data availability maintenance method for a data system according to another embodiment of this application is shown;

[0046] Figure 2b A schematic diagram illustrating an offline calculation process for advertising data according to an embodiment of this application is shown;

[0047] Figure 2c A schematic diagram illustrating a real-time calculation process for advertising data according to an embodiment of this application is shown.

[0048] Figure 3 A structural block diagram of a data availability maintenance apparatus for a data system according to an embodiment of this application is shown;

[0049] Figure 4 A schematic diagram of the structure of a computing device according to an embodiment of this application is shown. Detailed Implementation

[0050] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0051] First, the terms and concepts involved in one or more embodiments of this application will be explained.

[0052] Online Analytical Processing (OLAP) databases: Partitioning and parallel processing techniques are commonly used in OLAP systems. The importance of partitioning in OLAP systems is primarily reflected in database management. For example, database loading can be achieved through partition swapping, backups can be performed by backing up partitioned tablespaces, and data deletion can be done through partitions. Regarding the performance impact of partitioning, it can significantly speed up scans of large tables (by scanning only a single partition). Furthermore, combining partitioning with parallel processing can also greatly accelerate scans of the entire table.

[0053] Data system availability: The ability of a data system to correctly handle user requests and the correctness of the data in the data system in terms of business logic.

[0054] Data repair: Using certain technical means, abnormal data in a data system is corrected to ensure that the data system meets availability requirements.

[0055] Data system availability monitoring: Through data calculation, data comparison and other technical solutions, monitor whether there is abnormal data in the data system, and notify the system administrator to handle the abnormal data through abnormal alarms and other means.

[0056] Temporary table: A type of data table used to temporarily store data. In the data repair process, the temporary table and the production environment data table have the same table structure.

[0057] Data partitioning: Data storage systems are divided into different areas according to business rules to store data with different business attributes. This is a solution used to improve data access efficiency.

[0058] Apache Flink is a real-time data processing tool. It's a streaming data engine that provides communication, fault tolerance, and data distribution for distributed computing on streaming data. Flink is an Apache scalable data analytics framework that easily performs both streaming and batch processing.

[0059] Apache DORIS is a high-performance real-time analytics database based on a massively parallel processing (MPP) architecture, renowned for its extremely fast speed and ease of use. It returns query results with sub-second response times even with massive datasets, supporting not only high-concurrency query scenarios but also high-throughput complex analysis scenarios.

[0060] Figure 1 A flowchart illustrating a data availability maintenance method for a data system according to an embodiment of this application is shown, as follows: Figure 1 As shown, the method includes the following steps:

[0061] Step S110: Obtain platform data generated by the resource platform.

[0062] A resource platform refers to a platform used to manage the resources provided by resource providers for deployment. These resources may specifically include content materials such as advertisements, images, and videos. The platform data may include advertising data, etc. In this application, a data system is used to manage and analyze the platform data generated by the resource platform, calculating data metrics to provide data support for optimizing resource deployment strategies.

[0063] In this embodiment, a big data architecture can be adopted, which calculates data indicators through two sets of logic: offline data calculation and real-time data calculation. By comparing the offline data indicators and real-time data indicators corresponding to the same first preset time interval, abnormal data can be detected in a timely manner. In the case of abnormal data, data repair can be performed automatically and in a timely manner to ensure the high availability of data in the data system.

[0064] To ensure the availability of data in the data system, after acquiring the platform data generated by the resource platform, the platform data can be saved to an offline data warehouse for convenient offline computing chain computation and retrieval.

[0065] Step S120: Perform offline calculations on the platform data according to the first preset time interval to obtain the offline data indicators corresponding to each first preset time interval.

[0066] Specifically, the first preset time interval can be pre-configured according to the actual scenario; for example, the first preset time interval can be one hour or one day. In this step, the platform data can be calculated offline according to the first preset time interval to obtain the offline data indicators corresponding to each first preset time interval. The offline data indicators can include data indicators such as application activation data and new user data. Those skilled in the art can set the offline data indicators and the real-time data indicators mentioned below according to actual needs, which are not limited here.

[0067] Step S130: Perform real-time calculations on the platform data to obtain real-time data metrics.

[0068] Real-time computing can calculate real-time data metrics at the minute level or even smaller time scales. These real-time metrics correspond to offline data metrics and can include data metrics such as application activation data and new user data. The calculated real-time data metrics can be stored in an OLAP database. Based on the data partitioning characteristics of OLAP databases, offline and real-time data metrics can be stored and managed in separate partitions for easy comparison and retrieval later.

[0069] After obtaining the platform data generated by the resource platform in step S110, the platform data can be backed up to the offline data warehouse. After obtaining the real-time data indicators in step S130, the real-time data indicators obtained by the real-time computing link can be backed up to the offline data warehouse in an incremental manner to facilitate data retrieval during data recovery.

[0070] Step S140: For any first preset time interval, compare the offline data indicators corresponding to the first preset time interval with the real-time data indicators generated within the first preset time interval, and determine whether there is abnormal data in the data system within the first preset time interval based on the comparison results.

[0071] Taking a first preset time interval of one day as an example, the real-time data indicators generated each day are compared with the offline data indicators for that day. Generally, a threshold can be preset according to the actual application scenario. If the difference between the real-time data indicators generated within the first preset time interval and the corresponding offline data indicators exceeds the preset threshold, it is determined that there is abnormal data in the data system within the first preset time interval; if the difference between the real-time data indicators generated within the first preset time interval and the corresponding offline data indicators does not exceed the preset threshold, it is determined that there is no abnormal data in the data system within the first preset time interval.

[0072] Step S150: If abnormal data exists, perform data repair processing on the abnormal data.

[0073] In this step, if the comparison results indicate that abnormal data exists in the data system within the first preset time interval, the abnormal data needs to be repaired to reduce the impact of abnormal data on online business. Specifically, the platform data and real-time data indicators that have been backed up in the offline data warehouse can be recalled, and then data playback and calculation can be performed to determine the correct data indicators and complete the data repair.

[0074] According to the data availability maintenance method for a data system provided in this application embodiment, the method involves: acquiring platform data generated by a resource platform; performing offline calculations on the platform data according to a first preset time interval to obtain offline data indicators corresponding to each first preset time interval; performing real-time calculations on the platform data to obtain real-time data indicators; comparing the offline data indicators corresponding to any first preset time interval with the real-time data indicators generated within that first preset time interval, and determining whether there is abnormal data in the data system within that first preset time interval based on the comparison results; and performing data repair processing on the abnormal data if abnormal data is found. This method can monitor the data availability of the data system at a first preset time interval as the time granularity, enabling timely detection of abnormal data. In the event of abnormal data in the data system, it can also issue alarm prompts through instant messaging tools to promptly notify users on the system management side and initiate a data repair process to complete lossless data repair, thereby effectively reducing the impact of abnormal data on online business.

[0075] Figure 2a A flowchart illustrating a data availability maintenance method for a data system according to another embodiment of this application is shown, as follows: Figure 2a As shown, the method includes the following steps:

[0076] Step S210: Obtain platform data generated by the resource platform.

[0077] The data system can connect to the resource platform through a pre-defined interface to obtain platform data generated by the platform in real time. Platform data may include advertising data, etc.

[0078] Step S220: Save the platform data to the offline data warehouse. According to the preset data indicator calculation logic, perform offline calculations on the platform data in the offline data warehouse according to the second preset time interval to obtain the offline incremental data indicators corresponding to each second preset time interval, and save the offline incremental data indicators corresponding to each second preset time interval to the offline data warehouse.

[0079] In this embodiment, an offline data warehouse is built to store platform data so that the offline computing link can perform offline calculations based on the stored platform data. The offline data warehouse is also used to store offline data metrics.

[0080] Specifically, taking platform data, including advertising data, as an example, offline incremental data metrics can include incremental data corresponding to metrics such as application activation data, new user data, platform revenue data, and user retention data. In this step, the corresponding data metric calculation logic can be pre-set according to the category of platform data. For example, the corresponding data metric calculation logic can be set for platform data categories such as advertising, images, and videos.

[0081] Furthermore, the platform data in the offline data warehouse is calculated offline according to the second preset time interval to obtain the offline incremental data indicators corresponding to each second preset time interval, and the offline incremental data indicators corresponding to each second preset time interval are saved to the offline data warehouse; wherein, the second preset time interval is less than the first preset time interval.

[0082] For example, if the first preset time interval is one day and the second preset time interval is one hour, then for the platform data within one day, offline calculations are performed on the platform data in the offline data warehouse at hourly intervals to obtain the corresponding offline incremental data metrics for each hour. These hourly offline incremental data metrics are then saved to the offline data warehouse. For example, for advertising data, offline incremental data metrics may include: incremental activation data of the application account corresponding to the ad, new user data, user spending data for the ad, and user retention data on the ad page, etc. Figure 2b A schematic diagram illustrating an offline calculation process for advertising data according to an embodiment of this application is shown, such as... Figure 2b As shown, offline incremental data metrics are calculated hourly (e.g.) Figure 2b 2022020201 and 2022020202 represent the first and second hours of February 2, 2022, respectively. Furthermore, the offline incremental data indicators for each hour can be added to the offline full data indicators for the previous hour to form the offline full data indicators for that hour.

[0083] Step S230: Determine the offline data indicators corresponding to each first preset time interval based on the offline incremental data indicators corresponding to each second preset time interval.

[0084] Specifically, the offline data metrics for each day are determined by calculating the corresponding offline incremental data metrics for each hour.

[0085] Step S240: Based on the preset data indicator calculation logic, perform real-time calculation on the platform data to obtain real-time data indicators, and save the real-time data indicators to the database to form a fact table.

[0086] In this embodiment, in order to support large-scale data processing, the real-time data processing tool used in the real-time computing link can be Apache Flink. The real-time data metrics correspond to the offline data metrics. Depending on the platform data, daily application account activation data, new user data, platform revenue data, user retention data and other metrics can be calculated. Figure 2c A schematic diagram illustrating a real-time calculation process for advertising data according to an embodiment of this application is shown, such as... Figure 2c As shown, real-time calculations are performed at the minute level using Apache Flink's minute-level calculation window to obtain real-time data metrics, which are then saved to the Apache DORIS real-time analysis database of the OLAP database to form a fact table (i.e., a large wide table).

[0087] Step S250: For any first preset time interval, calculate the difference between the offline data indicator corresponding to the first preset time interval and the real-time data indicator generated within the first preset time interval, and obtain a comparison result containing the difference.

[0088] For example, with a first preset time interval of one day, the corresponding offline data metrics for each day are compared with the real-time data metrics calculated within that day. The difference between the two metrics is calculated, and a comparison result including the difference in metrics is obtained. Specifically, the corresponding offline data metrics for a day can be retrieved from the offline data warehouse, and the real-time data metrics calculated within that day can be obtained from the OLAP database. The difference between the two metrics is calculated, and a comparison result including the difference in metrics is obtained. For example, for advertising data, the user addition data in the corresponding offline data metrics for a day is retrieved from the offline data warehouse, and the corresponding user addition data in the real-time data metrics calculated within that day is obtained from the OLAP database. The difference in metrics for user addition data is calculated.

[0089] Step S260: Determine whether the difference in indicators exceeds the preset threshold; if yes, proceed to step S270; if no, proceed to step S280.

[0090] In this embodiment, depending on the platform data, the indicator data may include: incremental application account activation data, new user data, platform transaction data, user retention data, etc. For each data indicator, a threshold is preset to determine whether the difference between the calculation results of the offline computing link and the real-time computing link for any data indicator exceeds its corresponding preset threshold within the same first preset time interval.

[0091] Step S270: Determine that there is abnormal data in the data system within the first preset time interval, send an alarm notification to the system management terminal, and perform data repair processing on the abnormal data.

[0092] If the difference in indicators exceeds a preset threshold, it is determined that abnormal data exists in the data system within the first preset time interval. To reduce the impact of abnormal data on online business, an alarm notification can be sent to the system management terminal in a timely manner. Specifically, alarm notifications can be sent to the system management terminal through various methods, such as sending SMS messages or emails. In the event of abnormal data, timely data repair is also necessary to ensure the normal operation of online business.

[0093] It should be noted that, in order to facilitate the timely retrieval of usable data during data repair, the platform data and real-time data metrics can be backed up first. The platform data should be backed up to the offline data warehouse, and the real-time data metrics should be backed up to the offline data warehouse incrementally.

[0094] In one alternative implementation, the process of performing data repair on anomalous data may include the following steps 1-4:

[0095] Step 1: Create a temporary table.

[0096] Step 2: Retrieve the target real-time data metrics generated before the first preset time interval from the real-time data metrics that have been backed up in the offline data warehouse, and recall the target real-time data metrics to a temporary table.

[0097] In an optional implementation, step 2 further includes: obtaining, from the real-time data metrics backed up in the offline data warehouse, real-time data metrics generated within a preset number of first preset time intervals prior to the first preset time interval as target real-time data metrics.

[0098] Step 3: Retrieve the target platform data generated within the first preset time interval from the platform data backed up in the offline data warehouse, and perform data playback and calculation on the target platform data to obtain playback data indicators. Save the playback data indicators to a temporary table. Taking a preset quantity of 2 as an example, if it is found that there is abnormal data in the data system on February 3, 2022, the platform data generated by the resource platform on that day can be retrieved from the offline data warehouse, and the real-time indicator data corresponding to February 1 and February 2, 2022 can be retrieved. By playing back the platform data on February 3, 2022, and based on the real-time indicator data on February 1 and February 2, 2022, the data indicators of the played-back platform data are calculated to obtain the correct data indicators for February 3, 2022.

[0099] Step 4: Replace the fact table in the database that contains abnormal data with the temporary table.

[0100] Specifically, the temporary table can be easily replaced by renaming its suffix to match that of the fact table.

[0101] Step S280: Determine that there is no abnormal data in the data system within the first preset time interval.

[0102] If the difference in the indicators does not exceed the preset threshold, it is determined that there is no abnormal data in the data system within the first preset time interval.

[0103] According to the data availability maintenance method for a data system provided in this application embodiment, the method involves: acquiring platform data generated by a resource platform; performing offline calculations on the platform data according to a first preset time interval to obtain offline data indicators corresponding to each first preset time interval; performing real-time calculations on the platform data to obtain real-time data indicators; comparing the offline data indicators corresponding to any first preset time interval with the real-time data indicators generated within that first preset time interval, and determining whether there is abnormal data in the data system within that first preset time interval based on the comparison results; and performing data repair processing on the abnormal data if abnormal data is found. This method can monitor the data availability of the data system at a first preset time interval as the time granularity, enabling timely detection of abnormal data. In the event of abnormal data in the data system, it can also issue alarm prompts through instant messaging tools to promptly notify users on the system management side and initiate a data repair process to complete lossless data repair, thereby effectively reducing the impact of abnormal data on online business. This method leverages the data partitioning characteristics of OLAP databases. It creates an offline data warehouse to store offline computational data and uses a real-time computing link to calculate real-time data metrics, facilitating data comparison and identifying any abnormal data. Platform data and real-time data metrics are backed up to the offline data warehouse for easy retrieval of usable data and timely repair of abnormal data. After repairing abnormal data using a temporary table, it can be directly replaced with the fact table containing the abnormal data, thus achieving rapid and convenient data repair.

[0104] Figure 3 A structural block diagram of a data availability maintenance device for a data system according to an embodiment of this application is shown, such as... Figure 3 As shown, the device includes: a data acquisition module 310, an offline computing module 320, a real-time computing module 330, a comparison module 340, and a processing module 350.

[0105] The data acquisition module 310 is used to acquire platform data generated by the resource platform.

[0106] The offline calculation module 320 is used to perform offline calculations on platform data according to a first preset time interval to obtain offline data indicators corresponding to each first preset time interval.

[0107] In one optional implementation, the offline computing module 320 is further configured to: save platform data to an offline data warehouse; perform offline calculations on the platform data in the offline data warehouse according to a preset data indicator calculation logic and at a second preset time interval to obtain offline incremental data indicators corresponding to each second preset time interval, and save the offline incremental data indicators corresponding to each second preset time interval to the offline data warehouse; determine the offline data indicators corresponding to each first preset time interval based on the offline incremental data indicators corresponding to each second preset time interval; wherein the second preset time interval is less than the first preset time interval.

[0108] The real-time computing module 330 is used to perform real-time calculations on platform data to obtain real-time data metrics.

[0109] In an optional implementation, the real-time calculation module 330 is further configured to: perform real-time calculations on platform data according to preset data indicator calculation logic, obtain real-time data indicators, and save the real-time data indicators to the database to form a fact table.

[0110] The comparison module 340 is used to compare the offline data indicators corresponding to any first preset time interval with the real-time data indicators generated within the first preset time interval, and to determine whether there is abnormal data in the data system within the first preset time interval based on the comparison results.

[0111] In an optional implementation, the comparison module 340 is further configured to: for any first preset time interval, calculate the index difference between the offline data index corresponding to the first preset time interval and the real-time data index generated within the first preset time interval, and obtain a comparison result including the index difference; determine whether the index difference exceeds a preset threshold; if the index difference exceeds the preset threshold, determine that there is abnormal data in the data system within the first preset time interval; if the index difference does not exceed the preset threshold, determine that there is no abnormal data in the data system within the first preset time interval.

[0112] The processing module 350 is used to perform data repair processing on abnormal data if abnormal data exists.

[0113] In one alternative implementation, the processing module 350 is further configured to: back up platform data to an offline data warehouse; and back up real-time data metrics to the offline data warehouse incrementally.

[0114] In an optional implementation, the processing module 350 is further configured to: create a temporary table; obtain target real-time data indicators generated before the first preset time interval from the real-time data indicators backed up in the offline data warehouse, and recall the target real-time data indicators to the temporary table; obtain target platform data generated within the first preset time interval from the platform data backed up in the offline data warehouse, and perform data playback and calculation on the target platform data to obtain playback data indicators, and save the playback data indicators to the temporary table; and replace the fact table in the database containing abnormal data based on the temporary table.

[0115] In an optional implementation, the processing module 350 is further configured to: obtain, from the real-time data metrics backed up in the offline data warehouse, real-time data metrics generated within a preset number of first preset time intervals prior to the first preset time interval as target real-time data metrics.

[0116] In an optional implementation, the processing module 350 is further configured to: send an alarm notification to the system management terminal if abnormal data is found.

[0117] The descriptions of the above modules refer to the corresponding descriptions in the method embodiments, and will not be repeated here.

[0118] According to an embodiment of this application, a data availability maintenance device for a data system acquires platform data generated by a resource platform; performs offline calculations on the platform data according to a first preset time interval to obtain offline data indicators corresponding to each first preset time interval; performs real-time calculations on the platform data to obtain real-time data indicators; for any first preset time interval, compares the offline data indicators corresponding to the first preset time interval with the real-time data indicators generated within the first preset time interval, and determines whether there is abnormal data in the data system within the first preset time interval based on the comparison result; if abnormal data exists, performs data repair processing on the abnormal data. This device can monitor the data availability of the data system at a first preset time interval as the time granularity, realizing the timely detection of abnormal data; in the case of abnormal data in the data system, it can also issue alarm prompts through instant messaging tools to promptly notify users on the system management side, and can initiate a data repair process to complete lossless data repair, realizing effective repair of abnormal data, thereby effectively reducing the impact of abnormal data in the data system on online business.

[0119] This application also provides a non-volatile computer storage medium storing at least one executable instruction that can execute the data availability maintenance method of the data system in any of the above method embodiments.

[0120] Figure 4The diagram shows a structural schematic of a computing device according to an embodiment of the present application. The specific embodiments of the present application do not limit the specific implementation of the computing device.

[0121] like Figure 4 As shown, the computing device may include: a processor 402, a communications interface 404, a memory 406, and a communications bus 408.

[0122] in:

[0123] The processor 402, communication interface 404, and memory 406 communicate with each other via communication bus 408.

[0124] Communication interface 404 is used to communicate with other network elements such as clients or other servers.

[0125] The processor 402 is used to execute program 410, specifically to perform the relevant steps in the above-described data availability maintenance method embodiment for the data system.

[0126] Specifically, program 410 may include program code that includes computer operation instructions.

[0127] Processor 402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The computing device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0128] Memory 406 is used to store program 410. Memory 406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0129] Specifically, program 410 can be used to cause processor 402 to execute the data availability maintenance method of the data system in any of the above method embodiments. The specific implementation of each step in program 410 can be found in the corresponding descriptions of the steps and units in the above data system data availability maintenance embodiments, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0130] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of this application are not directed to any particular programming language. It should be understood that the contents of the embodiments of this application described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best implementation of the embodiments of this application.

[0131] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0132] Similarly, it should be understood that, in order to simplify this disclosure and aid in understanding one or more of the various inventive aspects, in the foregoing description of exemplary embodiments of the present application, various features of the present application embodiments are sometimes grouped together into a single embodiment, figure, or description thereof. However, this approach to disclosure should not be construed as reflecting an intention that the claimed embodiments of the present application require more features than expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the present application.

[0133] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0134] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are meant to be within the scope of the embodiments of this application and form different embodiments. For example, in the following claims, any one of the claimed embodiments can be used in any combination.

[0135] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components according to the embodiments of this application. The embodiments of this application can also be implemented as device or apparatus programs (e.g., computer programs and computer program products) for performing part or all of the methods described herein. Such programs implementing the embodiments of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0136] It should be noted that the above embodiments are illustrative of the embodiments of this application and not limiting of the embodiments of this application, and those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. Embodiments of this application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

Claims

1. A method for maintaining data availability in a data system, comprising: Acquire platform data generated by the resource platform and back up the platform data to an offline data warehouse; The platform data is calculated offline according to a first preset time interval to obtain offline data indicators corresponding to each first preset time interval. The platform data is calculated in real time to obtain real-time data metrics, and the real-time data metrics are backed up to the offline data warehouse in an incremental manner. For any first preset time interval, the offline data indicator corresponding to the first preset time interval is compared with the real-time data indicator generated within the first preset time interval, and the data system is judged to have any abnormal data within the first preset time interval based on the comparison result; If abnormal data is found, data repair processing will be performed on the abnormal data; The data repair process for the abnormal data further includes: Create a temporary table, retrieve the target real-time data indicator generated before the first preset time interval from the real-time data indicators that have been backed up in the offline data warehouse, and recall the target real-time data indicator to the temporary table; The target platform data generated within the first preset time interval is obtained from the platform data backed up in the offline data warehouse, and the target platform data is replayed and calculated to obtain replay data indicators. The replay data indicators are then saved to the temporary table. Replace the fact table in the database that contains abnormal data with the temporary table.

2. The method according to claim 1, wherein, The step of performing offline calculations on the platform data according to a first preset time interval to obtain offline data indicators corresponding to each first preset time interval further includes: The platform data is saved to an offline data warehouse; Based on the preset data indicator calculation logic, the platform data in the offline data warehouse is calculated offline according to the second preset time interval to obtain the offline incremental data indicators corresponding to each second preset time interval, and the offline incremental data indicators corresponding to each second preset time interval are saved to the offline data warehouse. Based on the offline incremental data indicators corresponding to each of the second preset time intervals, the offline data indicators corresponding to each of the first preset time intervals are determined; wherein, the second preset time interval is less than the first preset time interval.

3. The method according to claim 1, wherein, The step of performing real-time calculations on the platform data to obtain real-time data metrics further includes: Based on the preset data indicator calculation logic, the platform data is calculated in real time to obtain the real-time data indicators, and the real-time data indicators are saved to the database to form a fact table.

4. The method according to any one of claims 1-3, wherein, The step of comparing the offline data indicator corresponding to any first preset time interval with the real-time data indicator generated within the first preset time interval, and determining whether there is abnormal data in the data system within the first preset time interval based on the comparison result, further includes: For any first preset time interval, calculate the index difference between the offline data index corresponding to the first preset time interval and the real-time data index generated within the first preset time interval, and obtain a comparison result containing the index difference. Determine whether the difference in the indicators exceeds a preset threshold; If the difference in the indicators exceeds the preset threshold, it is determined that there is abnormal data in the data system within the first preset time interval; if the difference in the indicators does not exceed the preset threshold, it is determined that there is no abnormal data in the data system within the first preset time interval.

5. The method according to claim 1, wherein, The step of obtaining the target real-time data indicator generated before the first preset time interval from the real-time data indicators that have been backed up in the offline data warehouse further includes: The target real-time data indicator is obtained from the real-time data indicators that have been backed up in the offline data warehouse and generated within a preset number of time intervals before the first preset time interval.

6. The method according to any one of claims 1-3, wherein, The method further includes: If abnormal data is found, an alarm will be sent to the system management terminal.

7. A data availability maintenance device for a data system, comprising: The data acquisition module is used to acquire platform data generated by the resource platform; The offline calculation module is used to perform offline calculations on the platform data according to a first preset time interval to obtain offline data indicators corresponding to each first preset time interval. The real-time computing module is used to perform real-time calculations on the platform data to obtain real-time data metrics; The comparison module is used to compare the offline data indicators corresponding to any first preset time interval with the real-time data indicators generated within the first preset time interval, and determine whether there is abnormal data in the data system within the first preset time interval based on the comparison result. The processing module is used to perform data repair processing on the abnormal data if abnormal data exists. The processing module is further configured to: back up the platform data to an offline data warehouse; and back up the real-time data metrics to the offline data warehouse incrementally. The processing module is further used for: Create a temporary table, retrieve the target real-time data indicator generated before the first preset time interval from the real-time data indicators that have been backed up in the offline data warehouse, and recall the target real-time data indicator to the temporary table; The target platform data generated within the first preset time interval is obtained from the platform data backed up in the offline data warehouse, and the target platform data is replayed and calculated to obtain replay data indicators. The replay data indicators are then saved to the temporary table. Replace the fact table in the database that contains abnormal data with the temporary table.

8. A computing device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform an operation corresponding to the data availability maintenance method of a data system as described in any one of claims 1-6.

9. A computer storage medium storing at least one executable instruction that causes a processor to perform an operation corresponding to a data availability maintenance method for a data system as claimed in any one of claims 1-6.

10. A computer program product comprising at least one executable instruction that causes a processor to perform an operation corresponding to a data availability maintenance method for a data system as claimed in any one of claims 1-6.

Images