Data integrity automatic monitoring method, device, equipment and medium

Abstract

The present application discloses a method, device, equipment and medium for automatically monitoring data integrity, the method comprising: when determining a data integration task to be executed, obtaining a data offset of business data related to the data integration task; determining whether the business data related to the data integration task is complete according to the data offset; when determining that the business data related to the data integration task is complete, performing data integration processing on the business data related to the integration task, so as to write the data integration processing result into a subject area corresponding to the data integration task. The embodiment of the present application determines whether there is an error in the process of copying business data from a business system to a data lake through a data offset, ensures the integrity of data obtained from a data lake during the data integration phase, and thus improves the quality of data analysis.

Description

Technical Field

[0001] The present application relates to the technical field of industrial big data, and in particular to methods, devices, equipment and media for automatically monitoring data integrity. Background Art

[0002] With the continuous updating and iteration of big data technology, data management and analysis platforms have developed rapidly. These data management and analysis platforms can be used to achieve comprehensive analysis across businesses and systems. For example, based on Hadoop, data systems can be planned and designed according to the actual business needs of enterprises. Different data analysis architectures and framework components are used to solve practical problems according to different specific needs. However, there are data quality issues when importing data into the Hadoop platform through data collection tools. Summary of the invention

[0003] In view of the above-mentioned defects or deficiencies in the prior art, it is desirable to provide a method, device, equipment and medium for automatic monitoring of data integrity to improve the quality of data analysis.

[0004] In a first aspect, an embodiment of the present application provides a method for automatically monitoring data integrity, the method comprising:

[0005] When determining the data integration task to be executed, obtaining the data offset of the business data related to the data integration task;

[0006] Determine whether the business data related to the data integration task is complete based on the data offset;

[0007] When it is determined that the business data related to the data integration task is complete, data integration processing is performed on the business data related to the integration task to write the data integration processing result into the subject area corresponding to the data integration task.

[0008] In a second aspect, an embodiment of the present application provides a data integrity automatic monitoring device, the device comprising:

[0009] An offset acquisition unit, used to acquire a data offset of business data related to the data integration task when determining the data integration task to be executed;

[0010] An integrity determination unit, used to determine whether the business data related to the data integration task is complete according to the data offset;

[0011] The data integration unit is used to perform data integration processing on the business data related to the integration task when it is determined that the business data related to the data integration task is complete, so as to write the data integration processing result into the subject area corresponding to the integration task.

[0012] In a third aspect, an embodiment of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to implement the method described in the embodiment of the present application when executing the program.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and the computer program is used to implement the method described in the embodiment of the present application.

[0014] The embodiment of the present application provides a method, apparatus, device and medium for automatically monitoring data integrity. The method obtains the data offset of the business data related to the data integration task when determining the data integration task to be executed; then determines whether the business data related to the data integration task is complete based on the data offset; when determining that the business data related to the data integration task is complete, performs data integration processing on the business data related to the integration task to write the data integration processing result to the subject area corresponding to the data integration task. The embodiment of the present application uses the data offset to determine whether there is an error in the process of copying business data from the business system to the data lake, ensures the integrity of the data obtained from the data lake during the data integration stage, and thus improves the quality of data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0016] Figure 1 A schematic diagram of an application scenario of the data integrity automatic monitoring method provided in an embodiment of the present application is shown;

[0017] Figure 2 A schematic diagram of a process flow of a method for automatically monitoring data integrity proposed in an embodiment of the present application is shown;

[0018] Figure 3 A schematic diagram showing a flow chart of another method for automatically monitoring data integrity proposed in an embodiment of the present application;

[0019] Figure 4 A schematic diagram showing a flow chart of another method for automatically monitoring data integrity proposed in an embodiment of the present application;

[0020] Figure 5 A schematic diagram of a process for creating an associated data source table related to a data integration task proposed in an embodiment of the present application is shown;

[0021] Figure 6 A schematic diagram of the process flow of the task monitoring method proposed in the embodiment of the present application is shown;

[0022] Figure 7A reference schematic diagram of an associated data source table provided in an embodiment of the present application is shown;

[0023] Figure 8 A flowchart of a task flow monitoring process implemented by the Pentaho PDI tool provided in an embodiment of the present application is shown;

[0024] Fig. 9 A reference schematic diagram of a corresponding relationship reference table provided in an embodiment of the present application is shown;

[0025] Fig.10 A reference schematic diagram of a task scheduling record table provided in an embodiment of the present application is shown;

[0026] Fig.11 A reference schematic diagram of changing a task scheduling record table provided in an embodiment of the present application when data is incomplete is shown;

[0027] Fig.12 A reference schematic diagram of a task scheduling record table change under the condition of complete data provided by an embodiment of the present application is shown;

[0028] Fig.13 A schematic diagram of the structure of the automatic data integrity monitoring device proposed in an embodiment of the present application is shown;

[0029] Fig.14 A schematic diagram of the structure of another data integrity automatic monitoring device proposed in an embodiment of the present application is shown;

[0030] Fig.15 A schematic diagram of the structure of a computer system suitable for implementing a server of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0031] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant disclosure, rather than to limit the disclosure. It should also be noted that, for ease of description, only the parts related to the disclosure are shown in the accompanying drawings.

[0032] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0033] Please refer to Figure 1 , Figure 1 A schematic diagram of an application scenario of the method for automatically monitoring data integrity provided in an embodiment of the present application is shown.

[0034] like Figure 1As shown, the big data analysis platform includes a business system 101, a data analysis system 102, etc. The business system and the data analysis system can be independent physical servers, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0035] Figure 1 The business system 101 is one or more servers that provide business data. The business database provided by the business system is a relational database. In order to provide decision-making suggestions for the operation and development of the enterprise, it is necessary to integrate and analyze various business data scattered inside and outside the enterprise, and then obtain certain specific subject requirements for decision analysis.

[0036] The data analysis platform 102 can process data across databases or platforms to provide business analysis data for enterprises, and visualize the analysis results to managers. The data analysis platform 102 can use data warehouse, data lake technology, etc. For example, data analysis based on the Hadoop system involves cross-database and cross-platform data integration. The analysis database provided by the data analysis platform is a non-relational database.

[0037] The data required for the application of the above-mentioned data analysis platform 102 is usually obtained by integrating and processing source data (ie, original business data) provided by multiple business systems 101 .

[0038] In order to reduce the performance impact on the business system 101, the business data in the business system 101 is first copied to the data lake layer on the data analysis platform 102, and then the data is extracted from the data lake layer for integration and written into the data subject layer.

[0039] However, in the data analysis platform 102, the process of data replication and data extraction may have such problems: Assuming that the data integration tasks Job1 and Jobi that rely on the business data table S11 have started the data integration task for the [t1, t2] time range in the scheduling task, but the business data of the [t1, t2] time range corresponding to the business data table S11 may not be fully synchronized to the data lake for some reason, which will cause the business data in the [t1, t2] time period corresponding to the data topic 1 and topic i to be missing errors. However, this error is not easy to be discovered actively, and the application end uses the missing business data for analysis, which may lead to inaccurate analysis results.

[0040] In order to solve the above problems, the present application proposes a technical solution for an automatic data integrity monitoring method, which can effectively save query time and improve the query efficiency of product history data.

[0041] Please refer to the following Figure 2 , Figure 2 FIG. 1 is a flow chart showing a method for automatically monitoring data integrity according to an embodiment of the present application. Figure 2 As shown, the method can be performed by an automatic data integrity monitoring device, and the automatic data integrity monitoring device can be configured in a terminal device or a server. The method includes:

[0042] Step 201, when determining a data integration task to be executed, obtaining a data offset of business data related to the data integration task;

[0043] Step 202, determining whether the business data related to the data integration task is complete according to the data offset;

[0044] Step 203: When it is determined that the business data related to the data integration task is complete, data integration processing is performed on the business data related to the integration task to write the data integration processing result into the subject area corresponding to the integration task.

[0045] In the above steps, the data offset refers to the data item value of the deviation when the business data stored in the business system is copied to the analysis database. For example, the business data table S11 includes 10 business data items, and the business data corresponding to the 10 business data items are collected from 7:00-8:00 am on October 1, 2019. These business data are copied to the analysis database, and a business data table ods.S11 corresponding to the business data table is maintained in the analysis database. The ods.S11 has corresponding records according to the business data items. When the data offset is -3, it means that 3 data records in the data table are lost during the data copying process.

[0046] When determining the data integration task Job to be executed, the data offset of the business data related to the data integration task can be obtained. The data offset of the business data can be used to determine whether the business data in the data synchronization phase is completed. When it is complete, the data integration task is executed to ensure the correctness of the result of the data integration task. The integrated data provided to the subject is complete, thereby ensuring the accuracy of the analysis results when the application side uses the business data of the subject area to analyze.

[0047] The above-mentioned subject area, i.e., subject domain, refers to a data set constructed in a data mart according to a department or a special analysis purpose subject. The boundary of the data set is determined by at least one subject and the business relationship in the enterprise. The subject can be determined according to the analysis requirements. For example, a subject constructed for product defects stores data used to analyze product defects in the subject area corresponding to the subject.

[0048] The data offset of the business data related to the data integration task can be pre-stored in a data table, and the data offset can be obtained by calling the data table, thereby saving the time required for business data comparison and improving the efficiency of data integration.

[0049] Based on the above embodiments, the present application also proposes a method for automatically monitoring data integrity. Figure 3 , Figure 3 FIG. 2 is a flow chart of another method for automatically monitoring data integrity proposed in an embodiment of the present application. Figure 2 As shown, the method can be performed by an automatic data integrity monitoring device, and the automatic data integrity monitoring device can be configured in a terminal device or a server. The method includes:

[0050] Step 301, when determining a data integration task to be executed, obtaining a data offset of business data related to the data integration task;

[0051] Step 302, determining whether the business data related to the data integration task is complete according to the data offset;

[0052] Step 303, when it is determined that the business data related to the data integration task is complete, data integration processing is performed on the business data related to the integration task to write the data integration processing result into the subject area corresponding to the integration task;

[0053] Step 304, when it is determined that the business data related to the data integration task to be executed is incomplete, extracting the incomplete business data contained in the associated data source table;

[0054] Step 305, generating key information related to the data integration task based on the incomplete business data;

[0055] Step 306: Use the key information related to the data integration task as identification information of the data integration task.

[0056] In the above steps, the integrity of the business data related to the data integration task is judged, and the data integration task is executed when the data is complete. When the data is incomplete, the incomplete business data is monitored as abnormal information and notified to the relevant person in charge. The notification method includes but is not limited to email push, instant messaging, etc. Instant messaging, such as WeChat, SMS, office software, etc.

[0057] The above method provided in the embodiment of the present application can realize automatic monitoring of the integrated task flow, save task processing time, and improve the processing efficiency of the integrated tasks.

[0058] Based on the above embodiments, the present application also proposes a method for automatically monitoring data integrity. Figure 4 , Figure 4 FIG. 2 is a flow chart of another method for automatically monitoring data integrity proposed in an embodiment of the present application. Figure 4 As shown, the method can be performed by an automatic data integrity monitoring device, and the automatic data integrity monitoring device can be configured in a terminal device or a server. The method includes:

[0059] Step 401, when determining the data integration task to be executed, obtaining a pre-created associated data source table related to the data integration task;

[0060] Step 402, searching the associated data source table according to the task identifier of the data integration task to obtain the data offset of the business data related to the data integration task;

[0061] Step 403, cumulatively calculating the data offsets corresponding to the business data related to the data integration task, and obtaining the total amount of data offsets related to the data integration task;

[0062] Step 404, if the total amount of data offset is equal to the first value, it means that the business data related to the data integration task is complete;

[0063] Step 405, if the total amount of data offset is not equal to the first value, it means that the business data related to the data integration task is not complete;

[0064] Step 406: When it is determined that the business data related to the data integration task is complete, data integration processing is performed on the business data related to the integration task to write the data integration processing result into the subject area corresponding to the integration task.

[0065] Step 407, when it is determined that the business data related to the data integration task is incomplete, extract the incomplete business data contained in the associated data source table; generate key information related to the data integration task based on the incomplete business data; and use the key information related to the data integration task as identification information of the data integration task.

[0066] Based on the above embodiments, the embodiments of the present application can ensure that the business data used in the execution of the data integration task is complete by pre-creating an associated data source table.

[0067] For example, the associated data source table can be Figure 7 As shown, SourceTableName is the business data table stored in the business system, and TableName is the business data table created when the business data stored in the business system is copied to the data lake layer. By comparing these two business data tables, the data offset Offset of the business data can be determined.

[0068] After obtaining the data offset, the data offset can be compared with a preset threshold to determine whether the business data corresponding to the two tables are complete. The preset threshold can be a first value, for example, the first value is 0.

[0069] When Offset is equal to 0, it means that the business data replicated in the data lake layer is the same as the business data stored in the business system.

[0070] When Offset is not equal to 0, it means that the business data copied in the data lake layer is different from the business data stored in the business system.

[0071] When the data are the same, the data integration task is executed and the integration result is written into the subject area corresponding to the integration task. The application can obtain the business data it needs for data analysis through the message subscription mechanism.

[0072] If they are not the same, the incomplete business data is used to generate a key information, and then the identification information is used as the identification information of the data integration task. When scheduling the data integration task, the scheduled data integration task can be monitored in real time by maintaining a task scheduling record table.

[0073] The above task scheduling record table can be Fig.10As shown, JobName is the name of the data integration task, BatchID is the number of times the current task is scheduled, and 1 is added to the previous BatchID each time; StartTime is the starting value of the current scheduled business time, which is passed to the variable V$START_TIME in the task flow; EndTime is the end value of the current scheduled business time, which is passed to the variable V$END_TIME in the task flow; Status is the task status indicator, 0 means not executed, 1 means successful execution, and -1 means failed execution or being executed. Comment means if the task fails to execute, write the reason for the task failure; Increment means the time range of each scheduled business, that is, EndTime-StartTime, in min. InsertTime means recording Insert or Update time.

[0074] The above identification information can be a field information in the task scheduling record table, such as the Comment field, which can store key information generated in a string format. Fig.12 As shown, Fig.12 The figure shows the result of updating the task scheduling record table when the business data is incomplete. The status indicator corresponding to the data integration task Job1 and BatchID=212 is -1, and ods.S11 2019-10-01 07:00~2019-10-01 08:00 GAP-10 is written in its corresponding Comment field.

[0075] Then, through the automatic prompt method of the present application, the Comment field in the task scheduling record table corresponding to Job1 and BatchID=212 is sent as the content of an email or a text message to the relevant business person in charge.

[0076] The embodiment of the present application can effectively improve the execution efficiency of the data integration task through the above method.

[0077] Before the application proposes automatic monitoring of task flow, the embodiment of the application also actively obtains the data offset of business data after data synchronization. The embodiment of the application proposes to monitor the data offset of business data by creating an associated data source table. Please refer to Figure 5 , Figure 5 The flowchart of the method for creating an associated data source table related to a data integration task proposed in the embodiment of the present application is shown. Figure 5 As shown, the method includes:

[0078] Step 501, obtaining a first business data table stored in the business system and a second business data table stored in the analysis database. The second business data table is generated when the business data corresponding to the first business data table stored in the business system is copied to the analysis database;

[0079] Step 502, calculating the data offset corresponding to the business data table based on the first business data table and the second business data table;

[0080] Step 503: Write the data offset corresponding to the business data table into the associated data source table.

[0081] In the above steps, the first business data table is a record of the statistics of the business data stored in the business system. The second business data table is generated by copying the business data in the business system to the analysis database, where the analysis database can be a data lake. For example, when the data lake receives the business data copied from the business system, the second business data table is generated.

[0082] Copying the business data corresponding to the first business data table stored in the business system to the analysis database includes copying the business data corresponding to the first business data table in the business system to the second business data table in the analysis database through OGG+Kafka, which includes the following steps:

[0083] Extracting incremental change data of business data corresponding to the first business data table in the business system;

[0084] Write incremental change data to local files;

[0085] Read incremental change data from local files and push them to the Kafka message queue;

[0086] Get the business data stored in the Kafka message queue and copy it to the second business data table in the analysis database.

[0087] For business data tables with large amounts of business data, the message subscription mechanism using OGG+Kafka can save synchronization time more effectively than the periodic offline data collection method and ensure that business display results are more timely. That is, it can trigger business message replication when business data changes or arrives, thereby quickly realizing data synchronization and achieving cross-platform data synchronization at the second level.

[0088] The above analysis database is a distributed database such as KUDU database and Hbase database, which can analyze the data faster and in a timely manner when the data is updated.

[0089] When the analysis database is real-time, when the business data corresponding to the first business data table stored in the business system changes partially, copying the business data corresponding to the first business data table stored in the business system to the analysis database includes:

[0090] The business data with local changes is copied to the corresponding position in the second business data table.

[0091] Compared with the Hive database, when a data table error is found, the entire table is deleted and a new table is created to cover the storage location of the original table. In the embodiments provided by the present application, the above-mentioned distributed database can be used to improve the operation efficiency of the data table and make the operation more concise. For example, if local data in the full table changes, only the row vector of the local data change needs to be modified to update the data table.

[0092] The above-mentioned obtaining the first business data table stored in the business system and analyzing the second business data table stored in the database includes:

[0093] The first business data table stored in the business system and the second business data table stored in the analysis database are obtained at preset time intervals.

[0094] By regularly acquiring business data stored in the business system at preset time intervals, data redundancy can be reduced.

[0095] The above step 502 includes:

[0096] The data offset between the first business data table and the second business data table is calculated according to the business table type to obtain the data offset corresponding to the business data table.

[0097] The above business table types include full tables or incremental tables. Full tables can be used to store basic business data, and incremental tables can be used to store real-time updated business data.

[0098] If the business table type is a full table, comparing the data offset between all business data included in the first business data table and all business data included in the second business data table;

[0099] If the business table type is an incremental table, the data offset between the business data with business changes included in the first business data table and the business data with business changes included in the second business data table is compared.

[0100] After obtaining the data offset, write the data offset corresponding to the business data table into the associated data source table, such as Figure 7 As shown, Figure 7A reference schematic diagram of an associated data source table provided in an embodiment of the present application is shown, wherein the Offset field is a data offset field, which is used to indicate whether the business data is complete after being synchronized from the business system to the data lake layer.

[0101] For example, the result obtained by subtracting the corresponding table data volume in the business system from the table data volume stored in the data lake within the statistical time range is expressed as the data offset. Figure 7 The associated data source table is shown.

[0102] If it is an incrementally synchronized business table (i.e., incremental table), the data in the most recent cycle is calculated; if it is a fully synchronized business table (i.e., full table), the data in the entire table is calculated. For example, the business data table S11 of business 1 is an incrementally synchronized table. The data offset from 6:00 to 7:00 calculated at 7:02 is 0, and the data offset from 7:00 to 8:00 calculated at 8:02 is -10; the business data table Sn1 of business n is a fully synchronized table. The data offset calculated at every 08:01 is 0.

[0103] For records whose offset is not 0, you need to recalculate and update the offset value of the corresponding row in the table during the next calculation.

[0104] In the above-mentioned embodiments of the present application, the integrity of data replication in the data synchronization phase is automatically monitored to ensure the correctness of data in the data integration phase, thereby improving the execution efficiency of the data integration task.

[0105] Based on the above embodiments, the present application embodiment proposes a method for automatically monitoring the execution process of a task flow. Figure 6 , Figure 6 The following is a flow chart of a task monitoring method according to an embodiment of the present application. The method includes:

[0106] Step 601, determining one or more of at least one data integration task to be executed as a target data integration task;

[0107] Step 602: for the target data integration task, update the task status indication mark corresponding to the target data integration task to a first status mark;

[0108] Step 603, obtaining the data offset of the business data related to the target data integration task;

[0109] Step 604, determining whether the business data related to the target data integration task is complete according to the data offset;

[0110] Step 605, when it is determined that the business data related to the target data integration task is complete, data integration processing is performed on the business data related to the target data integration task to write the target data integration processing result into the subject area corresponding to the target data integration task;

[0111] Step 606: after the target data integration task is completed, update the task status indicator corresponding to the target data integration task to a second status indicator.

[0112] Step 607, returning to the loop to execute step 601, and executing steps 601 to 605 in sequence until all the data integration tasks to be executed are executed.

[0113] The above step 604 may include:

[0114] Step 6041, searching the associated data source table according to the task identifier of the target data integration task, and obtaining the data offset corresponding to the business data related to the target data integration task;

[0115] Step 6042, cumulatively calculating the data offsets corresponding to the business data related to the target data integration task, and obtaining the total amount of data offsets related to the target data integration task;

[0116] Step 6043, if the total target data offset is the first value, it means that the business data related to the target data integration task is complete;

[0117] Step 6044: If the total target data offset is the second value, it means that the business data related to the target data integration task is incomplete.

[0118] In the above steps, the embodiment of the present application can improve the operational efficiency of the data analysis platform by automatically monitoring the task flow of the data integration task, realize automatic monitoring of task execution and abnormal status, and automatically trigger notification when an abnormality is detected.

[0119] The following uses Pentaho's PDI tool to implement the method for automatically monitoring the execution process of task flows provided in the embodiment of the present application. Before the task flow starts, a corresponding reference table of data integration tasks and business data tables (also called data synchronization tables) is maintained in the system to save task scheduling time, such as Fig. 9 As shown, Fig. 9 A reference diagram of a corresponding relationship reference table provided in an embodiment of the present application is shown. In the corresponding relationship reference table, a mapping relationship between data integration tasks and business data tables is established, and the type of the business data table is marked at the same time, such as whether it is an incremental synchronization data table or a full synchronization data table.

[0120] like Figure 8 As shown, Figure 8 A flow chart of the current data integration task Job1 in the task flow monitoring process implemented by the PDI tool based on Pentaho in the ETL tool provided in an embodiment of the present application is shown.

[0121] Among them, task node ① (Get Schedule Time): from Fig.10 Get the data integration task Job1 from the task scheduling record table described, Fig.10 A reference schematic diagram of the task scheduling record table provided in the embodiment of the present application is shown. According to the task identifier and scheduling identifier of the data integration task, the time range of the business data to be processed in this scheduling is obtained. The task identifier is Job1, the scheduling identifier is BatchID, and the maximum scheduling identifier corresponding to the current task identifier is selected as the currently specified data integration task.

[0122] For example, BatchID=212, this record is newly inserted after the last data integration task was successfully executed (ie, the data integration task to be executed).

[0123] like Fig.12 As shown in a, the data integration task to be executed is first determined according to the task ID and scheduling ID of the data integration task. The BatchID, StartTime and EndTime values ​​in the task scheduling record table are assigned to the global variables V$BATCH_ID, V$START_TIME and V$END_TIME in Job1 respectively.

[0124] Task node ② (Update Job Status): After completing the above assignment, the Job1 task flow has been started, and the task flow Status of the corresponding record of Job1 in the update scheduling record table is -1 (ie, the task status indicator).

[0125] like Fig.12 As shown in b, after the assignment is completed, it indicates that the BatchIDth scheduling task of the data integration task Job1 has been started. In order to avoid task conflicts, the task status indication flag Status is pre-set to indicate and is modified to -1.

[0126] Task node ③ (Judge Data Integrity): judge whether the business data related to the data integration task Job1 in the associated data source table is complete. For example, the total data offset ZOffset associated with the data integration task Job1 can be obtained by calculating the corresponding data offsets of the business data tables S11 and S21.

[0127] Assume that ZOffset = 0, indicating that the business data associated with the data integration task Job1 is complete, then execute task nodes ④ and ⑤ in sequence, and successfully complete the data integration;

[0128] Assuming ZOffset≠0, execute task nodes ⑥ and ⑦ again to complete the exception notification processing.

[0129] The above calculation of the total data offset ZOffset associated with the data integration task Job1 can be performed according to the following formula:

[0130]

[0131] Among them, i represents the i-th business data table associated with the data integration task Job, and k represents the business data table set associated with the data integration task Job.

[0132] For incremental synchronization tables, you can filter out all related tables between V$START_TIME and V$END_TIME to determine the set of business data tables associated with the data integration task Job.

[0133] For a fully synchronized table, you only need to obtain the offset value corresponding to the row record where the data has changed.

[0134] For example, the data integration task Job1 requires data from two business data tables ods.S11 and ods.S21, and the most recent task scheduling requires the extraction of business time [7:00, 8:00] data.

[0135] Then the offset of ods.S11 and ods.S21 during [7:00, 8:00] is calculated as (-10)+(0)=-10.

[0136] Task node ④ (Main Process): If the result returned in task node ③ is 0, enter this task node to implement the business data processing process during V$START_TIME and V$END_TIME required by Topic 1, and complete the operation of writing into the target table.

[0137] Task node ⑤ (Insert Update Schedule): Entering this task node indicates that the data integration process has been successfully completed. At this time, the task status indicator in the task scheduling record table is updated to 1, and a new task record is inserted at the same time to define the necessary parameter values ​​for the next task scheduling.

[0138] like Fig.12c, after completing the data integration task with the current data integration task name Job1 and the scheduling identifier BatchID212, the task scheduling record table is updated, and the corresponding task status indicator is modified to 1. Then a new task record is inserted with the data integration task name Job1 and the scheduling identifier BatchID changed to 213.

[0139] Task node ⑥ (Insert Error Comment): For the associated data source table, extract all row records with Offset≠0, merge them into a string containing key information and write them into the Comment field of the task scheduling record table.

[0140] like Fig.11 As shown, when Offset≠0, the task scheduling record table also needs to be updated, that is, all row records with Offset≠0 are extracted and merged into a string containing key information, such as Fig.11 ods.S11 in the Comment field shown in a 2019-10-01 07:00~2019-10-01 08:00 GAP-10, or Fig.11 b ods.S11 GAP-10 in the Comment field shown in Figure 2.

[0141] Task node ⑦ (Notice): Start the automatic reminder process, read the Comment field of Job1 and BatchID=212 from the task scheduling record table, and notify the person in charge of the Comment field content via email push or SMS.

[0142] This application realizes the automatic monitoring process of data integration tasks by automatically scheduling data integration tasks, automatically checking data integrity and notifying exceptions, effectively improving the efficiency of data integration tasks and enhancing the quality of data analysis.

[0143] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a different order from the order marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0144] Please refer to the following Fig.13 , Fig.13 The schematic diagram of the structure of the data integrity automatic monitoring device provided in the embodiment of the present application is shown. The device can be configured in a server. The device includes:

[0145] The offset acquisition unit 801 is used to acquire the data offset of the business data related to the data integration task when determining the data integration task to be executed;

[0146] An integrity determination unit 802, configured to determine whether the business data related to the data integration task is complete according to the data offset;

[0147] The data integration unit 803 is used to perform data integration processing on the business data related to the integration task when it is determined that the business data related to the data integration task is complete, so as to write the data integration processing result into the subject area corresponding to the integration task.

[0148] The offset acquisition unit is also used to:

[0149] Obtaining a pre-created associated data source table related to the data integration task;

[0150] The associated data source table is searched according to the task identifier of the data integration task to obtain the data offset of the business data related to the data integration task.

[0151] The device also includes:

[0152] The associated table creation unit is used to create an associated data source table related to the data integration task.

[0153] The association table creation unit is used to obtain a first business data table stored in a business system and a second business data table stored in an analysis database, where the second business data table is generated when business data corresponding to the first business data table stored in the business system is copied to the analysis database; calculate a data offset corresponding to the business data table based on the first business data table and the second business data table; and write the data offset corresponding to the business data table into an association data source table.

[0154] The business data corresponding to the first business data table stored in the business system is copied to the analysis database, including:

[0155] Extracting incremental change data of business data corresponding to the first business data table in the business system;

[0156] Write incremental change data to local files;

[0157] Read incremental change data from local files and push them to the Kafka message queue;

[0158] Get the business data stored in the Kafka message queue and copy it to the second business data table in the analysis database.

[0159] When the business data corresponding to the first business data table stored in the business system is partially changed, copying the business data corresponding to the first business data table stored in the business system to the analysis database includes:

[0160] The business data with local changes is copied to the corresponding position in the second business data table.

[0161] The association table creation unit is used to obtain a first business data table stored in the business system and a second business data table stored in the analysis database at a preset time interval.

[0162] The association table creation unit may further include:

[0163] The first calculation subunit is used to calculate the data offset between the first business data table and the second business data table according to the business table type, and obtain the data offset corresponding to the business data table.

[0164] The first computing subunit is configured to:

[0165] If the business table type is a full table, comparing the data offset between all business data included in the first business data table and all business data included in the second business data table;

[0166] If the business table type is an incremental table, the data offset between the business data with business changes included in the first business data table and the business data with business changes included in the second business data table is compared.

[0167] You can also refer to the following Fig.14 , Fig.14 The schematic diagram of the structure of another data integrity automatic monitoring device provided in the embodiment of the present application is shown. The device can be configured in a server.

[0168] The device includes:

[0169] A task determination unit 901 is used to determine a data integration task to be performed;

[0170] The task determination unit 901 may include:

[0171] a task determination subunit, configured to determine one or more of at least one data integration task to be executed as a target data integration task;

[0172] The task status update subunit is used to update the task status indication identifier corresponding to the target data integration task to a first status identifier; after the target data integration task is completed, update the task status indication identifier corresponding to the target data integration task to a second status identifier, and return to the loop to execute the steps of determining the target data integration task and updating the task status identifier corresponding to the target data integration task until all the data integration tasks to be executed are executed.

[0173] An offset acquisition unit 902, configured to acquire a data offset of business data related to a data integration task when determining a data integration task to be executed;

[0174] An integrity determination unit 903, configured to determine whether the business data related to the data integration task is complete according to the data offset;

[0175] The data integration unit 904 is used to perform data integration processing on the business data related to the integration task when it is determined that the business data related to the data integration task is complete, so as to write the data integration processing result into the subject area corresponding to the integration task.

[0176] The key information processing unit 905 is used to extract the incomplete business data when it is determined that the business data related to the data integration task is incomplete; generate key information related to the data integration task based on the incomplete business data; and use the key information related to the data integration task as identification information of the data integration task.

[0177] The integrity determination unit 903 is further configured to:

[0178] Search the associated data source table according to the task identifier of the data integration task to obtain the data offset corresponding to the business data related to the data integration task;

[0179] The data offsets corresponding to the business data related to the data integration task are cumulatively calculated to obtain the total amount of data offsets related to the data integration task;

[0180] If the total amount of data offset is equal to the first value, it means that the business data related to the data integration task is complete;

[0181] If the total amount of data offset is not equal to the first value, it means that the business data related to the data integration task is incomplete.

[0182] It should be understood that the units or modules described in the above device are similar to those described in the reference Figure 2 The steps in the described method correspond to each other. Therefore, the operations and features described above for the method are also applicable to the above-mentioned device and the units contained therein, and will not be repeated here. The above-mentioned device can be pre-implemented in the browser or other security application of the electronic device, and can also be loaded into the browser or its security application of the electronic device by downloading or the like. The corresponding units in the above-mentioned device can cooperate with the units in the electronic device to implement the solution of the embodiment of the present application.

[0183] For the several modules or units mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0184] Reference below Fig.15 , Fig.15 A schematic diagram of the structure of a computer system suitable for implementing a server of an embodiment of the present application is shown.

[0185] like Fig.15 As shown, the computer system includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage portion 908 into a random access memory (RAM) 1003. Various programs and data required for system operation are also stored in the RAM 1003. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0186] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0187] In particular, according to an embodiment of the present disclosure, the above reference flow chart Figure 2 The described process can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a machine-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 1009, and / or installed from a removable medium 1011. When the computer program is executed by the central processing unit (CPU) 1001, the above-mentioned functions defined in the system of the present application are executed.

[0188] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0189] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of the code, and the aforementioned module, program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0190] The units or modules involved in the embodiments described in the present application may be implemented by software or by hardware. The described units or modules may also be arranged in a processor. For example, they may be described as: a processor includes an offset acquisition unit, an integrity determination unit, and a data integration unit. Among them, the names of these units or modules do not constitute a limitation on the units or modules themselves in certain cases. For example, the offset acquisition unit may also be described as "a unit for obtaining the data offset of the business data related to the data integration task".

[0191] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable storage medium stores one or more programs, and when the above programs are used by one or more processors to execute the data integrity automatic monitoring method described in the present application.

[0192] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the aforementioned disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A method for automatically monitoring data integrity, characterized in that: The method includes: When determining a data integration task to be executed, obtaining a data offset of business data related to the data integration task; including: Obtaining a pre-created associated data source table related to the data integration task; Searching the associated data source table according to the task identifier of the data integration task to obtain the data offset of the business data related to the data integration task; Determining whether the business data related to the data integration task is complete according to the data offset; When it is determined that the business data related to the data integration task is complete, performing data integration processing on the business data related to the integration task, so as to write the data integration processing result into the subject area corresponding to the data integration task; The step of creating an associated data source table related to the data integration task includes: Acquire a first business data table stored in the business system and a second business data table stored in the analysis database, wherein the second business data table is generated when business data corresponding to the first business data table stored in the business system is copied to the analysis database; Calculate the data offset corresponding to the business data table based on the first business data table and the second business data table; The data offset corresponding to the business data table is written into the associated data source table.

2. The method according to claim 1, characterized in that The method further includes: When it is determined that the business data related to the data integration task is incomplete, extracting the incomplete business data; generating key information related to the data integration task according to the incomplete business data; The key information related to the data integration task is used as identification information of the data integration task.

3. The method according to claim 1, characterized in that Copying the business data corresponding to the first business data table stored in the business system to the analysis database includes: Extracting incremental change data of business data corresponding to the first business data table in the business system; Writing the incremental change data into a local file of the business system; Read the incremental change data from the local file and push it to the Kafka message queue; The business data stored in the Kafka message queue is obtained and copied to the second business data table in the analysis database.

4. The method according to claim 1, characterized in that When the business data corresponding to the first business data table stored in the business system is partially changed, copying the business data corresponding to the first business data table stored in the business system to the analysis database includes: The business data that has undergone local changes is copied to a corresponding position in the second business data table.

5. The method according to claim 1, characterized in that The calculating the data offset corresponding to the business data table based on the first business data table and the second business data table comprises: The data offset between the first business data table and the second business data table is calculated according to the business table type to obtain the data offset corresponding to the business data table.

6. The method according to claim 5, characterized in that The calculating the data offset between the first business data table and the second business data table according to the business table type includes: If the business table type is a full table, comparing the data offset between all business data included in the first business data table and all business data included in the second business data table; If the business table type is an incremental table, the data offset between the business data with business changes included in the first business data table and the business data with business changes included in the second business data table is compared.

7. The method according to claim 1, characterized in that The step of determining the data integration task to be performed includes: determining one or more of the at least one data integration task to be performed as a target data integration task; For the target data integration task, updating the task status indication mark corresponding to the target data integration task to a first status mark; After the target data integration task is completed, the task status indication identifier corresponding to the target data integration task is updated to a second status identifier, and the steps of determining the target data integration task and updating the task status identifier corresponding to the target data integration task are returned to the loop until all the data integration tasks to be executed are executed.

8. The method according to claim 1, characterized in that The determining whether the business data related to the data integration task is complete according to the data offset includes: Searching the associated data source table according to the task identifier of the data integration task to obtain the data offset corresponding to the business data related to the data integration task; Accumulate and calculate the data offsets corresponding to the business data related to the data integration task to obtain the total amount of data offsets related to the data integration task; If the total amount of data offset is equal to the first value, it indicates that the business data related to the data integration task is complete; If the total amount of data offset is not equal to the first value, it means that the business data related to the data integration task is incomplete.

9. A data integrity automatic monitoring device, characterized in that: Applied to the method according to any one of claims 1 to 8, the device comprises: An offset acquisition unit, used to acquire a data offset of business data related to the data integration task when determining the data integration task to be executed; an integrity determination unit, configured to determine whether the business data related to the data integration task is complete according to the data offset; The data integration unit is used to perform data integration processing on the business data related to the integration task when it is determined that the business data related to the data integration task is complete, so as to write the data integration processing result into the subject area corresponding to the integration task.

10. An electronic device, characterized in that: The device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor is configured to implement the method according to any one of claims 1 to 8 when executing the program.

11. A computer-readable storage medium having a computer program stored thereon, wherein the computer program is used to implement the method according to any one of claims 1 to 8.

Images