A data repair method, device, apparatus and storage medium

Abstract

The application discloses a data repairing method and device, equipment and storage medium, and relates to the technical field of big data processing. The method comprises the following steps: collecting abnormal data generated when a current data real-time computing task is executed; the data real-time computing task is a task of performing real-time computing and processing on Binlog data by using Apache Flink; the abnormal data is written into an abnormal database for persistent storage; and the abnormal data pulled from the abnormal database is subjected to automatic repairing processing to obtain repaired data. The application collects abnormal data generated when a current data real-time computing task is executed, writes the collected abnormal data into an abnormal database for persistent storage, then subjects the abnormal data to be repaired pulled from the abnormal database to data processing, and sends the processed data to the current data real-time computing task for automatic data repairing, so that the abnormal processing mode of manual intervention is avoided, data consistency is ensured, and work efficiency and system stability are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data processing, and particularly relates to a data repair method and device, equipment and a storage medium. BACKGROUND

[0002] With the rapid development and continuous iteration of enterprises, more and more enterprises realize the importance of real-time data for scientific operation and decision analysis. In order to obtain data information faster and more accurately, and make effective operation and analysis decision, the enterprise needs to process and analyze business data from different business systems.

[0003] Currently, in the Apache Flink real-time computing process, the current system job mainly includes data extraction, data real-time computing and data output steps. Among them, the data extraction is mainly through the Apache Flink CDC (Change Data Capture) technology to extract the Binlog log of the MySQL database, and then perform data real-time computing, and output the computing result to the OLAP (OnLine Analytical Processing) storage system. However, in this process, there are abnormal situations such as data loss or incorrect data values, and for this data abnormality, the traditional abnormality processing method usually relies on manual intervention, which is time-consuming and prone to errors. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a data repair method, device, equipment and storage medium, which can avoid the abnormal processing method of manual intervention, realize the automatic repair of abnormal data, and thus ensure the data consistency. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses a data repair method, comprising:

[0006] Collecting abnormal data generated when executing a current data real-time computing task; the data real-time computing task is a task of using Apache Flink to perform real-time computing processing on Binlog data;

[0007] Writing the abnormal data into an abnormal database for persistent storage;

[0008] Performing data processing on the abnormal data pulled from the abnormal database to obtain processed data, and sending the processed data to the current data real-time computing task for automatic data repair.

[0009] Optionally, before the collecting abnormal data generated when executing a current data real-time computing task, the method further comprises:

[0010] collecting original Binlog data in a MySQL database through Apache Flink CDC technology;

[0011] converting the original Binlog data into standard Binlog data in a standard Binlog format so as to perform a current data real-time computing task based on the standard Binlog data;

[0012] The additional information of the standard Binlog data includes a Binlog database connection address, Binlog database library table information, and a timestamp field information of reading Binlog data.

[0013] Optionally, in the process of sending the processed data to the current data real-time computing task for data automatic repair, the method further comprises:

[0014] judging whether the processed data is repair data of the current data real-time computing task;

[0015] If the processed data is the repair data of the current data real-time computing task, further judging whether the processed data is delay out-of-order data;

[0016] If the processed data is the delay out-of-order data, discarding the processed data;

[0017] If the processed data is not the delay out-of-order data, re-executing the current data real-time computing task based on the processed data and the standard Binlog data.

[0018] Optionally, the judging whether the processed data is delay out-of-order data comprises:

[0019] judging whether a timestamp of the processed data is less than a last timestamp to obtain a corresponding judgment result; the last timestamp represents a timestamp when data with a same primary key as the processed data is sent to the current data real-time computing task last time;

[0020] determining whether the processed data is delay out-of-order data based on the judgment result;

[0021] Correspondingly, the determining whether the processed data is delay out-of-order data based on the judgment result comprises:

[0022] If the judgment result indicates that the timestamp of the processed data is less than the last timestamp, it is determined that the processed data is delay out-of-order data;

[0023] If the judgment result indicates that the timestamp of the processed data is not less than the last timestamp, it is determined that the processed data is not delay out-of-order data.

[0024] Optionally, the abnormal data generated when the current data real-time computing task is executed is collected, including:

[0025] The abnormal data generated when the current data real-time computing task is executed is collected through a side output stream, and the abnormal data is sent to a message middleware;

[0026] Correspondingly, the writing of the abnormal data into the abnormal database for persistent storage includes:

[0027] The abnormal data obtained from the message middleware is written into the abnormal database for persistent storage.

[0028] Optionally, before the writing of the abnormal data into the abnormal database for persistent storage, it further includes:

[0029] Judging whether the abnormal data is first written into the abnormal database;

[0030] If the abnormal data is first written into the abnormal database, an abnormal data primary key and a write time are generated, and a retry number field is initialized;

[0031] If the abnormal data is not first written into the abnormal database, the abnormal data primary key is obtained, and the retry number field and the write time are updated.

[0032] Optionally, the data processing of the to-be-repaired abnormal data pulled from the abnormal database includes:

[0033] The to-be-repaired abnormal data is pulled from the abnormal database according to a preset pulling time range and a preset maximum retry number condition;

[0034] The database link and the database table field value information are obtained according to additional information of the to-be-repaired abnormal data; wherein the additional information of the to-be-repaired abnormal data includes a Binlog database connection address, Binlog database table information, a timestamp field information of reading Binlog data, a retry number, an abnormal data primary key, and a task name field information;

[0035] The corresponding database JDBC link and DDL table structure query statement are generated according to the database link and the database table field value information;

[0036] The DDL table structure query statement is executed through the database JDBC link to obtain corresponding table structure, and the table structure is parsed to obtain corresponding primary key field information;

[0037] According to the primary key field information, a primary key field value corresponding to the abnormal data to be repaired is obtained;

[0038] According to the Binlog database table information, the primary key field information and the primary key field value, a corresponding query SQL statement is spliced;

[0039] The query SQL statement is executed through the database JDBC link to obtain corresponding query results;

[0040] The query results are converted into standard Binlog data in a standard Binlog format to obtain processed data.

[0041] Optionally, after the abnormal data is written into the abnormal database for persistent storage, the method further comprises:

[0042] The abnormal data in the abnormal database is displayed through a visual interface; the visual interface has a paging query function, a condition query function, a data deletion function and a repair API function.

[0043] In a second aspect, the present application discloses a data repair device, comprising:

[0044] An abnormal data acquisition module is configured to acquire abnormal data generated when a current data real-time computing task is executed; the data real-time computing task is a task of performing real-time computing and processing on Binlog data by using Apache Flink;

[0045] An abnormal data storage module is configured to write the abnormal data into an abnormal database for persistent storage;

[0046] An abnormal data repair module is configured to perform data processing on abnormal data to be repaired pulled from the abnormal database to obtain processed data, and send the processed data to the current data real-time computing task for automatic data repair.

[0047] In a third aspect, the present application discloses an electronic device, comprising:

[0048] A memory is configured to save a computer program;

[0049] A processor is configured to execute the computer program to realize the steps of the data repair method disclosed above.

[0050] In a fourth aspect, the present application discloses a computer readable storage medium for storing a computer program, wherein the computer program is executed by a processor to implement the steps of the data repair method disclosed above.

[0051] It can be seen that the present application provides a data repair method, which comprises: collecting abnormal data generated when a current data real-time computing task is executed; the data real-time computing task is a task of real-time computing and processing Binlog data by using Apache Flink; writing the abnormal data into an abnormal database for persistent storage; performing data processing on the abnormal data to be repaired pulled from the abnormal database to obtain processed data, and sending the processed data to the current data real-time computing task for automatic data repair. It can be seen that the present application collects abnormal data generated when a current data real-time computing task is executed, writes the collected abnormal data into an abnormal database for persistent storage, then performs data processing on the abnormal data to be repaired pulled from the abnormal database, sends the processed data to the current data real-time computing task for automatic data repair, thereby avoiding manual intervention in abnormal processing, ensuring data consistency, and improving work efficiency and system stability. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0053] Figure 1 A processing and analysis diagram of an existing service data disclosed by the present application;

[0054] Figure 2 A data repair method flow chart disclosed by the present application;

[0055] Figure 3 A data processing diagram disclosed by the present application;

[0056] Figure 4 A specific abnormal data persistent processing diagram disclosed by the present application;

[0057] Figure 5 A specific abnormal data automatic retry processing diagram disclosed by the present application;

[0058] Figure 6 A specific data repair method flow chart disclosed by the present application;

[0059] Figure 7 A data repair device structure schematic diagram disclosed by the present application is shown in the figure.

[0060] Figure 8 An electronic equipment structure diagram disclosed by the present application is shown in the figure. DETAILED DESCRIPTION

[0061] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0062] Currently, in the Apache Flink real-time computing processing, the current system job mainly includes data extraction, data real-time computing, and data output steps. Figure 1 As shown in the figure, the data extraction is mainly through the Apache Flink CDC technology to extract the Binlog log of the MySQL database, and then the Apache Flink is used for data real-time computing, and the computing result is written into the OLAP or OLTP (OnLine Transaction Processing, online transaction processing) database, wherein the OLAP database represents a kind of database system specially used to support fast analysis of large multidimensional data sets. However, in this process, there are abnormal situations such as data loss or incorrect data values, and for the data abnormal situation, the traditional exception handling method usually relies on manual intervention, and this data exception processing method is time-consuming and prone to error. Therefore, the present application provides a data repair scheme, which can avoid the abnormal handling mode of manual intervention, realize the automatic repair of abnormal data, thereby ensuring the data consistency, and improving the work efficiency and system stability.

[0063] The embodiment of the present application discloses a data repair method, as shown in the figure. Figure 2 The method comprises the following steps:

[0064] Step S11: collect abnormal data generated when a current data real-time computing task is executed; the data real-time computing task is a task of using Apache Flink to perform real-time computing processing on Binlog data.

[0065] It can be understood that there may be abnormal data such as incorrect data values when performing the current data real-time computing task, and then the abnormal data is collected. Specifically, the abnormal data generated when performing the current data real-time computing task is collected through a side output stream, and the abnormal data is sent to the message middleware. Wherein, Binlog represents a binary log file in a MySQL database, and Apache Flink represents an open source project of a distributed stream processing and batch processing framework.

[0066] In the embodiment, before collecting the abnormal data generated when performing the current data real-time computing task, the method can further include: collecting original Binlog data in the MySQL database through Apache Flink CDC technology; converting the original Binlog data into standard Binlog data in a standard Binlog format so as to perform the current data real-time computing task based on the standard Binlog data; wherein, additional information of the standard Binlog data includes a Binlog database connection address, Binlog database library table information, and a timestamp field information of reading Binlog data. It can be understood that the original Binlog data of the MySQL database is read through the Apache Flink CDC technology, and the original Binlog data is converted into the standard Binlog data in the standard Binlog format. The data real-time computing task is performed based on the standard Binlog data, and the abnormal data generated in the process of real-time computing the standard Binlog data can be collected through the side output stream. Moreover, the standard Binlog data converted from the original Binlog data has additional additional information, and the additional information of the standard Binlog data includes the Binlog database connection address, the Binlog database library table information, and the timestamp field information of reading the Binlog data. Wherein, the Apache Flink CDC technology represents a framework of change data capture technology based on Flink.

[0067] In the embodiment, the process of sending the processed data to the current data real-time computing task for automatic repair of the data can specifically include: determining whether the processed data is repair data of the current data real-time computing task; if the processed data is the repair data of the current data real-time computing task, further determining whether the processed data is delayed out-of-order data; if the processed data is the delayed out-of-order data, discarding the processed data; and if the processed data is not the delayed out-of-order data, re-executing the current data real-time computing task based on the processed data and the standard Binlog data. It can be understood that, in order to guarantee data consistency between the normal Binlog data and the processed data, it is necessary to first guarantee that the processed data is repair data of the current data real-time computing task, that is, to filter processed data that is not repair data of the current data real-time computing task. Specifically, the task name field in the additional information of the processed data is filtered to guarantee that the processed data is repair data of the current data real-time computing task, and then it is determined whether the processed data is delayed out-of-order data. Specifically, it is determined whether the timestamp of the processed data is less than the last timestamp to obtain a corresponding determination result; the last timestamp represents a timestamp when data with the same primary key as the processed data was last sent to the current data real-time computing task; it is determined whether the processed data is delayed out-of-order data based on the determination result; if the determination result indicates that the timestamp of the processed data is less than the last timestamp, it is determined that the processed data is delayed out-of-order data; and if the determination result indicates that the timestamp of the processed data is not less than the last timestamp, it is determined that the processed data is not delayed out-of-order data. That is, it is determined whether the timestamp in the additional information of the processed data is less than the timestamp of the read Binlog data in the additional information carried by the repair data with the same primary key as the processed data when the repair data was last sent to the current real-time data computing task. If the timestamp is less than the last timestamp, it indicates that the processed data is delayed out-of-order data, and the processed data is discarded. If the timestamp is not less than the last timestamp, it indicates that the processed data is not delayed out-of-order data, that is, when the processed data is sent to the current data real-time computing task, the processed data is not immediately used for corresponding repair processing, but it is guaranteed that the processed data is repair data of the current data real-time computing task and is not delayed out-of-order data, and the current data real-time computing task is re-executed based on the processed data and the standard Binlog data to perform corresponding automatic repair of the data.

[0068] For example, see Figure 3The method further includes: judging whether the processed data is repair data of the current data real-time computing task; if the processed data is repair data of the current data real-time computing task, further judging whether the processed data is delay out-of-order data; if the processed data is the delay out-of-order data, discarding the abnormal data; if the processed data is not the delay out-of-order data, re-executing the current data real-time computing task based on the processed data and the standard Binlog data, saving a corresponding computing result into a database, collecting abnormal data generated during re-execution of the current data real-time computing task through a side output stream, and sending the abnormal data to a message middleware Apache Kafka. Apache Kafka represents an open source, high-throughput distributed message queue system.

[0069] Step S12: writing the abnormal data into an abnormal database for persistent storage.

[0070] In the embodiment, after the abnormal data generated during execution of the current data real-time computing task is collected, the collected abnormal data is written into the abnormal database for persistent storage. Specifically, the abnormal data obtained from the message middleware is written into the abnormal database for persistent storage. It can be understood that after the abnormal data generated during execution of the current data real-time computing task is collected, the abnormal data is sent to the message middleware Apache Kafka, then the abnormal data is obtained from the message middleware Apache Kafka, and finally the abnormal data is written into the abnormal database for persistent storage.

[0071] In the embodiment, before the abnormal data is written into the abnormal database for persistent storage, the method further includes: judging whether the abnormal data is first-time written into the abnormal database; if the abnormal data is first-time written into the abnormal database, generating an abnormal data primary key and a write time, and initializing a retry number field; if the abnormal data is not first-time written into the abnormal database, obtaining the abnormal data primary key, and updating the retry number field and the write time. It can be understood that for the abnormal data first-time written into the abnormal database, an abnormal data primary key and a data write time need to be generated, and a retry number field needs to be initialized. Specifically, the abnormal data primary key is composed of a string concatenation of an Apache Kafka offset cursor and a partiton partition, and the data write time is generated, and the retry number field is initialized as 1. If the abnormal data is not first-time written into the abnormal database, the abnormal data primary key in the additional information is taken, the retry number is added by 1, the data write time is updated, and finally the abnormal data is written into the abnormal database. For example, refer to Figure 4As shown, it is judged whether the abnormal data is the first time to be written into the abnormal database; if yes, the primary key is calculated according to the rule, the retry number field is initialized, and the writing time is added; if not, the primary key in the value additional information is taken, the retry number is added by 1, the writing time is added, and finally the abnormal data is written into the abnormal database.

[0072] Step S13: data processing is performed on the abnormal data to be repaired pulled from the abnormal database to obtain processed data, and the processed data is sent to the current data real-time computing task for automatic data repair.

[0073] In the embodiment, the data to be repaired pulled from the exception database is processed, specifically, the exception data to be repaired is pulled from the exception database according to a preset pulling time range and a preset maximum retry number condition; database link and database table field value information are obtained according to additional information of the exception data to be repaired; wherein the additional information of the exception data to be repaired includes Binlog database connection address, Binlog database table information, timestamp field information of reading Binlog data, retry number, exception data primary key and task name field information; corresponding database JDBC (Java DataBase Connectivity, Java database connection) link and DDL (Data Definition Language, data definition language) table structure query statement are generated according to the database link and the database table field value information; the DDL table structure query statement is executed through the database JDBC link to obtain corresponding table structure, and the table structure is parsed to obtain corresponding primary key field information; the primary key field value corresponding to the exception data to be repaired is obtained according to the primary key field information; the query SQL (Structured Query Language, structured query language) statement is spliced according to the Binlog database table information, the primary key field information and the primary key field value; the query result is obtained by executing the query SQL statement through the database JDBC link; and the processed data is obtained by converting the query result into standard Binlog format standard Binlog data.It can be understood that the last pull time of the exception data table in the exception database is stored in the Flink State state, and then the last pull time is taken as the start time, the current time is taken as the end time, the preset pull time range is obtained, and the maximum retry times condition is set to prevent infinite retries, that is, the abnormal database is pulled according to the preset pull time range and the maximum retry times condition to obtain the to-be-repaired exception data, and then the to-be-repaired exception data is parsed, that is, the database link and the database table field value information are obtained according to the additional information of the to-be-repaired exception data, the database JDBC link and the DDL table structure query statement are generated according to the above information, and the DDL table structure query statement is executed to obtain the corresponding table structure, then the primary key field information is parsed, and finally the primary key field information can be written into the Flink State state, so that subsequent exception data of the same database table can be stored, and then the primary key field information in the Flink State state can be directly taken, and then the primary key field value corresponding to the to-be-repaired exception data is obtained according to the primary key field information, and the corresponding query SQL is spliced according to the database table information, the primary key field information and the primary key field value, and then the query SQL is executed through the database JDBC link to obtain the corresponding query result, then the Mysql field type is parsed according to the query result, the Java data type is obtained, the query result is converted into standard Binlog data to obtain the processed data, and the additional information additionally added for the repaired processed data during the data format conversion process includes the Binlog database connection address, the original Binlog database table information, the timestamp field information of executing the query SQL, the retry times, and the exception table primary key information. Wherein, the Flink State indicates the data that can be stored and managed in the Flink program.

[0074] For example, see Figure 5As shown, the abnormal data pulled from the abnormal database is subjected to automatic repair processing to obtain repaired data, mainly including the following aspects, namely Mysql DDL table structure analysis, Mysql data query, Mysql data result format conversion, and message middleware Apache Kafka. That is, the abnormal data is pulled from the abnormal database, the database link and the database table field value information are obtained according to the additional information of the abnormal data, the database JDBC link and the DDL table structure query statement are generated according to the above information, and the DDL table structure query statement is executed to obtain the corresponding table structure, then the primary key field information is parsed, the primary key field value corresponding to the abnormal data is obtained according to the primary key field information, and the corresponding query SQL is spliced according to the database table information, the primary key field information and the primary key field value, then the query SQL is executed through the database JDBC link to obtain the corresponding query result, then the query result is converted into standard Binlog data in standard Binlog format, and is sent to the message middleware Apache Kafka.

[0075] In the embodiment, the abnormal data in the abnormal database can also be displayed through a visual interface. The visual interface has a paging query function, a condition query function, a data deletion function and a repair API (Application Program Interface) function. It can be understood that a visual management service of abnormal data is provided, which includes a backend service and a front-end service. The backend service provides a paging query function, a condition query function, a data deletion function and a repair API function for abnormal data, and the front-end service provides a visual interface that can connect to the API of the backend service to display abnormal data and provide add, delete, modify and query functions.

[0076] It can be seen that in the embodiment of the application, abnormal data generated during execution of a current data real-time computing task is collected and written into an abnormal database for persistent storage, then the abnormal data to be repaired pulled from the abnormal database is subjected to data processing, and the processed data is sent to the current data real-time computing task for automatic data repair, so that manual intervention in abnormal processing can be avoided, data consistency is ensured, and work efficiency and system stability are improved.

[0077] For example, referring to Figure 6As shown, the original Binlog data of the MySQL database is read, real-time calculation of data is performed based on the read original Binlog data by using Apache Flink, and abnormal data generated in the real-time calculation process is sent to the message middleware Apache Kafka, and the abnormal data is stored in the OLAP / OLTP database through an abnormal data persistence task, that is, the abnormal data is obtained from the message middleware Apache Kafka, the abnormal data is processed, and then the abnormal data is written into the database for persistent storage, the abnormal data to be repaired is pulled from the OLAP / OLTP database through an abnormal data automatic retry task, the abnormal data to be repaired is processed, and then the processed data is sent to the message middleware Apache Kafka, and then the processed data is added to the current data real-time calculation task from the message middleware Apache Kafka to re-perform corresponding calculation, and the calculation result is written into the OLAP / OLTP database, so that the automatic retry repair of data is completed, and the abnormal data in the OLAP / OLTP database can also be displayed through a visual interface, and the functions of adding, deleting, modifying and inquiring are provided.

[0078] Correspondingly, the embodiment of the application also discloses a data repair device, as shown in Figure 7 The device comprises:

[0079] An abnormal data acquisition module 11 is configured to acquire abnormal data generated when a current data real-time calculation task is performed; the data real-time calculation task is a task of performing real-time calculation and processing on Binlog data by using Apache Flink;

[0080] An abnormal data storage module 12 is configured to write the abnormal data into an abnormal database for persistent storage.

[0081] An abnormal data repair module 13 is configured to perform data processing on abnormal data to be repaired pulled from the abnormal database to obtain processed data, and send the processed data to the current data real-time calculation task for automatic data repair.

[0082] As can be seen from the above, in the embodiment of the application, abnormal data generated when a current data real-time calculation task is performed is acquired, and the acquired abnormal data is written into an abnormal database for persistent storage, then abnormal data to be repaired pulled from the abnormal database is processed, the processed data is sent to the current data real-time calculation task for automatic data repair, so that manual intervention in the abnormal processing mode can be avoided, data consistency is ensured, and work efficiency and system stability are improved.

[0083] In some specific embodiments, the data repair device can further include, in particular:

[0084] An original data collection module is configured to collect original Binlog data in a MySQL database through Apache Flink CDC technology.

[0085] A format conversion module is configured to convert the original Binlog data into standard Binlog data in a standard Binlog format so as to perform a current data real-time computing task based on the standard Binlog data.

[0086] The additional information of the standard Binlog data includes a Binlog database connection address, Binlog database table information, and a timestamp field information of reading Binlog data.

[0087] In some specific embodiments, the abnormal data repair module 13 can include, in particular:

[0088] A first judgment module is configured to judge whether the processed data is repair data of a current data real-time computing task.

[0089] A second judgment module is configured to further judge whether the processed data is delay out-of-order data if the processed data is repair data of the current data real-time computing task.

[0090] A data discarding module is configured to discard the processed data if the processed data is the delay out-of-order data.

[0091] A data saving module is configured to re-perform a current data real-time computing task based on the processed data and the standard Binlog data if the processed data is not the delay out-of-order data.

[0092] In some specific embodiments, the second judgment module can include, in particular:

[0093] A timestamp judgment unit is configured to judge whether a timestamp of the processed data is less than a last timestamp to obtain a corresponding judgment result; the last timestamp represents a timestamp when data with a same primary key as the processed data is sent to a current data real-time computing task last time.

[0094] A delay out-of-order data determination unit is configured to determine whether the processed data is delay out-of-order data based on the judgment result.

[0095] In some specific embodiments, the delay out-of-order data determination unit can include, in particular:

[0096] a first determining sub-unit configured to determine that the processed data is delayed out-of-order data if the determination result indicates that the time stamp of the processed data is less than the last time stamp;

[0097] a second determining sub-unit configured to determine that the processed data is not delayed out-of-order data if the determination result indicates that the time stamp of the processed data is not less than the last time stamp.

[0098] In some specific embodiments, the abnormal data collection module 11 can specifically include:

[0099] an abnormal data collection unit configured to collect abnormal data generated when a current data real-time computing task is executed through a side output stream, and send the abnormal data to a message middleware;

[0100] In some specific embodiments, the abnormal data storage module 12 can specifically include:

[0101] an abnormal data storage unit configured to write the abnormal data obtained from the message middleware into an abnormal database for persistent storage.

[0102] In some specific embodiments, the data repair device can specifically include:

[0103] a third determining module configured to determine whether the abnormal data is first written into the abnormal database;

[0104] a first processing module configured to generate an abnormal data primary key and a write time, and initialize a retry number field if the abnormal data is first written into the abnormal database;

[0105] a second processing module configured to obtain the abnormal data primary key, and update the retry number field and the write time if the abnormal data is not first written into the abnormal database.

[0106] In some specific embodiments, the abnormal data repair module 13 can specifically include:

[0107] a data pulling unit configured to pull abnormal data to be repaired from the abnormal database according to a preset pulling time range and a preset maximum retry number condition;

[0108] a first obtaining unit configured to obtain database link and database table field value information according to additional information of the abnormal data to be repaired; wherein the additional information of the abnormal data to be repaired includes Binlog database connection address, Binlog database table information, time stamp field information of reading Binlog data, retry number, abnormal data primary key, and task name field information;

[0109] The generating unit is configured to generate corresponding database JDBC links and DDL table structure query statements according to the database links and the database table field value information.

[0110] The parsing unit is configured to execute the DDL table structure query statements through the database JDBC links to obtain corresponding table structures, and parse the table structures to obtain corresponding primary key field information.

[0111] The second obtaining unit is configured to obtain primary key field values corresponding to the abnormal data to be repaired according to the primary key field information.

[0112] The splicing unit is configured to splice corresponding query SQL statements according to the Binlog database table information, the primary key field information and the primary key field values.

[0113] The statement executing unit is configured to execute the query SQL statements through the database JDBC links to obtain corresponding query results.

[0114] The format converting unit is configured to convert the query results into standard Binlog data in a standard Binlog format to obtain processed data.

[0115] In some specific embodiments, the data repairing apparatus can specifically include:

[0116] The display module is configured to display the abnormal data in the abnormal database through a visual interface, and the visual interface has a paging query function, a condition query function, a data deletion function and a repairing API function.

[0117] Further, the embodiment of the present application also provides an electronic device. Figure 8 The electronic device 20 is shown in the structure diagram according to an exemplary embodiment, and the content in the diagram cannot be considered as any limitation on the use range of the present application.

[0118] Figure 8 The electronic device 20 is shown in the structure diagram according to an exemplary embodiment, and the content in the diagram cannot be considered as any limitation on the use range of the present application.

[0119] In this embodiment, the power supply 23 is configured to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 is configured to create a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which is not limited specifically herein; the input / output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application requirement, which is not limited specifically herein.

[0120] In addition, the memory 22 as a carrier of resource storage can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage mode can be temporary storage or permanent storage.

[0121] The operating system 221 is configured to manage and control each hardware device on the electronic device 20 and the computer program 222, and can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program capable of completing the data repair method executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 can further include a computer program capable of completing other specific work.

[0122] Further, the embodiment of the present application further discloses a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program is loaded and executed by a processor, and the data repair method steps disclosed in any of the foregoing embodiments are realized.

[0123] In the specification, each embodiment is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0124] Finally, it needs to be pointed out that in this document, relational terms such as first and second and the like can only be used to distinguish one entity or action from another entity or action, without necessarily requiring or implying that there is any such actual relationship or order between these entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, so that a process, method, article, or apparatus that comprises a list of elements does not include only those elements, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by the statement "comprising a" does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0125] The above describes in detail the data repair method, device, equipment and storage medium provided by the present application. The principles and implementation manners of the present application are described by applying specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges can be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A data repair method characterized by, The method comprises the following steps: collecting abnormal data generated during execution of a current data real-time computing task; the data real-time computing task is a task of real-time computing and processing Binlog data by using Apache Flink; writing the abnormal data into an exception database for persistent storage; performing data processing on the abnormal data pulled from the exception database to obtain processed data, and sending the processed data to the current data real-time computing task for automatic data repair; wherein the data processing on the abnormal data pulled from the exception database comprises the following steps: pulling abnormal data to be repaired from the exception database according to a preset pulling time range and a preset maximum retry number condition; obtaining database link and database table field value information according to additional information of the abnormal data to be repaired; wherein the additional information of the abnormal data to be repaired includes Binlog database connection address, Binlog database table information, timestamp field information of reading Binlog data, retry number, abnormal data primary key, and task name field information; generating corresponding database JDBC link and DDL table structure query statements according to the database link and the database table field value information; executing the DDL table structure query statements through the database JDBC link to obtain corresponding table structures, and parsing the table structures to obtain corresponding primary key field information; obtaining primary key field values corresponding to the abnormal data to be repaired according to the primary key field information; splicing corresponding query SQL statements according to the Binlog database table information, the primary key field information, and the primary key field values; executing the query SQL statements through the database JDBC link to obtain corresponding query results; converting the query results into standard Binlog data in a standard Binlog format to obtain processed data.

2. The data repair method of claim 1, wherein, Before collecting the abnormal data generated during execution of the current data real-time computing task, the method further comprises the following steps: collecting original Binlog data in a MySQL database by using Apache Flink CDC technology; converting the original Binlog data into standard Binlog data in a standard Binlog format so as to execute the current data real-time computing task based on the standard Binlog data; wherein the additional information of the standard Binlog data includes Binlog database connection address, Binlog database table information, and timestamp field information of reading Binlog data.

3. The data repair method of claim 2, wherein, In the process of sending the processed data to the current data real-time computing task for automatic data repair, the method further comprises the following steps: determining whether the processed data is repair data of the current data real-time computing task; if the processed data is the repair data of the current data real-time computing task, further determining whether the processed data is delayed out-of-order data; if the processed data is the delayed out-of-order data, discarding the processed data. If the processed data is not the delayed out-of-order data, a current data real-time computing task is re-executed based on the processed data and the standard Binlog data.

4. The data repair method of claim 3, wherein, The determining whether the processed data is the delayed out-of-order data comprises: determining whether a timestamp of the processed data is less than a last timestamp to obtain a corresponding determination result, the last timestamp indicating a timestamp when data with a same primary key as the processed data was last sent to the current data real-time computing task; determining whether the processed data is the delayed out-of-order data based on the determination result; Correspondingly, the determining whether the processed data is the delayed out-of-order data based on the determination result comprises: if the determination result indicates that the timestamp of the processed data is less than the last timestamp, determining that the processed data is the delayed out-of-order data; if the determination result indicates that the timestamp of the processed data is not less than the last timestamp, determining that the processed data is not the delayed out-of-order data.

5. The data repair method of claim 1, wherein, The collecting of the abnormal data generated when the current data real-time computing task is executed comprises: collecting the abnormal data generated when the current data real-time computing task is executed through a side output stream, and sending the abnormal data to a message middleware; Correspondingly, the writing of the abnormal data into the abnormal database for persistent storage comprises: writing the abnormal data obtained from the message middleware into the abnormal database for persistent storage.

6. The data repair method of claim 1, wherein, Before the writing of the abnormal data into the abnormal database for persistent storage, the method further comprises: determining whether the abnormal data is first written into the abnormal database; if the abnormal data is first written into the abnormal database, generating an abnormal data primary key and a write time, and initializing a retry number field; if the abnormal data is not first written into the abnormal database, obtaining the abnormal data primary key, and updating the retry number field and the write time.

7. The data recovery method of any one of claims 1 to 6, wherein, After the writing of the abnormal data into the abnormal database for persistent storage, the method further comprises: displaying the abnormal data in the abnormal database through a visual interface, the visual interface having a paging query function, a conditional query function, a data deletion function, and a repair API function.

8. A data recovery apparatus, characterized by comprising: The method comprises: an abnormal data collection module configured to collect abnormal data generated when a current data real-time computing task is executed; the data real-time computing task is a task of performing real-time computing processing on Binlog data by using Apache Flink; an abnormal data storage module configured to write the abnormal data into an abnormal database for persistent storage; an abnormal data repair module configured to perform data processing on to-be-repaired abnormal data pulled from the abnormal database to obtain processed data, and send the processed data to the current data real-time computing task for automatic data repair; the abnormal data repair module comprises: a data pulling unit configured to pull to-be-repaired abnormal data from the abnormal database according to a preset pulling time range and a preset maximum retry number condition; The first obtaining unit is configured to obtain database link and database table field value information according to additional information of the abnormal data to be repaired; wherein the additional information of the abnormal data to be repaired comprises Binlog database connection address, Binlog database table information, timestamp field information of read Binlog data, retry number, abnormal data primary key and task name field information; The generating unit is configured to generate corresponding database JDBC link and DDL table structure query statement according to the database link and the database table field value information; The parsing unit is configured to execute the DDL table structure query statement through the database JDBC link to obtain corresponding table structure, and parse the table structure to obtain corresponding primary key field information; The second obtaining unit is configured to obtain primary key field value corresponding to the abnormal data to be repaired according to the primary key field information; The splicing unit is configured to splice corresponding query SQL statement according to the Binlog database table information, the primary key field information and the primary key field value; The statement executing unit is configured to execute the query SQL statement through the database JDBC link to obtain corresponding query result; The format converting unit is configured to convert the query result into standard Binlog data in standard Binlog format to obtain processed data.

9. An electronic device, comprising: The memory is configured to save a computer program; The processor is configured to execute the computer program to realize steps of the data repair method according to any one of claims 1 to 7. The memory is configured to save a computer program; 10. A computer-readable storage medium, characterized in that, The processor is configured to execute the computer program to realize steps of the data repair method according to any one of claims 1 to 7.

Images