Data structure change notification method, device, equipment and storage medium

Abstract

A data structure change notification method, device, equipment and computer-readable storage medium. The method includes: obtaining the log change record of the monitored data source based on the log capture tool in Flink CDC; detecting whether the log change record is a data structure change record; if it is a data structure change record, parsing the log change record to obtain data structure change information; saving the data structure change information to the data collector in Flink CDC; and performing message notification based on the data structure change information in the data collector. Through the present application, data structure change events can be automatically identified based on log change records, and message notifications can be triggered, thereby realizing the monitoring of data structure changes of data sources; by automatically determining data structure change information, manual comparison of the structure of the data source and the data warehouse is avoided, and the data synchronization failure processing time is shortened.

Description

Technical Field

[0001] The present application relates to the field of data management technology, and in particular to a data structure change notification method, device, equipment and computer-readable storage medium. Background Art

[0002] With the rapid development of enterprise digital transformation, enterprises are carrying out digital construction. As the key infrastructure for enterprise data management and application, data warehouse is playing an increasingly important role. Data warehouse construction is not only the cornerstone of enterprise digital transformation, but also a key step for enterprises to achieve intelligent decision-making and business optimization. As data warehouses continue to concentrate enterprise information system data, how to ensure the consistency of data warehouse source layer data with data source information has become a difficulty in data warehouse construction. At present, there are some problems with data source data synchronization:

[0003] There is a lack of monitoring methods for data structure changes in data sources, and the only way to detect changes is to passively wait for data synchronization tasks to fail, which affects the use of data warehouses.

[0004] When the data structure of a data source changes, it is impossible to accurately capture the changes and the implementation staff must manually check, which is inefficient. Summary of the invention

[0005] To solve at least one of the above technical problems, the present application provides a data structure change notification method, device, equipment and computer-readable storage medium.

[0006] In a first aspect, an embodiment of the present application provides a data structure change notification method, the data structure change notification method comprising:

[0007] Obtain log change records of monitored data sources based on the log capture tool in Flink CDC;

[0008] Detecting whether the log change record is a data structure change record;

[0009] If it is a data structure change record, the log change record is parsed to obtain the data structure change information;

[0010] The data collector that saves data structure change information to Flink CDC;

[0011] Message notification is performed based on the data structure change information in the data collector.

[0012] In combination with the first aspect, in one implementation, the step of detecting whether the log change record is a data structure change record includes:

[0013] Detect whether the keySchema in the log change record is a preset string;

[0014] If it is a preset character string, it is determined that the log change record is a data structure change record.

[0015] In combination with the first aspect, in one implementation, if it is a data structure change record, the step of parsing the log change record to obtain the data structure change information includes:

[0016] If it is a data structure change record, the historyRecord information in the log change record is parsed to obtain the SQL statement in which the data structure has changed and the name of the database in which the data structure has changed.

[0017] In combination with the first aspect, in one implementation, the step of saving the data structure change information to the data collector in Flink CDC includes:

[0018] The SQL statement with changed data structure and the database name with changed data structure are constructed into a Tuple2 object, and the Tuple2 object is saved to the data collector in Flink CDC.

[0019] In combination with the first aspect, in one implementation, the step of performing message notification based on the data structure change information in the data collector includes:

[0020] Parse the Tuple2 object in the data collector to obtain the SQL statement in which the data structure is changed and the name of the database in which the data structure is changed;

[0021] The recipient information, notification method, SQL statement of the data structure change, and the database name of the data structure change are sent to the message notification platform, so that the message notification platform can send the SQL statement of the data structure change and the database name of the data structure change to the recipient corresponding to the recipient information according to the notification method.

[0022] In combination with the first aspect, in an implementation manner, after the step of performing message notification based on the data structure change information in the data collector, the method further includes:

[0023] The data structure change information is recorded in a change information table.

[0024] In a second aspect, an embodiment of the present application provides a data structure change notification device, the data structure change notification device comprising:

[0025] The acquisition module is used to obtain log change records of the monitored data source based on the log capture tool in Flink CDC;

[0026] A detection module, used to detect whether the log change record is a data structure change record;

[0027] The parsing module is used to parse the log change record to obtain the data structure change information if it is a data structure change record;

[0028] The saving module is used to save the data structure change information to the data collector in Flink CDC;

[0029] The notification module is used to make message notifications based on the data structure change information in the data collector.

[0030] In conjunction with the second aspect, in one implementation, the detection module is used to:

[0031] Detect whether the keySchema in the log change record is a preset string;

[0032] If it is a preset character string, it is determined that the log change record is a data structure change record.

[0033] In a third aspect, an embodiment of the present application provides a data structure change notification device, which includes a processor, a memory, and a data structure change notification program stored in the memory and executable by the processor, wherein when the data structure change notification program is executed by the processor, the steps of the data structure change notification method described in the first aspect are implemented.

[0034] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a data structure change notification program is stored, wherein when the data structure change notification program is executed by a processor, the steps of the data structure change notification method described in the first aspect are implemented.

[0035] The beneficial effects brought by the technical solution provided in the embodiments of the present application include:

[0036] In the embodiment of the present application, the log change record of the monitored data source is obtained based on the log capture tool in Flink CDC; the log change record is detected to see if it is a data structure change record; if it is a data structure change record, the log change record is parsed to obtain the data structure change information; the data structure change information is saved to the data collector in Flink CDC; and a message notification is made based on the data structure change information in the data collector. Through the embodiment of the present application, data structure change events can be automatically identified based on log change records, and message notifications are triggered, thereby realizing the monitoring of data structure changes of data sources; by automatically determining data structure change information, manual comparison of the structure of the data source and the data warehouse is avoided, and the data synchronization failure processing time is shortened. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flow chart of an embodiment of a method for notifying a data structure change of the present application;

[0038] Figure 2 This is a functional module diagram of an embodiment of a data structure change notification device of the present application;

[0039] Figure 3 This is a schematic diagram of the hardware structure of a data structure change notification device involved in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0041] First, some technical terms in the present application are explained to facilitate those skilled in the art to understand the present application.

[0042] Flink CDC (Change Data Capture) is an open source database change log capture and processing framework that can capture data changes from various databases (such as MySQL, PostgreSQL, Oracle, MongoDB, etc.) in real time and convert them into streaming data. Flink CDC can help real-time applications process and analyze these streaming data in real time, thereby realizing functions such as data synchronization, data pipeline, real-time analysis, and real-time application.

[0043] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0044] In a first aspect, an embodiment of the present application provides a data structure change notification method.

[0045] In one embodiment, referring to Figure 1 , Figure 1 This is a flow chart of an embodiment of a data structure change notification method of this application. Figure 1 As shown, the data structure change notification method includes:

[0046] Step S10, obtaining the log change record of the monitored data source based on the log capture tool in Flink CDC;

[0047] In this embodiment, according to the requirements of Flink CDC, the monitoring data source needs to enable the log archiving function, and assign the data source connection user the permission to read the archived logs, so as to facilitate reading the archived log information. Therefore, it is necessary to set the data source connection address, connection account, connection password, monitored data table range, log change range, log change start time, data source structure change capture and other information in Flink CDC in advance. On this basis, the log capture tool in Flink CDC can obtain the log change records of the monitored data source.

[0048] Step S20, detecting whether the log change record is a data structure change record;

[0049] In this embodiment, after obtaining the log change record of the monitored data source, it is necessary to further detect whether the log change record is a data structure change record. The key fields in the log change record can be identified to determine whether the log change record is a data structure change record.

[0050] Furthermore, in one embodiment, step S20 includes:

[0051] Detect whether the keySchema in the log change record is a preset string; if it is a preset string, determine that the log change record is a data structure change record.

[0052] In this embodiment, it can be implemented by a custom log capture parser. Specifically, the log capture parser inherits the DebeziumDeserializationSchema class of the Flink CDC library and rewrites the deserialize method therein.

[0053] The log change record is parsed through the deserialize method in the log capture parser, and the keySchema in the log change record is read. If the keySchema is a preset string, the log change record is determined to be a data structure change record.

[0054] Among them, the preset string is, for example, io.debezium.connector.mysql.SchemaChangeKey.

[0055] Step S30: if it is a data structure change record, the log change record is parsed to obtain data structure change information;

[0056] In this embodiment, if it is a data structure change record, the log change record is further parsed to determine the data structure change information.

[0057] Furthermore, in one embodiment, step S30 includes:

[0058] If it is a data structure change record, the historyRecord information in the log change record is parsed to obtain the SQL statement in which the data structure has changed and the name of the database in which the data structure has changed.

[0059] In this embodiment, if it is a data structure change record, the historyRecord information is parsed from the log change record, the ddl field is read from the historyRecord, and the SQL statement in which the data structure has changed is parsed. The databaseName field is read from the historyRecord, and the database name in which the data structure has changed is parsed. The above processing is implemented by the log capture parser.

[0060] Step S40, saving the data structure change information to the data collector in Flink CDC;

[0061] In this embodiment, after obtaining the data structure change information from the log change record, the data structure change information can be saved to the data collector in Flink CDC.

[0062] Furthermore, in one embodiment, step S40 includes:

[0063] The SQL statement with changed data structure and the database name with changed data structure are constructed into a Tuple2 object, and the Tuple2 object is saved to the data collector in Flink CDC.

[0064] In this embodiment, the processing is implemented by a log capture parser.

[0065] Step S50: sending a message notification based on the data structure change information in the data collector.

[0066] In this embodiment, the data structure change information in the data collector is notified via a message in a preset manner.

[0067] Furthermore, in one embodiment, step S50 includes:

[0068] Parse the Tuple2 object in the data collector to obtain the SQL statement in which the data structure is changed and the name of the database in which the data structure is changed;

[0069] The recipient information, notification method, SQL statement of the data structure change, and the database name of the data structure change are sent to the message notification platform, so that the message notification platform can send the SQL statement of the data structure change and the database name of the data structure change to the recipient corresponding to the recipient information according to the notification method.

[0070] In this embodiment, it can be implemented by processing the data structure change class SchemaChangeSin. Among them, the data structure change processing class inherits Flink's RichSinkFunction. Override the default open method and initialize the relevant settings of the notification center in the open method.

[0071] SchemaChangeSink overrides the invoke method, receives the Tuple2 object passed in from the data collector, parses the Tuple2 object to obtain the SQL statement for the data structure change and the database name for the data structure change. Then, the recipient information, notification method, SQL statement for the data structure change and the database name for the data structure change are sent to the message notification platform. The notification method can be one or more of many methods such as email, SMS, and APP notification.

[0072] After receiving this information, the message notification platform can send the SQL statement with the changed data structure and the database name with the changed data structure to the recipient corresponding to the recipient information in a notification manner.

[0073] In the embodiment of the present application, the log change record of the monitored data source is obtained based on the log capture tool in Flink CDC; the log change record is detected to see if it is a data structure change record; if it is a data structure change record, the log change record is parsed to obtain the data structure change information; the data structure change information is saved to the data collector in Flink CDC; and a message notification is made based on the data structure change information in the data collector. Through the embodiment of the present application, data structure change events can be automatically identified based on log change records, and message notifications are triggered, thereby realizing the monitoring of data structure changes of data sources; by automatically determining data structure change information, manual comparison of the structure of the data source and the data warehouse is avoided, and the data synchronization failure processing time is shortened.

[0074] Furthermore, in one embodiment, after step S50, the method further includes:

[0075] The data structure change information is recorded in a change information table.

[0076] In this embodiment, after the message notification is performed, the data structure change information is recorded in the change information table to facilitate subsequent search.

[0077] In a second aspect, an embodiment of the present application also provides a data structure change notification device.

[0078] In one embodiment, referring to Figure 2 , Figure 2 This is a functional module diagram of an embodiment of a data structure change notification device of the present application. Figure 2 As shown, the data structure change notification device includes:

[0079] The acquisition module 10 is used to obtain the log change records of the monitored data source based on the log capture tool in Flink CDC;

[0080] A detection module 20, used to detect whether the log change record is a data structure change record;

[0081] The parsing module 30 is used to parse the log change record to obtain data structure change information if it is a data structure change record;

[0082] A saving module 40 is used to save the data structure change information to the data collector in Flink CDC;

[0083] The notification module 50 is used to make message notification based on the data structure change information in the data collector.

[0084] Furthermore, in one embodiment, the detection module 20 is specifically used for:

[0085] Detect whether the keySchema in the log change record is a preset string;

[0086] If it is a preset character string, it is determined that the log change record is a data structure change record.

[0087] Furthermore, in one embodiment, the parsing module 30 is specifically configured to:

[0088] If it is a data structure change record, the historyRecord information in the log change record is parsed to obtain the SQL statement in which the data structure has changed and the name of the database in which the data structure has changed.

[0089] Furthermore, in one embodiment, the storage module 40 is specifically used for:

[0090] The SQL statement with changed data structure and the database name with changed data structure are constructed into a Tuple2 object, and the Tuple2 object is saved to the data collector in Flink CDC.

[0091] Further, in one embodiment, the notification module 50 is specifically used to:

[0092] Parse the Tuple2 object in the data collector to obtain the SQL statement in which the data structure is changed and the name of the database in which the data structure is changed;

[0093] The recipient information, notification method, SQL statement of the data structure change, and the database name of the data structure change are sent to the message notification platform, so that the message notification platform can send the SQL statement of the data structure change and the database name of the data structure change to the recipient corresponding to the recipient information according to the notification method.

[0094] Furthermore, in one embodiment, the data structure change notification device further includes a recording module, which is used to:

[0095] The data structure change information is recorded in a change information table.

[0096] Among them, the functional implementation of each module in the above-mentioned data structure change notification device corresponds to each step in the above-mentioned data structure change notification method embodiment, and its functions and implementation processes are no longer repeated here.

[0097] In a third aspect, an embodiment of the present application provides a data structure change notification device, which may be a personal computer (PC), a laptop computer, a server, or other device with data processing capabilities.

[0098] Reference Figure 3 , Figure 3 The hardware structure diagram of the data structure change notification device involved in the embodiment of the present application is shown in FIG. In the embodiment of the present application, the data structure change notification device may include a processor, a memory, a communication interface, and a communication bus.

[0099] The communication bus may be of any type and is used to interconnect the processor, the memory, and the communication interface.

[0100] The communication interface includes input / output (I / O) interface, physical interface and logical interface, etc., which are used to realize the interconnection of devices inside the data structure change notification device, and the interface used to realize the interconnection between the data structure change notification device and other devices (such as other computing devices or user devices). The physical interface can be an Ethernet interface, a fiber optic interface, an ATM interface, etc.; the user device can be a display, a keyboard, etc.

[0101] The memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0102] The processor may be a general-purpose processor, and the general-purpose processor may call the data structure change notification program stored in the memory and execute the data structure change notification method provided in the embodiment of the present application. For example, the general-purpose processor may be a central processing unit (CPU). The method executed when the data structure change notification program is called may refer to the various embodiments of the data structure change notification method of the present application, which will not be repeated here.

[0103] Those skilled in the art will understand that Figure 3 The hardware structure shown in the figure does not constitute a limitation on the present application, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0104] In a fourth aspect, an embodiment of the present application also provides a computer-readable storage medium.

[0105] A data structure change notification program is stored on a computer-readable storage medium of the present application, wherein when the data structure change notification program is executed by a processor, the steps of the data structure change notification method as described above are implemented.

[0106] Among them, the method implemented when the data structure change notification program is executed can refer to the various embodiments of the data structure change notification method of the present application, and will not be repeated here.

[0107] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0108] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit "first", "second" and "third" to different types.

[0109] In the description of the embodiments of the present application, "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a specific way.

[0110] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0111] In some processes described in the embodiments of the present application, multiple operations or steps that appear in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or in parallel, and the sequence number of the operation is only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.

[0112] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present application.

[0113] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A data structure change notification method, characterized in that: The data structure change notification method comprises: Obtain log change records of monitored data sources based on the log capture tool in Flink CDC; Detecting whether the log change record is a data structure change record; If it is a data structure change record, the log change record is parsed to obtain the data structure change information; The data collector that saves data structure change information to Flink CDC; Performing message notification based on the data structure change information in the data collector; The step of detecting whether the log change record is a data structure change record comprises: Detect whether the keySchema in the log change record is a preset string; If it is a preset character string, the log change record is determined to be a data structure change record; If it is a data structure change record, the steps of parsing the log change record to obtain the data structure change information include: If it is a data structure change record, the historyRecord information in the log change record is parsed to obtain the SQL statement in which the data structure has changed and the name of the database in which the data structure has changed; The step of saving the data structure change information to the data collector in Flink CDC includes: The SQL statement with changed data structure and the database name with changed data structure are constructed into a Tuple2 object, and the Tuple2 object is saved to the data collector in Flink CDC.

2. The data structure change notification method according to claim 1, characterized in that: The step of performing message notification based on the data structure change information in the data collector comprises: Parse the Tuple2 object in the data collector to obtain the SQL statement in which the data structure is changed and the name of the database in which the data structure is changed; The recipient information, notification method, SQL statement of the data structure change, and the database name of the data structure change are sent to the message notification platform, so that the message notification platform can send the SQL statement of the data structure change and the database name of the data structure change to the recipient corresponding to the recipient information according to the notification method.

3. The data structure change notification method according to claim 1 or 2, characterized in that: After the step of performing message notification based on the data structure change information in the data collector, the method further includes: The data structure change information is recorded in a change information table.

4. A data structure change notification device, characterized in that: The data structure change notification device comprises: The acquisition module is used to obtain log change records of the monitored data source based on the log capture tool in Flink CDC; A detection module, used to detect whether the log change record is a data structure change record; The parsing module is used to parse the log change record to obtain the data structure change information if it is a data structure change record; The saving module is used to save the data structure change information to the data collector in Flink CDC; A notification module, used for sending a message notification based on the data structure change information in the data collector; The detection module is specifically used for: Detect whether the keySchema in the log change record is a preset string; If it is a preset string, the log change record is determined to be a data structure change record; Parsing module, specifically used for: If it is a data structure change record, the historyRecord information in the log change record is parsed to obtain the SQL statement in which the data structure has changed and the name of the database in which the data structure has changed; Save module, specifically used for: The SQL statement with changed data structure and the database name with changed data structure are constructed into a Tuple2 object, and the Tuple2 object is saved to the data collector in Flink CDC.

5. A data structure change notification device, characterized in that: The data structure change notification device includes a processor, a memory, and a data structure change notification program stored in the memory and executable by the processor, wherein when the data structure change notification program is executed by the processor, the steps of the data structure change notification method as described in any one of claims 1 to 3 are implemented.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a data structure change notification program, wherein when the data structure change notification program is executed by a processor, the steps of the data structure change notification method according to any one of claims 1 to 3 are implemented.

Images