Flink CDC-based relational database table structure change early warning method and device

Abstract

The invention discloses a relational database table structure change early warning method and device based on Flink CDC. The method comprises the following steps: starting a Schema change capture function in a Source stage of the Flink CDC, and converting a captured log change event through JSON (JavaScript Object Notation) deserialization; the method comprises the following steps: identifying a table structure change event by customizing a ProcessFunction 1, analyzing a DDL keyword to identify a change type, and packaging a change event object; the method comprises the following steps of: associating business data to dynamically identify a key table by customizing ProcessFunction 2, defining a plurality of early warning levels in combination with a change type and attributes of the key table, and triggering notifications containing key information according to the levels. The method universally adapts to various relational databases, change recognition is fine, early warning is intelligently graded, real-time processing is achieved, seamless butt joint with a data pipeline is achieved, fault time is shortened, and consistency of a data source and a data intermediate station is guaranteed.

Description

Technical Field

[0001] This application belongs to the field of data management technology, and specifically relates to a method, apparatus, computer-readable storage medium, and electronic device for early warning of changes in the table structure of a relational database based on Flink CDC. Background Technology

[0002] In the process of enterprise digital transformation, data platforms such as data warehouses and data lakes have been widely deployed as core infrastructure for data-driven decision-making. Data consistency (especially data structure consistency) between data sources and data platforms is the fundamental guarantee for data synchronization and application. Traditional data synchronization mechanisms usually only passively detect changes after the synchronization task fails when data structure is adjusted, which can easily lead to data loss and quality degradation, thus affecting business analysis and decision-making. Even if changes are detected, they often rely on manual investigation, which is inefficient and prone to errors.

[0003] In the prior art, CN 117951149B discloses a data structure change notification method, which determines whether a data structure change has occurred by parsing FlinkCDC log change records and identifying specific keySchema fields. However, this solution still has many limitations: (1) Poor data source adaptability: It depends on specific keySchema fields (such as io.debezium.connector.mysql.SchemaChangeKey), so it is only applicable to MySQL and cannot meet the unified management needs of multiple relational databases (such as PostgreSQL, SQL Server, etc.); (2) Coarse granularity of change type identification: Although it can parse SQL statements, its ability to finely identify specific change types such as adding / deleting / modifying fields is obviously insufficient, making it difficult to support fine-grained early warning; (3) Low level of intelligence in early warning: It focuses on notification functions and cannot perform hierarchical early warning for change events (such as distinguishing between changes to key tables and non-key tables, assessing the potential impact of different change types on business, etc.).

[0004] In summary, developing a novel data structure change early warning method that is versatile, has high precision in identification, and provides intelligent early warning has become an industry requirement. Summary of the Invention

[0005] To address the aforementioned problems in existing technologies, this application proposes a novel early warning method for relational database table structure changes based on Flink CDC.

[0006] More precisely, this invention aims to address the problems of limited monitoring scope, imprecise identification of change types, and insufficient intelligence in early warning in existing technologies. It provides a table structure change early warning method and device based on Flink CDC, which can be universally adapted to multiple relational databases and can implement hierarchical early warning based on the content of the change and its business relevance, so as to achieve unified monitoring across multiple relational databases.

[0007] To achieve the above objectives, the present invention employs the following technical strategies: (1) Universality of multiple data sources: Based on the JSON field (op) to determine the key schema dependency of a specific database, it realizes unified schema change capture and early warning for multiple relational databases such as MySQL, PostgreSQL, and SQL Server.

[0008] (2) Refined change type identification: By parsing DDL keywords, various specific table structure change operations can be accurately identified, providing support for refined early warning and management.

[0009] (3) Real-time dynamic key table identification: Introduce the concept of key table, combine it with dynamic query mechanism to perceive business needs, and realize priority focus on key table changes.

[0010] (4) Intelligent hierarchical early warning mechanism: Combining change type and key table attributes, define three levels of early warning: CRITICAL, WARNING and INFO, linking differentiated notification methods to improve the effectiveness and timeliness of early warning.

[0011] (5) Integration with Flink stream processing platform: The entire process of schema change capture, parsing, identification and early warning is integrated into Flink stream processing tasks to achieve end-to-end real-time processing and seamless connection with existing data pipelines.

[0012] Specifically, this application provides the following technical solutions: The first aspect of this application provides a method for early warning of table structure changes in relational databases based on Flink CDC, such as... Figure 3 As shown, this method includes: S1. In the Source phase of Flink CDC, enable the database schema change capture function to obtain log change events containing data changes and table structure changes from the monitored relational database, and convert the captured log change events into JSON string format through the native JSON Debezium deserializer. S2. In the Flink data processing stage, table structure change events are identified by custom ProcessFunction 1. The identification includes determining whether the JSON message body contains specific fields to distinguish between data change and table structure change events, and extracting table structure change events. S3. Parse the table structure change event, extract the DDL statement, identify the specific table structure change type by parsing the DDL statement keywords, and encapsulate the database name, table name, change type and DDL statement as a change event object; S4. For change event objects, key tables are dynamically identified by associating business data through custom ProcessFunction 2, and multi-level early warning levels are defined by combining the change type of the change event object with the key table attributes; S5. Trigger the corresponding notification operation according to the warning level. The notification includes the database name, table name, change type, DDL statement and occurrence time.

[0013] Furthermore, in the method of this application, the specific field mentioned in step S2 is the op field; if the JSON message body contains the op field, it is determined to be a data change event; if it does not contain the op field, it is determined to be a table structure change event.

[0014] Furthermore, in the method of this application, the specific table structure change type mentioned in step S3 is identified by parsing DDL statement keywords, including at least one of creating a table, deleting a table, renaming a table, adding a field, deleting a field, renaming a field, and modifying the field type.

[0015] Furthermore, in the method of this application, the dynamic identification of key tables in step S4 includes: obtaining and defining key tables in real time by querying the table name whitelist of the business system, the configuration center, or analyzing the data characteristics of real-time data streams.

[0016] Furthermore, in the method of this application, the multi-level warning levels mentioned in step S4 include three levels: CRITICAL, WARNING, and INFO; wherein: The CRITICAL level corresponds to changes in the critical table, such as table deletion, table renaming, field deletion, field renaming, or field type modification. The WARNING level corresponds to changes in the addition of fields in critical tables, or changes in the deletion, renaming, deletion, renaming, or modification of field types in non-critical tables. INFO level corresponds to changes in the creation of a table, or changes in the addition of a field to a non-critical table.

[0017] Furthermore, in the method of this application, the notification operation in step S5 includes: sending high-priority notifications at the CRITICAL level via SMS, telephone, or instant messaging tools; sending medium-priority notifications at the WARNING level via email or instant messaging tools; and logging or sending notifications through low-priority systems at the INFO level.

[0018] Furthermore, the method of this application is integrated into the Flink stream processing platform, integrating the entire process of schema change capture, parsing, identification, and early warning into the Flink stream processing task to achieve end-to-end real-time processing.

[0019] Furthermore, the method in this application also includes: persistently storing the early warning information in a change log table of the database for auditing and tracing purposes.

[0020] The second aspect of this application provides a relational database table structure change early warning device based on Flink CDC. When the device operates, it implements the steps of the aforementioned relational database table structure change early warning method based on Flink CDC, such as... Figure 4 As shown, this device includes: The Acquisition module is used to enable the database schema change capture function in the Source phase of Flink CDC. It retrieves log change events containing data changes and table structure changes from the monitored relational database and converts the captured log change events into JSON string format through the native JSONDebezium deserializer. The identification module is used to identify table structure change events during the Flink data processing stage by using a custom ProcessFunction 1. The identification includes determining whether the JSON message body contains specific fields to distinguish between data change and table structure change events, and extracting table structure change events. The parsing module is used to parse the table structure change event, extract the DDL statement, identify the specific table structure change type by parsing the DDL statement keywords, and encapsulate the database name, table name, change type and DDL statement into a change event object; The early warning classification module is used to dynamically identify key tables by associating business data with custom ProcessFunction 2 for change event objects, and to define multiple early warning levels by combining the change type of the change event object with the attributes of the key tables; The notification sending module is used to trigger a corresponding notification operation based on the warning level. The notification includes the database name, table name, change type, DDL statement, and occurrence time.

[0021] A third aspect of this application provides an electronic device, including: a memory and a processor; Memory: Used to store computer programs; Processor: Used to execute the computer program to implement the steps of the aforementioned Flink CDC-based relational database table structure change early warning method.

[0022] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the aforementioned method for early warning of changes in relational database table structure based on Flink CDC.

[0023] In summary, compared with the prior art, the solution of the present invention has the following technical advantages: (1) High versatility and adaptability to multiple data sources: By judging whether the JSON message body contains the op field to distinguish between data changes and schema changes, it gets rid of the dependence on a specific key schema and can be widely adapted to various relational databases that support schema change log capture, such as MySQL, PostgreSQL, and SQL Server, thus improving the universality of the technology.

[0024] (2) Refined change identification: By parsing the keywords in the DDL statement, it is possible to accurately identify specific table structure change types such as ADD COLUMN, DROP COLUMN, RENAME_COLUMN, MODIFY_COLUMN_TYPE, providing a basis for accurate early warning and fault diagnosis.

[0025] (3) Improved flexibility and real-time performance: The identification and parsing of table structure change events are deployed in the data processing stage (ProcessFunction) of Flink jobs. Compared with the custom deserializer in the Source stage, it is easier to integrate with other data processing logic and can process and respond to schema changes in real time.

[0026] (4) Intelligent early warning classification: Combining dynamic identification of key tables and change types, define three levels of early warning: CRITICAL, WARNING, and INFO. Prioritize notification of serious changes that have a significant impact on business, reduce information overload, and ensure that critical changes are promptly addressed and processed.

[0027] (5) Shorten fault handling time and ensure data consistency: Through the real-time intelligent early warning mechanism, potential data structure inconsistency problems can be detected and notified in a timely manner, which helps operation and maintenance and development personnel to respond quickly, shorten the fault investigation and repair time, and ensure data consistency between data source and data platform.

[0028] Other features and advantages of this application will be set forth in detail in the following description, or will become apparent through the implementation of the relevant technical solutions of this application. The objectives and other advantages of this application can be achieved through the technical features and means explicitly pointed out in the description, claims, and drawings, and will be obtained through the implementation of these technical contents. Attached Figure Description

[0029] To more clearly illustrate the technical solution of this application, the accompanying drawings involved in the description of this invention will be briefly introduced below. It should be noted that the drawings only show some embodiments of this application. For those skilled in the art, other related drawings can be derived from these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of the overall design architecture of the proposed solution.

[0031] Figure 2 This is a schematic diagram illustrating the specific execution flow of the embodiments of this application.

[0032] Figure 3 This is a flowchart illustrating the overall implementation of the relational database table structure change early warning method based on Flink CDC in this application.

[0033] Figure 4 This is a structural diagram of the relational database table structure change early warning device based on Flink CDC in this application.

[0034] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be noted that the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0036] In this document, the term "comprising" and any variations thereof (such as "including," "including," etc.) are open-ended expressions and should be understood as "including but not limited to," meaning that the listed content is not exhaustive and may include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on," meaning that the basis or condition referred to may not be the only factor and may involve other relevant factors. The term "one embodiment" should be understood as "at least one embodiment," meaning that the described embodiment is not the only possible implementation, and other similar embodiments may exist.

[0037] In this application, the terms "a" and "a plurality of" are used to modify related elements or features, and their expression is illustrative rather than restrictive. Unless otherwise expressly stated in the context, "a" should be understood as "at least one," and "a plurality of" should be understood as "at least two." Those skilled in the art should reasonably interpret these terms based on the semantic and logical relationships of the context to ensure that they cover the possibility of "one or more."

[0038] Figure 1 The figure shows a schematic diagram of the overall design architecture of the present invention. The solution includes the following implementation process: 1. Data capture and preliminary parsing (Source phase): In the Source phase of Flink CDC, configure the database connection information and enable the schema change capture feature (e.g., by includingSchemaChanges(true)).

[0039] Using the native JSON Debezium deserializer, log change events (including data changes and schema changes) captured from the monitored data source are converted into JSON string format.

[0040] 2. Identification and analysis of table structure change events (Data Processing phase - ProcessFunction 1): During the data processing phase of a Flink job, a custom ProcessFunction operator 1 is used to identify and parse table structure change events.

[0041] (1) In the processElement method, receive change events in JSON string format.

[0042] (2) Event type judgment: Determine whether the received JSON message body contains the 'op' field: If the event contains the 'op' field (e.g., 'op': 'r', 'c', 'u', 'd', etc.), it is considered a data change event and is either ignored or subjected to other data processing. If the 'op' field is not included, it is determined to be a table structure change event and further processing will be performed.

[0043] (3) DDL statement and change type parsing: Extract the historyRecord field from the JSON message body of the table structure change event, and parse the DDL statement from it.

[0044] Identify specific table structure change types by scanning keywords in DDL statements (such as CREATE TABLE, DROP TABLE, RENAME TABLE, ADDCOLUMN, DROP COLUMN, CHANGE COLUMN, RENAME COLUMN, MODIFY COLUMN TYPE, etc.).

[0045] Define a ChangeType enumeration: CREATE_TABLE creates a table; DROP_TABLE deletes a table; RENAME_TABLE renames the table; Add a new field to ADD_COLUMN; DROP_COLUMN deletes the field; RENAME_COLUMN renames the field; MODIFY_COLUMN_TYPE modifies the field type; And assign the corresponding ChangeType to the change event based on the parsed keywords.

[0046] (4) Change object encapsulation: Extract the database name and table name from the source and historyRecord fields of the JSON message body.

[0047] When the DDL statement is ADD COLUMN, DROP COLUMN, RENAME_COLUMN, or MODIFY_COLUMN_TYPE, the affected field names are further parsed out.

[0048] The extracted database name, table name, field list (if any), DDL statement, and ChangeType are encapsulated into a SchemaChangeEvent object.

[0049] The SchemaChangeEvent object is sent to the downstream operator via the Collector.

[0050] 3. Key Table Identification and Early Warning Classification (Data Processing Stage - ProcessFunction 2): During the data processing phase of a Flink job, another custom ProcessFunction operator 2 is used to identify critical tables and perform early warning classification.

[0051] (1) In the processElement method, receive the SchemaChangeEvent object from ProcessFunction operator 1.

[0052] (2) Dynamic identification of key tables: In ProcessFunction operator 2, data tables that the business system is using or is interested in are obtained in real time through dynamic queries (e.g., querying the table name whitelist provided by the business system, the configuration center, or analyzing data characteristics in the real-time data stream).

[0053] Define these identified tables as key tables.

[0054] Data tables that are not identified as key tables are classified as non-key tables.

[0055] (3) Warning Level Definition: Based on the change type in SchemaChangeEvent and whether the table is a critical table, the following three warning levels are defined: (a) CRITICAL: The following changes have been made to the critical tables: DROP_TABLE, RENAME_TABLE, DROP_COLUMN, RENAME_COLUMN, and MODIFY_COLUMN_TYPE.

[0056] (b) WARNING: The critical table has undergone an ADD_COLUMN change.

[0057] The following changes occurred to non-critical tables: DROP_TABLE, RENAME_TABLE, DROP_COLUMN, RENAME_COLUMN, and MODIFY_COLUMN_TYPE.

[0058] (c) INFO (hint): CREATE_TABLE changes (regardless of whether the table is critical or non-critical).

[0059] A non-critical table has undergone an ADD_COLUMN change.

[0060] (4) Tiered early warning operation: Based on the determined warning level, trigger the corresponding warning action: (a) CRITICAL level: Send high-priority notifications to designated recipients via SMS, telephone, instant messaging tools (such as DingTalk, Slack, etc.).

[0061] (b) WARNING level: Send medium-priority notifications to designated recipients via email, instant messaging tools, etc.

[0062] (c) INFO level: Record the change information in the log file, or send it to the relevant parties via low-priority system notification or email.

[0063] The warning notification should include key information such as database name, table name, change type, DDL statement, and time of occurrence.

[0064] To more clearly illustrate the technical solution of this application, the following will provide further explanation through specific scenario embodiments.

[0065] Example 1: A Relational Database Table Structure Change Early Warning Method Based on Flink CDC Figure 2 The figure shows a schematic diagram of the specific execution flow of the solution in this embodiment, which includes the following process: 1. Source Connector Configuration: Use the corresponding database connector for Flink CDC (such as flink-connector-mysql-cdc, flink-connector-postgres-cdc, flink-connector-sqlserver-cdc, etc.).

[0066] In the connector configuration, set common parameters such as databaseList, tableList, username, password, hostname, and port.

[0067] Key configuration: Enable schema change capture, for example, by setting includeSchemaChanges(true).

[0068] Deserializer: Configured to use JsonDebeziumDeserializationSchema to convert change logs (including schema changes) captured by Debezium into JSON format strings.

[0069] 2. SchemaChangeEvent Identification and Parsing (ProcessFunction 1): Define a ProcessFunction named SchemaChangeParserFunction.

[0070] This function accepts a String (JSON string) as input and outputs a SchemaChangeEvent object.

[0071] In processElement(String jsonMessage, Context ctx, Collector <schemachangeevent>In the `out` method: (1) Use a JSON parsing library (such as Jackson or Gson) to parse jsonMessage.

[0072] (2) Check the "op" field: Check if the JSON object contains a field with the key "op": (a) If the "op" field exists, it indicates a data change, so return directly; (b) If the "op" field does not exist, it indicates a schema change.

[0073] (3) Parse historyRecord: Extract the value of the "historyRecord" field, which is a JSON string.

[0074] (4) Parse DDL: Perform secondary parsing on the JSON in historyRecord, find the "ddl" field, and obtain the DDL statement.

[0075] (5) Identify ChangeType: Use string matching or regular expressions to determine ChangeType based on keywords in the DDL statement.

[0076] For example: if (ddl.contains("CREATE TABLE")) { changeType =ChangeType.CREATE_TABLE;} Another example: if (ddl.contains("ALTER TABLE")&&ddl.contains("ADD COLUMN")){ changeType = ChangeType.ADD_COLUMN;} (6) Parse the database name and table name: obtain them from the "db" and "table" fields of the "source" object.

[0077] (7) Construct a SchemaChangeEvent object and call out.collect(schemaChangeEvent).

[0078] 3. Key Table Identification and Early Warning Classification (ProcessFunction 2): Define another ProcessFunction and name it SchemaChangeAlertingFunction.

[0079] This function receives a SchemaChangeEvent as input.

[0080] Key table management: (1) Design a configuration class KeyTableManager to be responsible for loading and managing the list of key tables.

[0081] (2) KeyTableManager can dynamically update the list of key tables from external files (such as application.properties), configuration services (such as Nacos Configuration Center) or database queries.

[0082] (3) Provide the isKeyTable(String databaseName, String tableName) method to identify key tables.

[0083] In processElement(SchemaChangeEvent event, Context ctx, Collector <alertmessage>In the `out` method: (a) Call KeyTableManager.isKeyTable(event.getDatabaseName(),event.getTableName()) to determine if it is a key table.

[0084] (b) Determine the alert level AlertLevel based on event.getChangeType() and whether it is a critical table.

[0085] (c) Construct the AlertMessage warning message.

[0086] (d) Based on the AlertLevel, determine whether to send a notification, and the type and priority of the notification.

[0087] 4. Early warning notification and persistence: You can integrate a message sending SDK (such as an SMS SDK or an email SDK) within the SchemaChangeAlertingFunction, or send the AlertMessage to another Sink Function, which will then be responsible for sending the notification.

[0088] Notification platform integration: Send AlertMessage to a unified message notification platform (such as an email server, SMS service provider, WeChat API, DingTalk API, etc.).

[0089] Persistence: AlertMessages can be recorded in the database change log table for traceability and auditing.

[0090] Example 2: A relational database table structure change early warning device based on Flink CDC This device is used to implement the aforementioned method for early warning of table structure changes in relational databases based on Flink CDC. This device includes: The Acquisition module is responsible for configuring and starting the Flink CDC Source, connecting to the monitored relational database, and capturing raw log change records.

[0091] Parsing and Identification Module: Contains the SchemaChangeParserFunction logic, receives the JSON string captured by the Source, identifies the Schema change event, and parses out the SchemaChangeEvent object, including the database name, table name, DDL statement, change type, and affected fields.

[0092] The alert grading module includes the SchemaChangeAlertingFunction logic, which receives SchemaChangeEvents, dynamically identifies key tables, determines the alert level (CRITICAL, WARNING, INFO) based on the change type and the importance of the table, and generates an alert message (AlertMessage).

[0093] Notification sending module (can be integrated into the early warning classification module or set up independently): responsible for sending AlertMessage to the designated recipient through preset notification methods (SMS, email, IM, etc.).

[0094] Key table management module (can be integrated into the early warning classification module or set up independently): responsible for dynamically loading, updating and querying key table information.

[0095] Persistence module: Responsible for recording AlertMessage or raw SchemaChangeEvent to the change information table in the database.

[0096] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of apparatus, methods, and computer program products according to various embodiments of this application, including architecture, functionality, and operation. In these figures, each block may represent a module, program segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should be noted that each block in the block diagrams and / or flowcharts, and combinations thereof, can be implemented using either a dedicated hardware-based system or a combination of dedicated hardware and computer instructions to achieve the specified function or operation.

[0097] like Figure 5 As shown, embodiments of this application also disclose an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing a processor-executable computer program, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 executes the executable computer program to implement the steps of the aforementioned Flink CDC-based relational database table structure change early warning method.

[0098] It is understood that, in addition to memory and a processor, this electronic device may also include input devices (such as a keyboard), output devices (such as a display), and other communication modules. These input devices, output devices, and other communication modules all communicate with the processor through I / O interfaces (i.e., input / output interfaces).

[0099] The operations described in this application can be implemented by writing computer program code using one or more programming languages ​​or a combination thereof. The programming languages ​​include, but are not limited to, the following types: Object-oriented programming languages, such as Java, Smalltalk, C++, etc. Conventional procedural programming languages, such as "C" or similar programming languages.

[0100] The execution methods of program code include, but are not limited to: It runs entirely on the user's computer; Part of it executes on the user's computer, and part of it executes on a remote computer; Execute as a standalone software package; It is executed entirely on a remote computer or server.

[0101] In scenarios involving remote computers, the remote computer can connect to the user's computer via any type of network, including but not limited to local area networks (LANs) or wide area networks (WANs). Furthermore, the remote computer can also connect to external computers through an internet service provider, for example, by utilizing the internet for connection.

[0102] Furthermore, this application also discloses a computer-readable storage medium, which, when the instructions in the computer-readable storage medium are executed by the processor of an electronic device, enables the electronic device to perform the various steps of the relational database table structure change early warning method based on FlinkCDC disclosed in this application.

[0103] In the context of this application, a computer-readable storage medium refers to a tangible medium capable of storing computer program code and related data. Specific examples include, but are not limited to, the following: (1) Portable computer disk: such as floppy disks and other removable magnetic storage media.

[0104] (2) Hard disk: including mechanical hard disks and solid-state hard disks and other fixed storage devices.

[0105] (3) Random Access Memory (RAM): A volatile storage medium used for temporary storage of data and program code.

[0106] (4) Read-only memory (ROM): a non-volatile storage medium used to store fixed programs and data.

[0107] (5) Erasable programmable read-only memory (EPROM) or flash memory: non-volatile storage media that supports multiple erasures and reprogrammings.

[0108] (6) Fiber optic storage devices: storage media based on fiber optic technology.

[0109] (7) Portable compact disc read-only memory (CD-ROM): a read-only medium that stores data in the form of an optical disc.

[0110] (8) Optical storage devices: such as DVDs, Blu-ray discs and other storage media based on optical principles.

[0111] (9) Magnetic storage devices: such as magnetic tapes, disks and other storage media based on magnetic principles.

[0112] (10) Any suitable combination of the above: for example, combining multiple storage media to meet different storage needs.

[0113] These computer-readable storage media can be used to store the program code and related data described in this application to support program execution and persistent data storage.

[0114] Specifically, according to embodiments of this application, the processes described in the flowcharts can be implemented as computer software programs. For example, embodiments of this application relate to a computer program product comprising a computer program carried on a non-transitory computer-readable medium. This computer program includes program code for executing the Flink CDC-based relational database table structure change early warning method disclosed in this application. When the computer program is executed by a processing device, it can achieve the functions defined in the embodiments of this application.

[0115] While the foregoing discussion contains several specific implementation details, these details should not be construed as limiting the scope of this application. The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features. Furthermore, this application should also cover other technical solutions formed by any combination of the above-described technical features or their equivalents without departing from the foregoing disclosed concept.

[0116] Those skilled in the art should also understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features, without departing from the spirit and scope of the technical solutions of the embodiments of this application. These modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of this application.< / alertmessage> < / schemachangeevent>

Claims

1. A Flink CDC-based relational database table structure change early warning method, characterized in that, The method comprises: S1, in the Source stage of Flink CDC, enable the database Schema change capture function, obtain the log change event containing data change and table structure change from the monitored relational database, and convert the captured log change event into a JSON string format through a native JSON Debezium deserializer; S2, in the Flink data processing stage, identify table structure change events through a self-defined ProcessFunction 1, the identification includes judging whether the JSON message body contains a specific field to distinguish data change and table structure change events, and extracting table structure change events; S3, parse the table structure change event, extract the DDL statement, identify the specific table structure change type through the analysis of the DDL statement keyword, and encapsulate the database name, table name, change type and DDL statement into a change event object; S4, for the change event object, dynamically identify the key table through a self-defined ProcessFunction 2, define multiple levels of warning levels according to the change type of the change event object and the key table attributes; S5, trigger the corresponding notification operation according to the warning level, the notification includes the database name, table name, change type, DDL statement and occurrence time.

2. The method of claim 1, wherein, The specific field in step S2 is the op field; if the JSON message body contains the op field, it is determined as a data change event; if it does not contain the op field, it is determined as a table structure change event.

3. The method of claim 1, wherein, The specific table structure change type in step S3 is identified by analyzing the DDL statement keyword, including at least one of creating a table, deleting a table, renaming a table, adding a field, deleting a field, renaming a field, and modifying a field type.

4. The method of claim 1, wherein, In step S4, the dynamic identification of the key table includes: querying the table name whitelist of the business system, the configuration center or analyzing the real-time data stream data characteristics, and defining the key table in real time.

5. The method of claim 1, wherein, The multiple levels of warning levels in step S4 include CRITICAL, WARNING and INFO levels; wherein: The CRITICAL level corresponds to the deletion of a table, renaming a table, deleting a field, renaming a field or modifying a field type change of a key table; The WARNING level corresponds to the addition of a field change of a key table, or the deletion of a table, renaming a table, deleting a field, renaming a field or modifying a field type change of a non-key table; The INFO level corresponds to the creation of a table change, or the addition of a field change of a non-key table.

6. The method of claim 1, wherein, The notification operation in step S5 includes: the CRITICAL level sends a high-priority notification through a short message, a telephone or an instant messaging tool; the WARNING level sends a medium-priority notification through an email or an instant messaging tool; and the INFO level records a log or sends a notification through a low-priority system.

7. The method of claim 1, wherein, The method is integrated in the Flink stream processing platform, and the Schema change capture, analysis, identification and warning are integrated in the Flink stream processing task to realize end-to-end real-time processing.

8. The method of claim 1, wherein, In the method, the warning information is also stored in the change log table of the database for audit and trace.

9. A Flink CDC-based relational database table structure change early warning device, characterized in that, The device realizes the steps of the Flink CDC-based relational database table structure change early warning method as claimed in any one of claims 1-8 when the device is running, and the device comprises: An acquisition module is configured to, in a Source stage of Flink CDC, enable a database Schema change capture function, acquire log change events containing data changes and table structure changes from a monitored relational database, and convert the captured log change events into a JSON string format through a native JSONDebezium deserializer. An identification module is configured to, in a Flink data processing stage, identify table structure change events through a custom ProcessFunction 1, wherein the identification includes judging whether a JSON message body contains a specific field to distinguish data changes from table structure change events and extracting table structure change events. An analysis module is configured to analyze the table structure change events, extract DDL statements, identify specific table structure change types through analysis of DDL statement keywords, and encapsulate a database name, a table name, a change type, and a DDL statement into a change event object. An early warning grading module is configured to, for the change event object, dynamically identify key tables through a custom ProcessFunction 2 in association with business data, define multiple early warning levels in combination with the change type of the change event object and the properties of the key tables. A notification sending module is configured to trigger corresponding notification operations according to the early warning levels, and the notification includes a database name, a table name, a change type, a DDL statement, and a time of occurrence.

10. An electronic device, comprising: comprise: a memory and a processor; the memory is configured to store a computer program; the processor is configured to execute the computer program to realize the steps of the Flink CDC-based relational database table structure change early warning method as claimed in any one of claims 1-8.

Images