Multi-source heterogeneous JSON data stream processing method based on Flink SQL

Abstract

The invention discloses a multi-source heterogeneous JSON (JavaScript Object Notation) data stream processing method based on Flink SQL (Structured Query Language), which specifically comprises the following steps of: establishing a Kafka connection table, defining a Topic field and an original data field extracted from Kafka metadata in a table structure of the connection table, and starting a dynamic Schema mode; creating an independent structure context manager for each physical processing task instance of the Flink SQL, wherein the manager maintains structure contexts of different Topic in a key value pair form; analyzing the key value pair of each JSON, performing key name standardization processing, and executing newly added field addition or type conflict processing based on the Schema cache; recursively unfolding the object and the array type field, and generating an unfolding structure according to the maximum unfolding depth limitation and the position marking rule; constructing an output line according to the Schema cache, filling a missing field into a null value, and injecting a Kafka metadata field and an expansion position mark; and outputting Row objects in a uniform format to form a dynamic Schema data stream, and butting an Flink Table API (Application Program Interface) to support direct query of a dynamic extension field.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data stream processing, and mainly relates to a multi-source heterogeneous JSON data stream processing method based on Flink SQL. BACKGROUND

[0002] The Flink SQL under the prior art has obvious limitations in processing JSON data: firstly, the fixed Schema parsing mechanism requires complete definition of the data structure in advance, and when undefined fields or type changes occur in the input data, the system will throw an exception to cause job interruption; secondly, the nested structure processing seriously depends on the user-defined function (UDTF), and the developer needs to write complex logic to expand the JSON array, which not only increases the development burden, but also reduces the processing performance; thirdly, the key name compatibility is seriously missing, and the fields with the same semantics will be regarded as different fields due to the naming difference (such as orderId / order_id / ORDER_ID), causing data processing abnormalities; fourthly, when facing the multi-Topic heterogeneous data scene, the traditional scheme needs to create an independent physical task for each Topic, causing serious fragmentation of Slot resources. Although the industry proposes improvement schemes such as Schema Registry, the centralized architecture increases the operation and maintenance burden, and cannot solve the key problems of deep nested processing.

[0003] The above technical defects bring three business pain points: frequent job interruption causes data processing delay, high development and maintenance cost, and low resource utilization in multi-source scenarios. To solve these problems, the present application proposes three innovations: firstly, a dynamic structure perception engine is established, which automatically detects JSON structure changes at runtime, expands the virtual table structure when adding new fields, and performs safe type conversion (STRING compatible with all types) when type conflict occurs, realizing zero interruption of business; secondly, a in-place recursive expansion operator is designed, which automatically locates the nested array based on JSONPath and performs distributed row-level splitting, avoiding the performance loss of traditional UDTF; finally, a key name intelligent normalization system is built, which automatically unifies different naming formats through a rule engine and a historical mapping library. These improvements are combined with the multi-Topic physical task merging architecture to make the resource utilization rate increase to more than 85%. SUMMARY

[0004] The core of the present application is to complete the unified processing of multi-source heterogeneous JSON streams in a single Flink SQL task. According to the first aspect of the present application, a multi-source heterogeneous JSON data stream processing method based on Flink SQL is proposed, and the specific steps are as follows:

[0005] A Kafka connection table is established, and a Topic field extracted from Kafka metadata and an original data field are defined in a table structure of the connection table, and a dynamic Schema mode is started;

[0006] An independent structural context manager is created for each physical processing task instance of the Flink SQL, which maintains the structural context of different Topics in the form of key-value pairs;

[0007] The following parsing steps are performed on each JSON record entering the system:

[0008] S1, parse the JSON key-value pair, perform key name standardization processing, and perform new field addition or type conflict processing based on the Schema cache;

[0009] S2, recursively expand object and array type fields, and generate an expanded structure according to the maximum expansion depth limit and position marker rule;

[0010] S3, construct an output row according to the Schema cache, fill in missing fields with null values, and inject Kafka metadata fields and expansion position markers;

[0011] S4, output a Row object in a unified format to form a dynamic Schema data stream, and connect to the Flink Table API to support direct querying of dynamically expanded fields.

[0012] Preferably, the Kafka connection table uses a custom dynamic JSON format, subscribes to multiple Kafka Topics, and specifies the Topic list through a semicolon separator.

[0013] Preferably, in the structural context of different Topics, each structural context includes: a Schema cache, a key name normalization dictionary, and a type compatibility strategy set; the Schema cache is used to record the field names and corresponding types identified under the current Topic; the key name normalization dictionary is used to store mapping rules from original key names to standard key names, the mapping rules include conversion rules based on predefined regular expressions and historical mapping cache; the type compatibility strategy set is used to define processing rules when different types conflict.

[0014] Preferably, the initialization and update of the structural context are completed through a custom deserializer, specifically including:

[0015] Initialize shared resources such as type compatibility strategies in the open method of the deserializer;

[0016] When processing each Kafka message, extract the Topic to which the current message belongs according to Kafka metadata;

[0017] Maintain the mapping relationship between Topic and context in memory, create the corresponding structural context object when detecting a new Topic;

[0018] Call the structural context object during parsing to perform Schema cache update, key name normalization and type compatibility processing.

[0019] Preferably, the key name normalization process specifically includes:

[0020] Use regular expressions to remove illegal characters other than alphanumeric and underscores;

[0021] Convert the key name to lowercase uniformly;

[0022] Query the historical mapping dictionary, if there is a mapping rule, directly use it, otherwise save the new conversion result.

[0023] Preferably, the type compatibility strategy includes:

[0024] Automatic upgrade to higher precision type in numerical type conflict;

[0025] Incompatible type conflicts are uniformly converted into strings, and abnormal logs are recorded.

[0026] Preferably, in the process of recursively expanding the object and array type field, an independent record is generated for each array element when the array is expanded, and a position marker field is attached, which includes: _array_index_n, used to indicate the array index position; _parent_path, used to identify the parent field path.

[0027] Preferably, the recursive expansion of the array type field includes:

[0028] In the initial step, the first array is expanded to generate array elements and the corresponding index field _array_index_0, and the original field name is recorded as the parent path identifier;

[0029] In the recursive step, when it is detected that the array element is still an array type, the next array is expanded, a new index field is dynamically generated, and the current position is appended to the parent path;

[0030] In the expansion process, the index position marker and the parent path marker of each array element are retained to form a hierarchical unique location;

[0031] In the final output, filter out the intermediate array structure, only retain the non-array atomic elements, and output the index marker and parent path together.

[0032] According to a second aspect of the present application, a computer program product is provided, on which one or more computer programs are stored, which one or more computer programs implement the above method when executed by a computer processor.

[0033] The one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0034] The method solves the industry problem of Flink SQL processing heterogeneous JSON data, realizes real-time data processing without interruption through a dynamic structure perception mechanism, and makes the system have strong adaptability in the face of dynamic scenarios such as field increase and decrease, type change, etc. The key name intelligent normalization design fundamentally eliminates the data loss problem caused by naming differences, and the single-task multi-source processing architecture significantly improves resource utilization. The technology has excellent landing effect, and its high reliability has been verified in e-commerce, Internet of Things and other scenarios, supporting real-time processing of billions of data per day, and greatly reducing the development and operation complexity. In the future, it can continue to expand value in three aspects: combining with machine learning to predict data structure evolution trend and realize intelligent preprocessing; deploying lightweight parsing engine in edge computing scenarios to support massive heterogeneous device data access; and replicating the technology paradigm to SparkStreaming and other stream processing engines to promote the evolution of the real-time computing ecosystem to unified, efficient and easy-to-use, and provide core foundation support for digital transformation. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 A flowchart of a multi-source heterogeneous JSON data stream processing method based on Flink SQL is shown according to an embodiment of the present application. DETAILED DESCRIPTION

[0036] Figure 1 A flowchart of a multi-source heterogeneous JSON data stream processing method based on Flink SQL is shown according to an embodiment of the present application, as shown in Figure 1

[0037] 1. Data access layer;

[0038] Receive multiple Kafka Topics (such as Topic1, Topic2, Topic3), define KafkaSource table through Flink SQL, and access multiple Topics to a physical processing task; all data from different Topics first enter the unified access layer to form a data stream to be processed.

[0039] ​In Flink SQL, a custom dynamic JSON format ('format' ='multi-json') is used to define the Kafka connection table. The table can subscribe to multiple Kafka Topics at the same time, and the Topic list is specified by a semicolon separator. The table structure contains a topic field extracted from Kafka metadata (used to identify the data source) and an original data field, while enabling dynamic schema mode to adapt to subsequent structure changes.

[0040] In one embodiment of the application, by defining a Flink SQL physical table, multiple Kafka Topics (click_log and purchase_log) are unified to the same logical table multi_source_table, and the metadata field from Kafka is explicitly declared in the table, while enabling dynamic schema expansion mode. The example DDL is as follows:

[0041]

[0042]

[0043] In this embodiment:

[0044] The topic field directly extracts the source Topic of the Kafka message through METADATA FROM'topic', which is used to identify the data source;

[0045] The event_time field extracts the Kafka message timestamp through METADATA FROM'timestamp', and combines WATERMARK to define the event time waterline, which is used for subsequent event time semantic processing;

[0046] The data field is not predefined in the table structure, and the actual runtime is automatically extended by the custom dynamic JSON format parser to realize the parsing of dynamically changing fields under different Topics;

[0047] Multi-Topic subscription uses'topic' = 'click_log; purchase_log' to subscribe to multiple Kafka Topics at a time, forming a unified access portal.

[0048] Dynamic schema mode enables dynamic field expansion through'schema.dynamic' = 'true', so that newly appeared fields can be directly parsed and included in the schema cache.

[0049] Through the above method, unified access of multi-source data in Flink SQL job can be realized, and a standardized entrance is provided for subsequent dynamic parsing engine, schema cache update and nested structure expansion.

[0050] 2. Data shunting layer;

[0051] The data of the unified entrance is routed to different partition processing subtasks (partition 1, partition 2, and partition 3) according to its Kafka partition information or other rules, and each partition is independently processed by a corresponding Flink parallel subtask, ensuring data parallel consumption and processing efficiency.

[0052] 3. Dynamic parsing engine;

[0053] An independent structure context manager is created for each physical processing task (i.e., Flink SQL parallel instance), which is the dynamic parsing engine. The manager maintains the structure context (Context) of different Topics in the form of key-value pairs, and each Context includes the following components:

[0054] (1) Schema cache: records the recognized field names and types under the current Topic (such as user_id:STRING, amount:DOUBLE). The first time a Topic message is processed, it is initialized as an empty Schema, and it is dynamically expanded with data parsing.

[0055] (2) Key name normalization dictionary: stores the mapping rules of original key names to standard key names (such as OrderID→order_id). The rules come from:

[0056] 1) Predefined regular expressions (such as removing spaces, converting to lowercase, and converting camel case to underscore);

[0057] 2) Historical mapping cache (records previously processed key name mapping relationships).

[0058] (3) Type compatibility strategy set: defines the processing rules when different types conflict. Numerical type conflicts are automatically upgraded to higher precision types, and incompatible type conflicts are uniformly converted to strings, and abnormal logs are recorded (such as integer to float, numerical to string, etc.).

[0059] The initialization and update of the structure context are completed through a custom deserializer, which specifically includes:

[0060] Initialize shared resources such as type compatibility strategies in the open method of the deserializer;

[0061] When processing each Kafka message, the Topic to which the current message belongs is extracted according to Kafka metadata;

[0062] Maintain the mapping relationship between Topic and Context in memory, and create the corresponding structural Context object when a new Topic is detected;

[0063] Call the structural Context object during parsing to perform Schema cache update, key name normalization and type compatibility processing.

[0064] In an embodiment of the present application, the algorithm is implemented as follows:

[0065]

[0066] The structural Context is initialized in the open method of the deserializer, and is dynamically updated in subsequent processing by the following methods:

[0067]

[0068]

[0069] 4. Real-time parsing and dynamic structure adaptation process;

[0070] The following parsing steps are performed for each JSON record entering the system:

[0071] (1) Structure detection and cache update;

[0072] Parse all key-value pairs of the JSON object, identify the key name and the basic type of the corresponding value (string, integer, float, boolean, object, array, etc.).

[0073] Key name standardization processing:

[0074] 1) Use regular expression[^a-zA-Z0-9_] to remove illegal characters;

[0075] 2) Convert to all lowercase (e.g. UserId→user_id);

[0076] 3) Query the historical mapping dictionary: if there is a matching rule, directly use it (e.g. OrderID→order_id), otherwise save the new conversion result.

[0077] Compare the Schema cache in the current Context:

[0078] Add new fields to the cache with the current detected type; type conflicts are handled according to the type compatibility strategy, which includes:

[0079] Automatic upgrade of numerical type conflicts to higher precision types;

[0080] Incompatible type conflicts are unified into strings, and exception logs are recorded.

[0081] (2) Uniform output structure construction;

[0082] 1) Recursively traverse the JSON object, and generate independent records for each array element when the array is expanded, and append the position marker field, which includes: _array_index_n, indicating the array index position; _parent_path, identifying the parent field path.

[0083] 2) Expansion rules:

[0084] The maximum expansion depth is 5 levels by default;

[0085] In the initial step, the expansion operation is performed on the first-level array to generate array elements and the corresponding index field _array_index_0, and the original field name is recorded as the parent path identifier;

[0086] In the recursive step, when it is detected that the array element is still an array type, the next-level array is continued to be expanded, a new index field is dynamically generated, and the current position is appended to the parent path;

[0087] During the expansion process, the index position marker and the parent path marker of each array element are retained to form a unique location of the hierarchical structure;

[0088] In the final output, the intermediate array structure is filtered out, and only the atomic elements that are not arrays are retained, and the index marker and the parent path are output together.

[0089] 3) Example: The original record {"items":[{"id":"A1"},{"id":"A2"}]} is expanded into two rows:

[0090] {"_array_index_0":0,"items_id":"A1"}

[0091] {"_array_index_0":1,"items_id":"A2"}

[0092] (3) Nested structure expansion;

[0093] According to the latest Schema in Context, the output row is created, all fields are sorted in lexicographical order to ensure consistent structure, the existing fields in the current record are directly filled with converted numerical values, and the fields that exist in the Schema but are missing in the record are filled with NULL.

[0094] (4) Pass the unified data stream downstream;

[0095] 1) The final output Row object structure:

[0096]

[0097]

[0098] 2) Dynamic Schema stream docking Flink Table API:

[0099] The dynamically expanded fields can be directly queried, for example:

[0100]

[0101] In summary, the application is mainly applied to the real-time processing scene of multi-source heterogeneous JSON data in a big data platform. With the rapid development of the Internet of Things, e-commerce and financial technology and other fields, the JSON data streams generated by various systems show the characteristics of structural diversification, non-uniform format and frequent dynamic changes. This data heterogeneity is reflected in the aspects of field increase and decrease, nesting level difference, key name naming specification inconsistency (such as case difference, camel case and underscore mixed use) and the like. Although the traditional stream processing system such as Apache Flink provides the Flink SQL interface to simplify the development of stream computing tasks, its native JSON parsing capability has many limitations when facing heterogeneous data streams. The application aims to design a method for real-time and efficient processing of multiple data source heterogeneous JSON data streams in a single Flink SQL physical task, solve the core problems of structural dynamic change, key name normalization, multi-source merging processing and the like, and provide a unified data processing basis for real-time data warehouse, real-time risk control, Internet of Things monitoring and the like scenes.

Claims

1. A method for processing multi-source heterogeneous JSON data stream based on Flink SQL, characterized in that, The application relates to a dynamic schema data stream processing method based on Flink SQL and Kafka, and belongs to the technical field of big data processing. A Kafka connection table is established, a Topic field and original data fields are defined in the table structure of the connection table, and a dynamic schema mode is started; An independent structural context manager is created for each physical processing task instance of Flink SQL, which maintains the structural context of different Topics in the form of key-value pairs; The following parsing steps are performed on each JSON record entering the system: S1, parsing the JSON key-value pair, performing key name standardization processing, and performing new field addition or type conflict processing based on the Schema cache; S2, recursively expanding object and array type fields, and generating an expanded structure according to the maximum expansion depth limit and position marker rules; S3, constructing an output row according to the Schema cache, filling in missing fields with null values, and injecting Kafka metadata fields and expansion position markers; S4, outputting a uniform format Row object to form a dynamic schema data stream, and connecting to the Flink Table API to support direct query of dynamically expanded fields. 2.The method of claim 1, wherein, The Kafka connection table adopts a custom dynamic JSON format, subscribes to multiple Kafka Topics, and specifies the Topic list through a semicolon separator. 3.The method of claim 1, wherein, In the structural context of different Topics, each structural context includes a Schema cache, a key name normalization dictionary and a type compatibility strategy set; the Schema cache is used to record the identified field names and corresponding types under the current Topic; the key name normalization dictionary is used to store the mapping rules of original key names to standard key names, and the mapping rules include conversion rules based on a predefined regular expression and a historical mapping cache; and the type compatibility strategy set is used to define processing rules when different types conflict.

4. The multi-source heterogeneous JSON data stream processing method according to claim 1, wherein, Initialization and updating of the structural context are completed through a custom deserializer, and specifically include: Initializing shared resources including type compatibility strategies in the open method of the deserializer; Extracting the Topic to which the current message belongs according to Kafka metadata when processing each Kafka message; Maintaining a mapping relationship between Topics and contexts in the memory, and creating a corresponding structural context object when a new Topic is detected; Calling the structural context object to perform Schema cache updating, key name normalization and type compatibility processing during the parsing process.

5. The multi-source heterogeneous JSON data stream processing method according to claim 1, wherein, The key name standardization processing specifically includes: Removing illegal characters other than alphanumeric characters and underscores by using a regular expression; Converting the key name into a lowercase form; Querying the historical mapping dictionary, and directly adopting the mapping rules if the mapping rules exist, otherwise saving the new conversion result.

6. The multi-source heterogeneous JSON data stream processing method according to claim 3, wherein, The type compatibility strategies include: Upgrading conflicts between numerical types to higher precision types automatically; Converting incompatible types into strings uniformly, and recording exception logs.

7. The multi-source heterogeneous JSON data stream processing method according to claim 1, characterized in that, In the process of recursively expanding the object and array type fields, an independent record is generated for each array element during array expansion, and a position marker field is appended, which includes: _array_index_n, used to represent the array index position; and _parent_path, used to identify the parent field path.

8. The multi-source heterogeneous JSON data stream processing method according to claim 1, characterized in that, The recursive expansion of the array type field includes: In the initial step, an expansion operation is performed on the first-level array to generate array elements and a corresponding index field _array_index_0, and the original field name is recorded as the parent path identifier; In the recursive step, when it is detected that the array element is still of the array type, the next-level array is continuously expanded, a new index field is dynamically generated, and the current position is appended in the parent path; During the expansion process, the index position marker and the parent path marker of each array element are retained to form a hierarchical unique location; In the final output, the intermediate array structure is filtered out, only the atomic elements that are not arrays are retained, and the index marker and the parent path are output together.

Images