A flink-based unified stream-batch integrated data engine design method

Abstract

This invention belongs to the field of integrated stream and batch computing technology, specifically relating to a design method for a unified stream and batch integrated data engine based on Flink. This invention achieves unified management and scheduling of stream processing, batch processing, and mixed scenarios through business tag-driven access and computing task identification. It eliminates the scenario adaptation problems caused by the generalization of traditional Flink stream and batch processing strategies. Data fingerprints are generated at the parsing, execution, and target layers, and a data fingerprint chain is constructed to achieve full-link tracking of data field structure, parsing rules, and processing procedures, improving data integrity and traceability. Real-time monitoring of the running status is provided, and dynamic parameter adjustment strategies are generated in conjunction with business tags, enabling the system to optimize resource allocation, processing latency, and throughput according to different business processing needs, thereby improving the execution efficiency and stability of stream and batch tasks.

Description

Technical Field

[0001] This invention belongs to the field of integrated stream and batch computing technology, specifically involving a design method for a unified stream and batch integrated data engine based on Flink. Background Technology

[0002] With the rapid development of the Internet, the Internet of Things, and enterprise informatization, various application systems, smart terminals, and sensing devices continuously generate massive amounts of multi-source data with diverse structures. This data includes, but is not limited to: user behavior logs, real-time sensor monitoring data, transaction records, equipment operation data, social media information, and enterprise business data. The data sources are wide-ranging, the formats are complex, and the update speed is rapid, including both continuously generated real-time streaming data and periodically or batch-generated historical data.

[0003] In existing systems, different types of data are typically stored in different databases, data warehouses, or stream processing systems, resulting in significant differences in data structures, field naming, data types, granularity, and business semantics. Furthermore, as enterprise business scenarios become increasingly complex, different computing tasks have varying requirements for data processing, including real-time analysis, batch statistics, hybrid stream and batch processing, and complex event processing.

[0004] Existing technologies often suffer from problems such as low data access efficiency, static parsing rules, separation of stream and batch processing, difficulty in state tracking, and insufficient dynamic optimization capabilities when dealing with massive, multi-source, and heterogeneous data. These issues make it difficult to meet the comprehensive needs of modern enterprises for data real-time performance, accuracy, and scalability. Summary of the Invention

[0005] The purpose of this invention is to provide a design method for a unified stream-batch integrated data engine based on Flink, which can perform unified access and management, flexible parsing and dynamic processing of multi-source data, and support data processing and intelligent decision-making needs in complex business scenarios.

[0006] The specific technical solution adopted by this invention is as follows: A design method for a unified stream-batch integrated data engine based on Flink, comprising: Configure business tags for the data sources to be processed, and bind the business tags to the corresponding data sources. Based on the data sources bound to the business tags, obtain the data collection method and the computing tasks to be executed, and connect the data sources to the unified stream and batch computing engine based on Flink through the collection method to form the raw data to be processed. For the raw data and computing tasks to be executed that are collected through acquisition, data parsing processing is performed according to the bound business tags to convert the raw data into parsed data. During the data parsing process, parsed metadata is generated. Based on parsing metadata, extract field structure and parsing rule features to generate parsing layer data fingerprints; Receive parsed data and computation tasks, identify the task type based on the bound business tags, convert the computation tasks into logical execution plans based on the task type identification results, and generate a unified batch execution plan; Load the unified execution plan for streaming and batch processing in the Flink runtime environment, perform data processing on the parsed data, and collect operator processing rules and field flow information in the unified execution plan during the data processing process to generate execution layer data fingerprints. The processing results obtained from data processing are written to the target storage. After the processing results are written, the field structure information of the processing results in the target storage is collected to generate the target layer data fingerprint. The parsing layer data fingerprint, the execution layer data fingerprint, and the generated target layer data fingerprint are sequentially associated according to the data processing order to construct a data fingerprint chain. Based on the data fingerprint chain, the consistency of the data processing process is verified to locate the data processing stage that causes inconsistency. Based on the bound business tags, the operational status of data access, data parsing, and data processing is monitored. Based on the monitoring results, the data processing parameters of the bound business tags are adjusted to optimize the operational effect under different business processing requirements.

[0007] In a preferred embodiment, business tags are configured for the data sources to be processed, and the business tags are bound to the corresponding data sources. Based on the data sources bound to the business tags, the data acquisition method and the computational tasks to be executed are obtained. The data sources are then connected to a unified stream-batch computing engine based on Flink through the acquisition method to form the raw data to be processed, including: Configure business tags for the data sources to be processed. The business tags are used to characterize business processing requirements, data structure characteristics and processing methods. The business tags are then bound to the corresponding data sources and stored to generate tagged data source information. Based on the processing requirements and data characteristics indicated by the business tags in the tagged data source information, the data collection method of the data source is obtained, and the computing tasks to be executed corresponding to the data source are obtained. Based on the data acquisition method, the data from the data source is connected to the unified stream and batch computing engine based on Flink to generate the raw data for the computing tasks to be executed.

[0008] In a preferred embodiment, the raw data acquired through data collection and the computational task to be executed are processed by performing data parsing based on the bound business tags. This process converts the raw data into parsed data. During data parsing, parsed metadata that satisfies the computational task is generated, including: Based on the accessed raw data, the computing tasks to be executed, and the business tags, obtain the parsing rules and parsing structure of the raw data; Based on the parsing rules and parsing structure, the original data is parsed and processed. The parsing process includes field splitting, type identification, and structure mapping, transforming the original data into parsed data that meets the processing requirements of the computing task. During the parsing and processing of the raw data, parsing metadata is generated synchronously based on the business tags and computational requirements of the computational tasks. The parsing metadata is used to describe the parsed data structure, field semantics, data granularity, and computational constraint information.

[0009] In a preferred embodiment, based on parsed metadata, field structure and parsing rule features are extracted to generate a parsing layer data fingerprint, including: Based on the parsed data and the corresponding parsed metadata, obtain the field structure information, field type information and parsing rule characteristics of the parsed data, and obtain the business tag information and the characteristics of the computation task to be executed corresponding to the parsed data; The field structure information, field type information, and parsing rule features are associated and integrated with business tag information and calculation task features, and feature identification information is generated to represent the overall state of the parsed data according to the preset feature organization rules. During the data access and parsing process, the data source identification information, data processing batch information, and parsing generation time corresponding to the parsed data are obtained. The feature identification information is associated with the data source identification information, data processing batch information, and parsing generation time and recorded as a data fingerprint of the parsing layer and stored.

[0010] In a preferred embodiment, the system receives parsed data and computation tasks. Based on the bound business tags, it identifies the task type of the computation tasks. Then, based on the task type identification results, it uniformly converts the computation tasks into logical execution plans and generates a unified stream and batch execution plan, including: Receive parsed data and the corresponding computational tasks to be executed, and obtain the business tag information bound to the computational tasks; Based on the processing mode indicated by the business tag, the computing task is identified by task type to obtain the task type identification result, which includes stream processing mode, batch processing mode or stream-batch hybrid processing mode. Based on the task type identification results, the computation task is converted into a logical execution description that is compatible with the processing method, and a unified stream-batch execution plan is generated based on the logical execution description for execution in the Flink-based unified stream-batch computation engine.

[0011] In a preferred embodiment, a unified streaming and batch execution plan is loaded into the Flink runtime environment to perform data processing on the parsed data. During data processing, operator processing rules and field transition information from the unified streaming and batch execution plan are collected to generate execution layer data fingerprints, including: Load the unified execution plan for streaming and batch processing in the Flink-based runtime environment, and perform data processing operations on the parsed data according to the unified execution plan for streaming and batch processing. During the execution of data processing operations, the operator configuration content used to describe the data processing logic in the unified execution plan of batch processing is obtained, and the field flow information of the data in each processing stage is parsed. Based on the operator configuration content and field flow information, an execution layer data fingerprint is generated.

[0012] In a preferred embodiment, the processing result obtained from data processing is written to the target storage. After the processing result is written, the field structure information of the processing result in the target storage is collected to generate a target-layer data fingerprint, including: Write the processing results obtained from data processing to the target storage according to the preset data writing rules; After the processing result is written, the field name and field type information corresponding to the processing result are read from the target storage; Generate target layer data fingerprints based on field names and field type information.

[0013] In a preferred embodiment, the parsing layer data fingerprint, the execution layer data fingerprint, and the generated target layer data fingerprint are sequentially associated according to the data processing order to construct a data fingerprint chain. Based on this data fingerprint chain, consistency checks are performed on the data processing process to pinpoint the stages where inconsistencies occur, including: The parsing layer data fingerprint, the execution layer data fingerprint, and the target layer data fingerprint are linked together in a chain according to the processing order of parsing, operator execution, and result writing to construct a data fingerprint chain; Based on the field structure characteristics and processing rule characteristics between adjacent data fingerprints in the data fingerprint chain, the consistency of data at different processing stages is verified step by step. When an inconsistency is detected between adjacent data fingerprints, the specific data processing stage that caused the inconsistency is located based on the position of the data fingerprint corresponding to the inconsistency in the data fingerprint chain.

[0014] In a preferred embodiment, the operational status of data access, data parsing, and data processing is monitored based on the bound business tags. Based on the monitoring results, the data processing parameters for the bound business tags are adjusted to optimize the operational performance under different business processing requirements, including: Based on the bound business tags, the operational status of the data access phase, data parsing phase, and data processing phase is continuously monitored to obtain operational status parameters; The running status parameters are compared with the preset processing targets corresponding to the business tags to generate adjustment instructions for the data processing parameters; According to the adjustment instructions, the data processing parameters corresponding to the business tags are dynamically adjusted so that the running status of the data processing process meets the processing objectives corresponding to the business tags.

[0015] And, a design terminal for a unified stream-batch integrated data engine based on Flink, including: One or more processors; A storage device on which one or more programs are stored; When one or more programs are executed by one or more processors, the one or more processors implement a unified stream-batch integrated data engine design method based on Flink.

[0016] The technical effects achieved by this invention are as follows: This invention achieves unified management and scheduling of streaming, batch, and hybrid scenarios by driving access and identifying computational tasks through business tags. It eliminates the scenario adaptation problems caused by the generalization of traditional Flink streaming and batch processing strategies. Data fingerprints are generated at the parsing, execution, and target layers, and a data fingerprint chain is constructed to achieve full-link tracking of data field structure, parsing rules, and processing procedures, improving data integrity and traceability. Real-time monitoring of the running status is provided, and dynamic parameter adjustment strategies are generated in conjunction with business tags. This allows the system to optimize resource allocation, processing latency, and throughput according to different business processing needs, improving the efficiency and stability of streaming and batch task execution. The data fingerprint chain and consistency verification mechanism can automatically locate abnormal stages in data processing, supporting backtracking analysis and correction, reducing the risk of data loss or computational errors, and enhancing system reliability and auditing capabilities. The system design allows for visual operation of business tags, parsing rules, and execution plan configuration, reducing technical barriers and facilitating rapid expansion and maintenance. Attached Figure Description

[0017] Figure 1 This is a flowchart of the method provided by the present invention. Detailed Implementation

[0018] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0019] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0020] Secondly, the term "an embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in a preferred embodiment" appearing in different places throughout this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that mutually excludes other embodiments.

[0021] Furthermore, the present invention will be described in detail with reference to the schematic diagrams. When describing the embodiments of the present invention in detail, the schematic diagrams are merely examples for ease of explanation and should not limit the scope of protection of the present invention.

[0022] Please see the appendix Figure 1 As shown, a design method for a unified stream-batch integrated data engine based on Flink is provided, including: S1. Configure business tags for the data source to be processed, and bind the business tags to the corresponding data source. Based on the data source bound by the business tags, obtain the data collection method and the computing task to be executed, and connect the data source to the unified stream batch computing engine based on Flink through the collection method to form the raw data to be processed. S2. For the raw data and computational tasks to be executed that are collected through the acquisition method, perform data parsing processing according to the bound business tags, convert the raw data into parsed data, and generate parsed metadata during the data parsing process; S3. Based on parsing metadata, extract field structure and parsing rule features to generate parsing layer data fingerprints; S4. Receive parsed data and computation tasks. Based on the bound business tags, identify the task type of the computation tasks. Based on the task type identification results, convert the computation tasks into logical execution plans and generate a unified batch execution plan. S5. Load the unified execution plan for streaming and batch processing in the Flink runtime environment, perform data processing on the parsed data, and collect operator processing rules and field flow information in the unified execution plan during the data processing process to generate execution layer data fingerprints. S6. Write the processing results obtained from data processing to the target storage. After the processing results are written, collect the field structure information of the processing results in the target storage and generate the target layer data fingerprint. S7. Associate the parsing layer data fingerprint, the execution layer data fingerprint, and the generated target layer data fingerprint in sequence according to the data processing order to construct a data fingerprint chain. Based on the data fingerprint chain, perform consistency verification on the data processing process and locate the data processing stage that causes inconsistency. S8. Based on the bound business tags, monitor the running status of data access, data parsing and data processing. Based on the monitoring results, adjust the data processing parameters of the bound business tags to optimize the running effect under different business processing requirements.

[0023] As described in steps S1 to S8 above, business tags are configured for the data source to be processed, and the tags are bound to the data source. A suitable data acquisition method is selected through tag-driven processing, and the data is input into the Flink engine to form raw data. Business tags represent data processing requirements, data structure characteristics, and processing methods. For the input raw data and the computational tasks to be executed, parsing processing is performed based on the business tags, including field splitting, type identification, and structure mapping, converting the raw data into parsed data. Simultaneously, parsing metadata is generated, describing field structure, field semantics, data granularity, and computational constraints. Based on the parsing metadata, field structure and parsing rule features are extracted, and combined with business tag information and computational task characteristics, a feature set is formed, which is then used to generate the parsing layer data. Fingerprinting records data source identifiers, batch information, and parsing time. It receives parsed data and computation tasks, identifies task types (stream processing, batch processing, or hybrid stream / batch processing) based on business tags, and converts tasks into unified logical execution descriptions. It then generates a unified stream / batch execution plan that can be executed on Flink, enabling unified scheduling and execution of stream / batch tasks. The unified execution plan is loaded into the Flink environment to perform computations on the parsed data. During execution, it collects operator processing rules and field transition information to generate execution-layer data fingerprints, providing a basis for task execution status monitoring and consistency verification. After writing the processing results to target storage, it collects the result field structure information in the target storage to generate target-layer data fingerprints, completing the status identification of the entire data processing process. This approach associates data fingerprints from the parsing, execution, and target layers according to their processing order, constructing a data fingerprint chain. It verifies data consistency at different processing stages level by level and identifies the stages where inconsistencies occur, achieving end-to-end data integrity and traceability. It continuously monitors the data access, parsing, and processing processes, acquiring operational status parameters such as latency, throughput, and resource usage. These parameters are compared with preset processing targets corresponding to business tags to generate adjustment instructions, enabling dynamic adjustment of data processing parameters. This ensures that the operational status meets performance and reliability requirements under different business scenarios. By driving access and computation task identification through business tags, it achieves unified management and scheduling for stream processing, batch processing, and hybrid scenarios, eliminating the limitations of traditional Flink stream batch processing strategies. To address scenario adaptation issues, data fingerprints are generated at the parsing, execution, and target layers, and a data fingerprint chain is constructed. This enables end-to-end tracking of data field structure, parsing rules, and processing procedures, improving data integrity and traceability. Real-time monitoring of operational status is provided, and dynamic parameter adjustment strategies are generated based on business tags. This allows the system to optimize resource allocation, processing latency, and throughput according to different business processing needs, improving the efficiency and stability of batch and stream task execution. The data fingerprint chain and consistency verification mechanism can automatically locate abnormal data processing stages, supporting backtracking analysis and correction, reducing the risk of data loss or calculation errors, and enhancing system reliability and auditing capabilities. The system design allows for visualized operation of business tags, parsing rules, and execution plan configuration, reducing the technical threshold.It facilitates rapid expansion and maintenance.

[0024] In a preferred implementation, business tags are configured for the data sources to be processed, and the business tags are bound to the corresponding data sources. Based on the data sources bound to the business tags, the data acquisition method and the computation tasks to be executed are obtained. The data sources are then connected to the unified stream and batch computing engine based on Flink through the acquisition method to form the raw data to be processed, including: S101. Configure business tags for the data source to be processed. The business tags are used to characterize business processing requirements, data structure characteristics and processing methods. The business tags are bound to the corresponding data sources and stored to generate tagged data source information. S102. Based on the processing requirements and data characteristics indicated by the business tags in the tagged data source information, obtain the data collection method of the data source and obtain the computational tasks to be executed corresponding to the data source. S103. Based on the data acquisition method, the data from the data source is connected to the unified stream batch computing engine based on Flink to generate the raw data for the computing tasks to be executed.

[0025] As described in steps S101 to S103 above, business tags are configured for the data source to be processed. These business tags characterize the business processing requirements, data structure characteristics, and processing methods of the data source. A binding relationship is established between the business tags and the corresponding data source, and the binding information is stored to form complete tagged data source information. Based on the business processing requirements and data characteristics recorded in the tagged data source information, a suitable data acquisition method (e.g., real-time streaming acquisition, batch import, or a hybrid method) is determined. Simultaneously, the corresponding computational tasks to be executed are obtained according to the business tags or data source metadata, ensuring that data acquisition and computational tasks are accurately matched. The data acquisition method can be obtained by querying data source metadata, analyzing data source attributes, or referring to predefined business strategy rules, ensuring that the acquisition method is consistent with the data source characteristics and business requirements. Following the determined data acquisition method, the data from the data source is connected to Flink's unified stream and batch computing engine to form raw data to be processed. After entering the engine, the raw data will automatically be associated with its business tags and computational task information to be executed, providing basic data and contextual information for the parsing, computation, and fingerprint generation steps.

[0026] In a preferred embodiment, the raw data acquired through data collection and the computational task to be executed are processed by performing data parsing based on the bound business tags, converting the raw data into parsed data. During the data parsing process, parsed metadata that meets the computational task is generated, including: S201. Based on the accessed raw data, the computing tasks to be executed, and the business tags, obtain the parsing rules and parsing structure of the raw data; S202. Based on the parsing rules and parsing structure, the original data is parsed and processed. The parsing process includes field splitting, type identification, and structure mapping, transforming the original data into parsed data that meets the processing requirements of the computing task. S203. During the parsing process of the original data, parsing metadata is generated synchronously based on the business tags and the computational requirements of the computational tasks. The parsing metadata is used to describe the parsed data structure, field semantics, data granularity and computational constraint information.

[0027] As described in steps S201 to S203 above, for the accessed raw data, combined with the corresponding computational task to be executed and its business tag, the parsing rules and parsing structure of the raw data are obtained. The parsing rules and parsing structure include field names, field types, relationships between fields, and data processing requirements. This information is the basis for parsing the raw data and ensuring matching with the computational task. The business tag provides the context of the data source's business scenario, processing method, and structural characteristics, enabling the parsing rules to dynamically adapt to the processing requirements of different types of data sources. Based on the obtained parsing rules and parsing structure, the raw data is systematically parsed. The parsing process includes field splitting (decomposing composite fields into standard fields), type identification (identifying...). The parsing process involves identifying field data types (e.g., strings, integers, floating-point numbers) and mapping structures (mapping raw fields to a unified parsed field structure). The result of the parsing process is parsed data that meets the requirements of the computation task to be executed. Its field structure, data type, and field combination conform to the input requirements of the computation task. During the parsing process, parsing metadata is generated synchronously based on business tags and the processing requirements of the computation task. Parsing metadata includes the field structure, field semantics, data granularity, and computational constraint information of the parsed data. It is used to describe the data state and structural characteristics of the parsing stage. Parsing metadata provides a reference basis and data structure basis for data processing, data fingerprint generation, and consistency verification, ensuring that the data processing process is traceable and auditable.

[0028] In a preferred implementation, based on parsing metadata, field structure and parsing rule features are extracted to generate a parsing layer data fingerprint, including: S301. Based on the parsed data and the corresponding parsed metadata, obtain the field structure information, field type information and parsing rule characteristics of the parsed data, and obtain the business tag information and the characteristics of the computation task to be executed corresponding to the parsed data. S302. Associate and integrate the field structure information, field type information, and parsing rule features with the business tag information and calculation task features, and generate feature identification information to represent the overall state of the parsed data according to the preset feature organization rules. S303. During the parsing data access and parsing processing, obtain the data source identification information, data processing batch information and parsing generation time corresponding to the parsing data, and associate the feature identification information with the data source identification information, data processing batch information and parsing generation time, and store it as a parsing layer data fingerprint.

[0029] As described in steps S301 to S303 above, based on the parsed data and its corresponding parsed metadata, the field structure information, field type information, and parsing rule features of the parsed data are obtained. Simultaneously, the business tag information and the features of the computational tasks to be executed corresponding to the parsed data are obtained to identify the business scenario, processing mode, and computational constraints of the data processing. Through this operation, the fields and structural features contained in the parsed metadata are explicitly extracted. The obtained field structure information, field type information, and parsing rule features are then associated and integrated with the business tag information and computational task features. According to preset feature organization rules, feature identification information is generated to characterize the overall state of the parsed data. This feature identification information uniquely characterizes the data structure and processing rule state during the parsing stage, ensuring accurate identification of the data state and processing method under different business scenarios. During the parsing data access and parsing processing process, the following features are obtained: The data source identification information, data processing batch information, and parsing generation time corresponding to the parsed data are associated and recorded with the feature identification information to generate a parsing layer data fingerprint. (The data fingerprint generation algorithm extracts and standardizes key features such as field structure information, field type information, parsing or operator processing rules, and business tags at each processing stage of the data. The above features are encoded according to preset feature organization rules and a deterministic mapping method (such as hash calculation or feature encoding) is used to generate a unique and stable data fingerprint.) The generation and storage of the parsing layer data fingerprint provides a traceable and referable basis for the execution layer and target layer data fingerprint reference and end-to-end consistency verification. The parsing layer data fingerprint records the data source, batch, and generation time information, providing a reliable reference for operation status monitoring and dynamic adjustment of data processing parameters, and optimizing system performance under different business processing requirements.

[0030] In a preferred implementation, the system receives parsed data and computation tasks. Based on the bound business tags, it identifies the task type of the computation tasks. Then, based on the task type identification results, it uniformly converts the computation tasks into logical execution plans and generates a unified stream / batch execution plan, including: S401. Receive parsed data and the corresponding computation task to be executed, and obtain the business tag information bound to the computation task; S402. Based on the processing mode indicated by the business tag, perform task type identification on the computing task to obtain the task type identification result, wherein the task type identification result includes stream processing mode, batch processing mode or stream-batch hybrid processing mode. S403. Based on the task type identification result, the computation task is converted into a logical execution description that is compatible with the processing method, and a unified stream-batch execution plan is generated based on the logical execution description for execution in the Flink-based unified stream-batch computation engine.

[0031] As described in steps S401 to S403 above, the parsed data generated during the parsing phase and the corresponding computational tasks to be executed are received. The business tag information bound to the computational tasks is obtained. The business tags contain information such as business processing requirements, data structure characteristics, and processing modes, providing an application basis for task type identification. Based on the processing mode indicated in the business tags, the computational tasks are identified as task types. The identification results include stream processing, batch processing, or mixed stream-batch processing, which are used to determine the execution mode of the computational tasks in the unified stream-batch computation engine. Through task type identification, the optimal execution logic can be accurately matched according to different task characteristics, improving computational efficiency and resource utilization. Based on the task type identification results, the computational tasks are converted into logical execution descriptions that are compatible with the processing methods, clarifying the data processing steps, operator order, and field flow rules. Based on the logical execution descriptions, a unified stream-batch execution plan that can be executed in the Flink unified stream-batch computation engine is generated, ensuring that data processing of different task types can be uniformly scheduled and executed. Through the unified execution plan, task execution strategies can be monitored and adjusted at runtime, achieving dynamic optimization for multiple business scenarios and large-scale data processing.

[0032] In a preferred implementation, a unified streaming and batch execution plan is loaded into the Flink runtime environment to perform data processing on the parsed data. During data processing, operator processing rules and field transition information from the unified streaming and batch execution plan are collected to generate an execution layer data fingerprint, including: S501: Load the unified execution plan for streaming and batch processing in the Flink-based runtime environment, and perform data processing operations on the parsed data according to the unified execution plan for streaming and batch processing. S502. During the execution of data processing operations, obtain the operator configuration content used to describe the data processing logic in the unified execution plan of the batch and stream, and parse the field flow information of the data in each processing stage. S503. Generate execution layer data fingerprints based on operator configuration content and field flow information.

[0033] As described in steps S501 to S503 above, a unified execution plan for streaming and batch processing is loaded into the Flink runtime environment. Based on this execution plan, specific data processing operations are performed on the parsed data, including data filtering, aggregation, transformation, and join operations. These operations strictly follow the processing order and operator configuration defined in the logical execution plan to ensure the task executes according to the expected logic. During data processing, the configuration information of each operator in the unified execution plan is automatically obtained, such as operator type, input / output fields, and processing rules. Simultaneously, the field flow information of the parsed data in each processing stage is tracked, recording the transfer of data fields between operators. Depending on the situation, based on the collected operator configuration and field flow information, an execution layer data fingerprint is generated to uniquely identify the processing status of the parsed data during the execution phase. The data fingerprint records the field structure, operator processing rules, and field flow characteristics, which can be used for differentiated task tracking, anomaly detection, and end-to-end consistency verification. Whether it is a streaming task, a batch task, or a mixed streaming and batch task, the processing status can be accurately represented by the execution layer data fingerprint, enabling unified monitoring and management. By recording field flow and operator rules, data anomalies or deviations that occur during execution can be compared and verified to ensure the consistency of data processing results with the parsing and target phases.

[0034] In a preferred embodiment, the processing result obtained from data processing is written to the target storage. After the processing result is written, the field structure information of the processing result in the target storage is collected to generate a target layer data fingerprint, including: S601. Write the processing results obtained from data processing to the target storage according to the preset data writing rules; S602. After the processing result is written, read the field name and field type information corresponding to the processing result from the target storage; S603. Generate target layer data fingerprints based on field names and field type information.

[0035] As described in steps S601 to S603 above, according to preset data writing rules, the data results processed in the Flink execution environment are written to the target storage. The writing rules may include field mapping, data format conversion, partitioning strategy, or indexing strategy to ensure the correct storage and accessibility of data in the target storage. After the data is written, the field information corresponding to the processing result is read from the target storage, including field name, field type, and field order, etc. The collected field structure information provides a direct basis for the generation of the target layer data fingerprint, ensuring that the fingerprint can accurately reflect the actual state of the data in the target storage. The target layer data fingerprint is generated based on the collected field structure information to uniquely represent the data structure state of the processing result in the target storage. The data fingerprint records the field name, field type, and structural characteristics, which can be used for data consistency verification, difference detection, and end-to-end data tracing. It can dynamically monitor the data structure written to the target storage and provide basic data support for data access, query optimization, and structural adjustment under different business needs.

[0036] In a preferred embodiment, the parsing layer data fingerprint, the execution layer data fingerprint, and the generated target layer data fingerprint are sequentially associated according to the data processing order to construct a data fingerprint chain. Based on this data fingerprint chain, consistency checks are performed on the data processing process to pinpoint the data processing stage where inconsistencies occur, including: S701. The parsing layer data fingerprint, the execution layer data fingerprint, and the target layer data fingerprint are linked together in a chain according to the processing order of parsing, operator execution, and result writing to construct a data fingerprint chain. S702. Based on the field structure features and processing rule features between adjacent data fingerprints in the data fingerprint chain, the consistency of data at different processing stages is verified step by step. S703. When an inconsistency is detected between adjacent data fingerprints, the specific data processing stage that caused the data inconsistency is located based on the position of the data fingerprint corresponding to the inconsistency in the data fingerprint chain.

[0037] As described in steps S701 to S703 above, the parsing layer data fingerprint, execution layer data fingerprint, and target layer data fingerprint are sequentially associated according to the data processing order to form a data fingerprint chain. The data fingerprint chain records the state changes of data at each processing stage in a chain structure, including the data characteristics and structural information of the parsing stage, execution stage, and result writing stage. The field structure characteristics and processing rule characteristics between adjacent data fingerprints in the data fingerprint chain are compared and analyzed to check whether the data is consistent across different processing stages. The verification content includes the consistency of field names, field types, field order, and parsing or processing rules to ensure that no structural errors or rule deviations occur in the data during batch processing. When inconsistencies are found between adjacent data fingerprints, the data processing stage that caused the anomaly (e.g., parsing stage, operator execution stage, or result writing stage) is accurately located based on the position of the inconsistent data fingerprint in the fingerprint chain. Through the chain structure of the fingerprint chain, the source of the problem can be quickly traced, providing a basis for repair, reprocessing, or optimization. The data fingerprint chain provides a complete context for anomaly analysis and tracing, and processing parameters can be adjusted or specific stages can be re-executed based on the location results to achieve dynamic optimization.

[0038] In a preferred implementation, the operational status of data access, data parsing, and data processing is monitored based on the bound business tags. Based on the monitoring results, the data processing parameters for the bound business tags are adjusted to optimize the operational performance under different business processing requirements, including: S801. Based on the bound business tags, continuously monitor the operational status of the data access phase, data parsing phase, and data processing phase, and obtain operational status parameters. S802. Compare the running status parameters with the preset processing targets corresponding to the business tags, and generate adjustment instructions for the data processing parameters. S803. According to the adjustment instruction, dynamically adjust the data processing parameters corresponding to the business tag so that the running status of the data processing process meets the processing target corresponding to the business tag.

[0039] As described in steps S801 to S803 above, based on the bound business tags, the operational status of the data access phase, data parsing phase, and data processing phase is monitored in real time. The monitoring content includes key indicators such as processing latency, resource usage (e.g., CPU and memory utilization), and data throughput characteristics (e.g., the amount of data processed per unit time). By referencing the business tags, the monitoring data is associated with specific business scenarios and task types, providing contextual information for optimization. The collected operational status parameters are compared with the preset processing targets corresponding to the business tags, such as latency targets, throughput targets, or resource usage constraints. Based on the comparison results, data processing parameter adjustment instructions are generated, clarifying which processing parameters need to be modified (e.g., parallelism, operator scheduling strategy, cache size, batch processing size, etc.). The adjustment instructions are directly associated with the business tags and their corresponding... The computational task ensures that optimization measures are traceable within the business context. Based on the generated adjustment instructions, the data processing parameters corresponding to the business tags are dynamically adjusted to optimize the data processing process in real time. Adjustments can be applied to data acquisition rate, parsing operator configuration, and unified batch execution plan parameters, so that the system's operating state gradually approaches the processing target set by the business tag. By binding the context association of business tags, personalized monitoring and optimization of different business tasks can be achieved to meet the performance requirements of multiple business scenarios. When the operating state deviates from the preset target, parameters are adjusted in a timely manner to avoid processing failures or performance degradation due to resource bottlenecks or data volume fluctuations, thereby enhancing system stability. Monitoring, comparison, instruction generation, and dynamic adjustment form a closed-loop control mechanism, enabling the data processing system to have adaptive capabilities and automatically optimize according to business and operating status, reducing manual intervention.

[0040] And, a design terminal for a unified stream-batch integrated data engine based on Flink, including: One or more processors; A storage device on which one or more programs are stored; When one or more programs are executed by one or more processors, the one or more processors implement a unified stream-batch integrated data engine design method based on Flink.

[0041] The above description is merely a preferred embodiment of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention. Structures, devices, and operating methods not specifically described or explained in this invention are implemented according to conventional methods in the art unless otherwise specified or limited.

Claims

1. A design method for a unified stream-batch integrated data engine based on Flink, characterized in that, include: Configure business tags for the data sources to be processed, and bind the business tags to the corresponding data sources. Based on the data sources bound to the business tags, obtain the data collection method and the computing tasks to be executed, and connect the data sources to the unified stream and batch computing engine based on Flink through the collection method to form the raw data to be processed. For the raw data and computing tasks to be executed that are collected through acquisition, data parsing processing is performed according to the bound business tags to convert the raw data into parsed data. During the data parsing process, parsed metadata is generated. Based on parsing metadata, extract field structure and parsing rule features to generate parsing layer data fingerprints; Receive parsed data and computation tasks, identify the task type based on the bound business tags, convert the computation tasks into logical execution plans based on the task type identification results, and generate a unified batch execution plan; Load the unified execution plan for streaming and batch processing in the Flink runtime environment, perform data processing on the parsed data, and collect operator processing rules and field flow information in the unified execution plan during the data processing process to generate execution layer data fingerprints. The processing results obtained from data processing are written to the target storage. After the processing results are written, the field structure information of the processing results in the target storage is collected to generate the target layer data fingerprint. The parsing layer data fingerprint, the execution layer data fingerprint, and the generated target layer data fingerprint are sequentially associated according to the data processing order to construct a data fingerprint chain. Based on the data fingerprint chain, the consistency of the data processing process is verified to locate the data processing stage that causes inconsistency. Based on the bound business tags, the operational status of data access, data parsing, and data processing is monitored. Based on the monitoring results, the data processing parameters of the bound business tags are adjusted to optimize the operational effect under different business processing requirements.

2. The design method for a unified stream-batch integrated data engine based on Flink according to claim 1, characterized in that, Configure business tags for the data sources to be processed, and bind the business tags to the corresponding data sources. Based on the data sources bound to the business tags, obtain the data acquisition method and the computation tasks to be executed, and connect the data sources to the unified stream and batch computing engine based on Flink through the acquisition method to form the raw data to be processed, including: Configure business tags for the data sources to be processed. The business tags are used to characterize business processing requirements, data structure characteristics and processing methods. The business tags are then bound to the corresponding data sources and stored to generate tagged data source information. Based on the processing requirements and data characteristics indicated by the business tags in the tagged data source information, the data collection method of the data source is obtained, and the computing tasks to be executed corresponding to the data source are obtained. Based on the data acquisition method, the data from the data source is connected to the unified stream and batch computing engine based on Flink to generate the raw data for the computing tasks to be executed.

3. The design method for a unified stream-batch integrated data engine based on Flink according to claim 1, characterized in that, For the raw data and computational tasks to be executed via acquisition, data parsing processing is performed based on the bound business tags to convert the raw data into parsed data. During the data parsing process, parsed metadata that meets the computational tasks is generated, including: Based on the accessed raw data, the computing tasks to be executed, and the business tags, obtain the parsing rules and parsing structure of the raw data; Based on the parsing rules and parsing structure, the original data is parsed and processed. The parsing process includes field splitting, type identification, and structure mapping, transforming the original data into parsed data that meets the processing requirements of the computing task. During the parsing and processing of the raw data, parsing metadata is generated synchronously based on the business tags and computational requirements of the computational tasks. The parsing metadata is used to describe the parsed data structure, field semantics, data granularity, and computational constraint information.

4. The design method for a unified stream-batch integrated data engine based on Flink according to claim 1, characterized in that, Based on parsing metadata, field structure and parsing rule features are extracted to generate parsing layer data fingerprints, including: Based on the parsed data and the corresponding parsed metadata, obtain the field structure information, field type information and parsing rule characteristics of the parsed data, and obtain the business tag information and the characteristics of the computation task to be executed corresponding to the parsed data; The field structure information, field type information, and parsing rule features are associated and integrated with business tag information and calculation task features, and feature identification information is generated to represent the overall state of the parsed data according to the preset feature organization rules. During the data access and parsing process, the data source identification information, data processing batch information, and parsing generation time corresponding to the parsed data are obtained. The feature identification information is associated with the data source identification information, data processing batch information, and parsing generation time and recorded as a data fingerprint of the parsing layer and stored.

5. The design method for a unified stream-batch integrated data engine based on Flink according to claim 1, characterized in that, The system receives and parses data and computation tasks. Based on the bound business tags, it identifies the task type of the computation tasks. Based on the task type identification results, it converts the computation tasks into a unified logical execution plan and generates a unified stream and batch execution plan, including: Receive parsed data and the corresponding computational tasks to be executed, and obtain the business tag information bound to the computational tasks; Based on the processing mode indicated by the business tag, the computing task is identified by task type to obtain the task type identification result, which includes stream processing mode, batch processing mode or stream-batch hybrid processing mode. Based on the task type identification results, the computation task is converted into a logical execution description that is compatible with the processing method, and a unified stream-batch execution plan is generated based on the logical execution description for execution in the Flink-based unified stream-batch computation engine.

6. The design method for a unified stream-batch integrated data engine based on Flink according to claim 1, characterized in that, Load the unified execution plan for streaming and batch processing in the Flink runtime environment, perform data processing on the parsed data, and collect operator processing rules and field transition information from the unified execution plan during data processing to generate execution layer data fingerprints, including: Load the unified execution plan for streaming and batch processing in the Flink-based runtime environment, and perform data processing operations on the parsed data according to the unified execution plan for streaming and batch processing. During the execution of data processing operations, the operator configuration content used to describe the data processing logic in the unified execution plan of batch processing is obtained, and the field flow information of the data in each processing stage is parsed. Based on the operator configuration content and field flow information, an execution layer data fingerprint is generated.

7. The design method for a unified stream-batch integrated data engine based on Flink according to claim 1, characterized in that, The processing results are written to the target storage. After the processing results are written, the field structure information of the processing results in the target storage is collected to generate the target layer data fingerprint, including: Write the processing results obtained from data processing to the target storage according to the preset data writing rules; After the processing result is written, the field name and field type information corresponding to the processing result are read from the target storage; Generate target layer data fingerprints based on field names and field type information.

8. The design method for a unified stream-batch integrated data engine based on Flink according to claim 1, characterized in that, The parsing layer data fingerprint, execution layer data fingerprint, and generated target layer data fingerprint are sequentially associated according to the data processing order to construct a data fingerprint chain. Based on this data fingerprint chain, consistency checks are performed on the data processing process to pinpoint the stages where inconsistencies occur, including: The parsing layer data fingerprint, the execution layer data fingerprint, and the target layer data fingerprint are linked together in a chain according to the processing order of parsing, operator execution, and result writing to construct a data fingerprint chain; Based on the field structure characteristics and processing rule characteristics between adjacent data fingerprints in the data fingerprint chain, the consistency of data at different processing stages is verified step by step. When an inconsistency is detected between adjacent data fingerprints, the specific data processing stage that caused the inconsistency is located based on the position of the data fingerprint corresponding to the inconsistency in the data fingerprint chain.

9. The design method for a unified stream-batch integrated data engine based on Flink according to claim 1, characterized in that, Based on the bound business tags, the operational status of data access, data parsing, and data processing is monitored. Based on the monitoring results, the data processing parameters for the bound business tags are adjusted to optimize the operational performance under different business processing requirements, including: Based on the bound business tags, the operational status of the data access phase, data parsing phase, and data processing phase is continuously monitored to obtain operational status parameters; The running status parameters are compared with the preset processing targets corresponding to the business tags to generate adjustment instructions for the data processing parameters; According to the adjustment instructions, the data processing parameters corresponding to the business tags are dynamically adjusted so that the running status of the data processing process meets the processing objectives corresponding to the business tags.

10. A terminal design for a unified stream-batch integrated data engine based on Flink, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When one or more programs are executed by one or more processors, the one or more processors implement the design method of the unified stream-batch integrated data engine based on Flink as described in any one of claims 1 to 9.

Images