Method and device for improving Flink multi-stream association query performance

Abstract

According to the method and device for improving the Flink multi-stream association query performance, through the hierarchical design of data stream integration, extraction, conversion and loading, wide table generation and data summarization and storage, an efficient data processing assembly line is constructed, the data stream is standardized, the wide table is generated, repeated association operation among multi-stream data is reduced, and the data processing efficiency is improved. According to the method, the state storage requirement is reduced, the change write-in of the streaming data is optimized by utilizing the Upsert capability of Kafka, the resource waste caused by the traditional repeated data storage is avoided, the calculation amount and the redundancy of data transmission are reduced by screening the data in the summarized data layer according to the theme and granularity, the state expansion and the data divergence are reduced, the calculation resource consumption is obviously reduced, and the calculation efficiency is improved. The memory and storage occupation are reduced, the processing logic of multi-stream association is simplified, and the maintainability of the system is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and device for improving Flink multi-stream association query performance. Background Art

[0002] At present, Flink (multi-stream association) provides powerful real-time stream processing capabilities and is widely used in scenarios that require real-time analysis, such as finance and the Internet of Things. Flink adapts to a powerful ecosystem: it seamlessly integrates with big data technologies such as Kafka, Hadoop, and Doris to form a powerful data processing ecosystem that supports multiple data sources and storage backends. At the same time, it provides highly available state management and fault-tolerant mechanisms, supports snapshots and restarts, and ensures data reliability and consistency in the event of a failure. Based on the above characteristics, Flink has gradually become an important solution in the field of big data real-time computing and is widely used in scenarios such as real-time analysis.

[0003] When using Kafka and Flink for multi-stream association, data divergence may cause state expansion, which will cause a series of problems. State expansion directly leads to a significant increase in the memory and storage resources required for Flink jobs. If the state exceeds the available memory, memory overflow will occur, which may cause the application to crash or performance degradation. The state snapshot and restart mechanism requires additional storage space, especially when processing high-frequency updates, the storage pressure increases. As the state continues to grow, Flink's processing performance may be affected: the time for state checks, updates, and snapshots will increase, resulting in an increase in overall processing latency and affecting real-time performance; larger states increase latency when accessing the state, especially in high-concurrency scenarios. Managing and maintaining large-scale states brings additional complexity: in multi-stream association scenarios, ensuring the consistency of the state becomes more complex and requires more sophisticated designs and strategies. When the state becomes large, monitoring and debugging become more difficult, and it becomes more difficult to identify performance bottlenecks and potential problems. Flink's fault-tolerance mechanism relies on state snapshots and restart strategies. State expansion will slow down the snapshot process, which may affect the overall throughput of the system; in the event of a failure, the time to restore the state may increase significantly, affecting business continuity. As the complexity of the state increases, the maintainability of the code may also suffer.

[0004] Therefore, in view of the above problems, a new solution is urgently needed to suppress the state expansion that may be caused by multi-stream association. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a method and device for improving the performance of Flink multi-stream association queries, so as to eliminate or improve one or more defects existing in the prior art, and solve the problems in the prior art of increased resource consumption, performance degradation, increased complexity, high fault tolerance cost and reduced maintainability caused by state expansion due to data divergence when using Kafka and Flink for multi-stream association.

[0006] One aspect of the present invention provides a method for improving the performance of Flink multi-stream association query, the method being executed by a distributed computing Flink engine composed of multiple servers, the method comprising the following steps:

[0007] The operational data processing layer reads the Kafka data stream of the data center and integrates the data streams from different sources into a unified format for output;

[0008] The fact wide table layer extracts, transforms and loads the data stream in the unified format, and rewrites it into the Kafka to generate fact wide table data by Upsert (update or insert);

[0009] The summary data layer filters and aggregates the fact wide table data according to preset topics and preset statistical granularity to generate multiple preset data summary tables;

[0010] The preset data summary table is written into the Doris database by the summary data layer for the back-end interface to call.

[0011] In some embodiments, the amount of data read by the operational data processing layer includes order flow and payment flow, and the preset data summary table includes: a ledger detail table, a real-time charge total table and a ledger charge information table.

[0012] In some embodiments, in the method, integrating the data streams from different sources into a unified format for output includes:

[0013] Clean the data stream to remove duplicate data, correct data outliers, and fill in missing data;

[0014] Performing field mapping and conversion on the data stream, renaming the original fields and standardizing the formats;

[0015] The fact wide surface layer performs extraction, transformation and loading operations on the data stream in a unified format, including:

[0016] Performing data aggregation on the data stream, including grouping statistics, merging and calculation;

[0017] The data stream is segmented to separate the complex structure into a plurality of preset simple fields.

[0018] In some embodiments, the method further comprises:

[0019] Analyze the synchronization blocking points at the data processing layer, the fact wide table layer, the summary data layer, and the summary data layer, and find multiple target data flows that belong to synchronization processing;

[0020] Based on Flink's asynchronous sending and receiving mechanism, the target data stream is optimized using a non-blocking operator chain, and a non-blocking back pressure mechanism is enabled for processing.

[0021] In some embodiments, the method further comprises:

[0022] A message queue is used as an intermediary between the data processing layer, the fact wide table layer, the summary data layer, and the summary data layer, so that the data completed by each layer can be pushed to the next layer independently.

[0023] In some embodiments, the method further comprises:

[0024] Use Flink's RocksDB state backend to support the persistence of the data stream, and store the intermediate processing state in Redis for downstream asynchronous queries.

[0025] In some embodiments, the method further comprises:

[0026] Flink’s task indicator monitoring mechanism is introduced to collect data throughput and latency of the data processing layer, the fact wide table layer, the summary data layer, and the summary data layer, sample and analyze the processing time of the data flow at each layer, and adjust the asynchronous queue size and parallelism according to the processing time corresponding to each layer.

[0027] On the other hand, the present invention also provides a device for improving the performance of Flink multi-stream association queries, comprising a processor, a memory, and a computer program / instruction stored in the memory, wherein the processor is used to execute the computer program / instruction, and when the computer program / instruction is executed, the device implements the steps of the above method.

[0028] On the other hand, the present invention further provides a computer-readable storage medium having a computer program / instruction stored thereon, wherein the computer program / instruction implements the steps of the above method when executed by a processor.

[0029] On the other hand, the present invention also provides a computer program product, comprising a computer program / instruction, characterized in that the computer program / instruction implements the steps of the above method when executed by a processor.

[0030] The beneficial effects of the present invention are at least:

[0031] The method and device for improving the performance of Flink multi-stream association query described in the present invention build an efficient data processing pipeline through the hierarchical design of integrating data flow, extraction, transformation and loading, wide table generation, data aggregation and storage, standardize data flow and generate wide tables, reduce repeated association operations between multi-stream data, reduce state storage requirements, use Kafka's Upsert capability to optimize the change writing of streaming data, avoid resource waste caused by traditional duplicate data storage, and reduce the amount of calculation and data transmission redundancy, reduce state expansion and data divergence, significantly reduce computing resource consumption, reduce memory and storage occupancy, simplify the processing logic of multi-stream association, and improve the maintainability of the system.

[0032] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0033] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute a limitation of the present invention. In the drawings:

[0035] Figure 1 The figure is a flow chart of a method for improving the performance of Flink multi-stream association query according to an embodiment of the present invention.

[0036] Figure 2 This is a logical schematic diagram of a method for improving Flink multi-stream association query performance according to another embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0038] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0039] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0040] In the face of the problem that state expansion directly leads to a significant increase in the memory and storage resources required for Flink jobs, and the state exceeds the available memory, resulting in memory overflow, causing application crashes or performance degradation, one aspect of the present invention provides a method for improving the performance of Flink multi-stream association queries, the method is executed by a distributed computing Flink engine composed of multiple servers, such as Figure 1 As shown, the method includes the following steps S101 to S104:

[0041] Step S101: The operation data processing layer reads the Kafka data stream of the data center and integrates the data streams from different sources into a unified format for output.

[0042] Step S102: The fact wide table layer extracts, transforms and loads the data stream in the unified format, and rewrites it into Kafka through Upsert (update or insert) to generate fact wide table data.

[0043] Step S103: The summary data layer filters and aggregates the fact wide table data according to preset themes and preset statistical granularities to generate multiple preset data summary tables.

[0044] Step S104: The summary data layer writes the preset data summary table into the Doris database for the back-end interface to call.

[0045] Steps S101 to S104 optimize data processing efficiency, ensure data consistency, and finally write the results to the Doris database for query by introducing ODS (operational data processing layer), DWD (fact wide table layer, or data preprocessing layer) and DWS (summary data layer). The ODS layer integrates data streams from different sources, provides unified data input for the DWD layer, and simplifies data processing logic. The fact wide table layer preprocesses and deduplicates data, greatly reducing the complexity and resource consumption of downstream DWS layer tasks. Through the sequential execution of Flink tasks, the data is processed step by step from the ODS to the DWS layer to ensure the consistency and integrity of the data between the layers. The final summary data is written to the Doris database to reduce the amount of data queried by the interface, improve query efficiency, and enhance external service capabilities.

[0046] The ODS layer (Operational Data Store) serves as the receiving and integration layer for raw data. It reads the data stream from the middle platform Kafka and integrates data streams from different sources into a unified format for output. The data streams here are usually unprocessed and contain various real-time or near real-time raw event data. It provides unified data input for the DWD layer to facilitate subsequent ETL operations.

[0047] The DWD layer (Data Warehouse Detail Layer) is used as a fact wide table layer to pre-process the data of the ODS layer (such as ETL operations) and write it to Kafka in Upsert mode. Perform ETL processing, including data cleaning, conversion, deduplication, etc., to avoid interference of duplicate data on downstream calculations. Relieve the processing pressure of the DWS layer and generate high-quality fact wide table data for subsequent statistical analysis.

[0048] The DWS layer (Data Warehouse Summary Layer) is a summary data layer. It generates summary tables based on the fact wide table data of the DWD layer, filtered and aggregated according to different topics and statistical granularity.

[0049] like Figure 2 As shown, first read the data source Kafka as the ods layer. The present invention adds a dwd layer as a preprocessing layer between the ods layer and the Flink direct processing: the result data of the ods layer is used as the dwd layer, and is written to Kafka in Upsert mode, which greatly buffers the processing pressure of the downstream. Among them, the ods layer reads the data stream of the data center Kafka, integrates the data streams of different sources involved, and provides a unified output for the downstream; the dwd layer is a fact wide table layer. In this example, the dwd layer receives the data stream of the ods layer, and rewrites it to Kafka after ETL processing. At this time, the data stream will automatically perform deduplication once as a wide table of account business facts; the dws layer is a summary data layer, and the dwd layer is processed into three dws tables according to different topics and statistical granularity through screening and aggregation operations, namely, the account detail table, the real-time charge total table, and the account charge information table. At the same time, such processing naturally guarantees the sequential execution of Flink tasks in each layer of ods-dwd-dws, ensuring data consistency. Finally, the four data tables are stored in the Doris database for use in Java backend interface queries, which reduces the amount of data for interface queries and optimizes the interface's external capabilities.

[0050] In some embodiments, the amount of data read by the operational data processing layer includes order flow and payment flow, and the preset data summary tables include: a ledger detail table, a real-time charge summary table, and a ledger charge information table.

[0051] In this embodiment, a Kafka cluster is deployed, which is a distributed message queue system composed of multiple servers. Real-time data streams (such as order streams, payment streams, etc.) are received from multiple data sources. A Flink computing cluster is deployed, which is a distributed real-time computing engine composed of multiple servers. It executes real-time computing tasks of each layer of ODS, DWD, and DWS, and realizes ETL (extraction, transformation, loading) operations such as data cleaning, conversion, deduplication, and aggregation. A Doris database cluster is deployed, which is a distributed database system composed of multiple servers. It stores the summary data tables generated by the DWS layer, including ledger details tables, real-time charge summary tables, and ledger charge information tables, providing efficient multi-dimensional query functions and supporting the back-end business system to quickly access data.

[0052] In step S101, the data streams from different sources are integrated into a unified format for output, including steps S201 to S202:

[0053] Step S201: Clean the data stream to remove duplicate data, correct data outliers, and fill in missing data.

[0054] Step S202: Map and convert the fields of the data stream, rename the original fields and standardize the formats.

[0055] In step S102, the fact wide surface layer extracts, transforms and loads the data stream in the unified format, including steps S301 to S302:

[0056] Step S301: performing data aggregation on the data stream, including grouping statistics, merging and calculation.

[0057] Step S302: segment the data stream to break the complex structure into a plurality of preset simple fields.

[0058] In step S102, the core function of writing to Kafka in the Upsert mode is to deduplicate data and update the status, thereby simplifying the downstream stream processing logic and reducing the performance pressure caused by data redundancy.

[0059] In a Kafka topic, when receiving records with the same primary key, Upsert can directly update existing records instead of simply appending new records. Upsert can avoid the storage of duplicate data, reduce redundancy, and ensure that data always reflects the latest business status. When consuming downstream, no additional deduplication logic is required, which simplifies stream processing. By identifying primary keys (such as order ID, user ID, etc.), duplicate records are automatically removed, and only the latest valid data is retained, reducing the pressure of duplicate data on storage and downstream computing.

[0060] In multi-stream data processing, data sources may come from different business systems (such as order streams and payment streams), and these data streams may have the following characteristics: duplicate records exist (such as multiple updates of the same order); different streams need to be associated (such as matching order and payment data); the data flow is large and the real-time requirement is high. To solve these problems, the Upsert method provides key support at the source and Kafka layers. Upsert is based on primary key deduplication and update in multi-stream operations. Each record contains a unique primary key (such as order ID). When a new record is written to a Kafka topic, if it is detected that a record with the same primary key already exists, the old record is replaced (that is, an update operation). Specifically, the Compact Log function supported by Kafka is used: Kafka retains the latest version of the record with the same primary key through a cleanup strategy (Log Compaction) and discards the old version. When writing data, Flink's Kafka Connector can use the Upsert mode to specify the primary key field. When multi-stream data is associated (such as order stream and payment stream), the results are directly merged when they are output to the Kafka topic. The status update between streams is realized through the primary key (such as order ID), so that the associated data exists directly in Kafka in the latest state. It can realize dynamic update of business status. For example, in the ledger system, the data stream of the DWD layer contains the transaction details of each account, and the latest balance status of each account in the Kafka topic is directly updated through the Upsert mode. When the downstream DWS layer reads, there is no need to scan all historical transaction records, only the latest status needs to be consumed. The specific implementation mechanism is that Flink uses the state management function to calculate the latest status of each primary key in real time, and triggers the Upsert operation when writing to Kafka.

[0061] Complete data simplification at the source: After the data stream is processed by Upsert at the source (Kafka), it is already deduplicated and up-to-date data. Downstream tasks directly consume the optimized data to avoid repeated calculations. After using Kafka's Upsert mode, the DWD layer does not need to store complete historical data, but only needs to maintain the latest data status. This reduces the consumption of computing and storage resources.

[0062] Therefore, the core function of writing to Kafka in the Upsert mode is to support real-time data updates and deduplication. It directly updates the primary key at the Kafka layer, front-ends and simplifies the complexity of multi-stream associations, allowing downstream tasks to directly consume optimized data streams, greatly improving the efficiency and simplicity of data stream processing. In the entire technical solution, the application of the Upsert mode realizes real-time updates, simplifies logic, and reduces storage pressure, providing key support for efficient processing from ODS to DWD.

[0063] In some embodiments, the method further includes steps S401-S402:

[0064] Step S401: Analyze the synchronization blocking points at the data processing layer, the fact wide table layer, the summary data layer, and the summary data layer, and find multiple target data flows that belong to synchronization processing.

[0065] Step S402: Based on Flink's asynchronous sending and receiving mechanism, the target data stream is optimized using a non-blocking operator chain, and a non-blocking back pressure mechanism is enabled for processing.

[0066] In steps S401 to S402, first, at the data processing layer, by analyzing the consumption and distribution logic of the Kafka data stream, identify the synchronization processing points that may cause blocking, such as the reading order or partition imbalance of multiple data stream sources. By adjusting the parallelism and distribution strategy of the data stream, it can be distributed to the subsequent processing links in a more balanced manner, and in this process, Flink's non-blocking operator chain (Non-blocking Operator Chain) is introduced to enable multiple operators to run in a pipeline manner on the same thread, reducing thread switching overhead. Secondly, at the fact wide table layer, for possible blocking points in ETL operations (such as associated calculations or updates when wide tables are generated), an asynchronous I / O (Async I / O) mechanism is used to convert the Upsert operation of the wide table into an asynchronous task. By batch pre-fetching or caching of associated data, and non-blocking processing methods, the consumption rate of upstream and downstream data streams is accelerated, and the risk of state storage expansion is reduced.

[0067] In the data aggregation layer, for data screening and aggregation operations with preset topics and statistical granularity, asynchronous grouping and partition aggregation methods in distributed computing are used to avoid overall blocking problems caused by delays in processing single partition data. In specific implementation, Flink's event time window combined with the Watermark mechanism can be used to ensure the orderliness of data and prevent the window trigger from blocking other calculations by waiting for delayed data to arrive.

[0068] Finally, in the summary data layer, the operation of writing the summary table to the Doris database is optimized using a non-blocking back pressure mechanism. Through Flink's asynchronous Sink function, batch writing and flow control mechanisms are supported to avoid the backlog of front-end data due to low database write rates. At the same time, by monitoring the back pressure indicators of the flow, data sharding and submission frequency are dynamically adjusted to further optimize throughput.

[0069] Through the above method, using Flink's asynchronous sending and receiving mechanism and non-blocking operator chain, it is possible to effectively reduce synchronization blocking points, improve the parallelism and throughput of the overall data stream processing, and ultimately achieve full utilization of resources and maximize performance.

[0070] In some embodiments, the method further comprises: using a message queue as an intermediary between the data processing layer, the fact wide table layer, the summary data layer, and the summary data layer, so that the data completed by each layer can be independently pushed to the next layer.

[0071] Specifically, first, a message queue suitable for distributed architecture (such as Kafka, RabbitMQ or RocketMQ) is introduced between each layer as an intermediary for data exchange. In the data processing layer, the read original data stream is formatted and pushed directly to the specified topic or queue of the message queue. Each data source corresponds to an independent topic to achieve source separation. In the fact wide table layer, the data in the message queue is consumed, ETL processing is performed and a fact wide table is generated. After the wide table is generated, the result is written back to the message queue again through Upsert. At this point, the message queue can support different partitioning strategies and distribute the wide table data to different topics according to business logic for consumption by the next layer.

[0072] In the summary data layer, we subscribe to the wide table topic data from the message queue, and perform aggregation and filtering according to the preset topic and statistical granularity. To ensure efficiency, we can dynamically allocate independent consumption instances for each aggregation task, making full use of the partition parallelism of the message queue. After processing, the summary results are pushed to another topic or queue to provide clear data input for the summary data layer.

[0073] In the summary data layer, the summary results are consumed from the message queue and written into the Doris database in batches. This decoupling method allows upstream and downstream processing tasks to run independently, smoothing traffic fluctuations through the buffering capacity of the message queue and avoiding overall blockage caused by performance bottlenecks at a certain layer. In addition, the persistence and retry mechanism of the message queue also provides higher fault tolerance, significantly improving the reliability and maintainability of the system.

[0074] In some embodiments, the method further includes: using Flink's RocksDB state backend to support data stream persistence, storing intermediate processing states in Redis for downstream asynchronous queries.

[0075] RocksDB is the persistent state backend provided by Flink, which can efficiently process large-scale state data and support high reliability. It achieves efficient fault tolerance and recovery capabilities through local disk storage and incremental snapshots. Redis is a high-performance in-memory database that is very suitable for storing and quickly accessing intermediate states of processing to support downstream asynchronous queries.

[0076] In some embodiments, the method further includes: introducing Flink's task indicator monitoring mechanism to collect data throughput and latency of the data processing layer, fact wide table layer, summary data layer, and summary data layer, sampling and analyzing the processing time of the data flow at each layer, and adjusting the asynchronous queue size and parallelism according to the corresponding processing time of each layer.

[0077] On the other hand, the present invention also provides a device for improving the performance of Flink multi-stream association queries, comprising a processor, a memory, and a computer program / instruction stored in the memory, wherein the processor is used to execute the computer program / instruction, and when the computer program / instruction is executed, the device implements the steps of the above method.

[0078] Corresponding to the above method, the present invention also provides an apparatus / system, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the apparatus / system implements the steps of the method described above.

[0079] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned edge computing server deployment method are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0080] The embodiment of the present invention further provides a computer device, which may include a processor, a memory and an image acquisition device, wherein the processor and the memory may be connected via a bus or other means, and this embodiment takes the bus connection as an example. The image acquisition device may be connected to the processor and the memory via a wired or wireless manner.

[0081] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0082] The memory is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs and modules. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, the image color correction method in the above method embodiment is implemented.

[0083] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the storage may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0084] The one or more modules are stored in the memory and when executed by the processor, perform the following steps: Figure 1-Figure 2 Method in the embodiment shown.

[0085] As an implementation method, the functions of the receiver and the transmitter in the present invention may be implemented by a transceiver circuit or a dedicated chip for transceiver, and the processor may be implemented by a dedicated processing chip, a processing circuit or a general-purpose chip.

[0086] As another implementation, it is possible to use a general-purpose computer to implement the authentication device and authentication server provided in the embodiment of the present invention, that is, the program code for implementing the functions of the processor, receiver, and transmitter is stored in a memory, and the general-purpose processor implements the functions of the processor, receiver, and transmitter by executing the code in the memory.

[0087] In summary, the method and device for improving the performance of Flink multi-stream association query described in the present invention, through the hierarchical design of integrating data flow, extraction, transformation and loading, wide table generation, data aggregation and storage, builds an efficient data processing pipeline, standardizes data flow and generates wide tables, reduces repeated association operations between multi-stream data, reduces state storage requirements, and optimizes the change writing of streaming data by using Kafka's Upsert capability to avoid the waste of resources caused by traditional duplicate data storage. By filtering data by topic and granularity at the summary data layer, the amount of calculation and redundancy of data transmission are reduced, state expansion and data divergence are reduced, computing resource consumption is significantly reduced, memory and storage occupancy are reduced, the processing logic of multi-stream association is simplified, and the maintainability of the system is improved.

[0088] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0089] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0090] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0091] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for improving the performance of Flink multi-stream association queries, characterized in that: The method is executed by a distributed computing Flink engine composed of multiple servers, and the method includes the following steps: The operational data processing layer reads the Kafka data stream of the data center and integrates the data streams from different sources into a unified format for output; The fact wide table layer extracts, transforms and loads the data stream in the unified format, and rewrites it into Kafka to generate fact wide table data through Upsert. The summary data layer filters and aggregates the fact wide table data according to preset topics and preset statistical granularity to generate multiple preset data summary tables; The preset data summary table is written into the Doris database by the summary data layer for the back-end interface to call.

2. The method for improving Flink multi-stream association query performance according to claim 1 is characterized in that: The data volume read by the operation data processing layer includes order flow and payment flow, and the preset data summary table includes: a ledger detail table, a real-time charging total table and a ledger charging information table.

3. The method for improving Flink multi-stream association query performance according to claim 1 is characterized in that: In the method, the data streams from different sources are integrated into a unified format for output, including: Clean the data stream to remove duplicate data, correct data outliers, and fill in missing data; Performing field mapping and conversion on the data stream, renaming the original fields and standardizing the formats; The fact wide surface layer performs extraction, transformation and loading operations on the data stream in a unified format, including: Performing data aggregation on the data stream, including grouping statistics, merging and calculation; The data stream is segmented to separate the complex structure into a plurality of preset simple fields.

4. The method for improving Flink multi-stream association query performance according to claim 1 is characterized in that: The method further comprises: Analyze the synchronization blocking points at the data processing layer, the fact wide table layer, the summary data layer, and the summary data layer, and find multiple target data flows that belong to synchronization processing; Based on Flink's asynchronous sending and receiving mechanism, the target data stream is optimized using a non-blocking operator chain, and a non-blocking back pressure mechanism is enabled for processing.

5. The method for improving Flink multi-stream association query performance according to claim 1 is characterized in that: The method further comprises: A message queue is used as an intermediary between the data processing layer, the fact wide table layer, the summary data layer, and the summary data layer, so that the data completed by each layer can be pushed to the next layer independently.

6. The method for improving Flink multi-stream association query performance according to claim 1 is characterized in that: The method further comprises: Use Flink's RocksDB state backend to support the persistence of the data stream, and store the intermediate processing state in Redis for downstream asynchronous queries.

7. The method for improving Flink multi-stream association query performance according to claim 4 is characterized in that: The method further comprises: Flink’s task indicator monitoring mechanism is introduced to collect data throughput and latency of the data processing layer, the fact wide table layer, the summary data layer, and the summary data layer, sample and analyze the processing time of the data flow at each layer, and adjust the asynchronous queue size and parallelism according to the processing time corresponding to each layer.

8. A device for improving the performance of Flink multi-stream association query, comprising a processor, a memory, and a computer program / instruction stored in the memory, characterized in that: The processor is used to execute the computer program / instructions. When the computer program / instructions are executed, the device implements the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method as claimed in any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images