Data processing method and device for stream processing framework, equipment and medium

Abstract

The invention provides a data processing method and device for a stream processing framework, equipment and a medium, and relates to the technical field of big data, in particular to the technical field of data warehouses. The method comprises the steps of pausing a real-time task of a stream processing framework in response to a first task control instruction, obtaining a first consumption data position corresponding to the real-time task when the real-time task is paused, and starting a first offline batch task, the first task control instruction is sent based on a real-time monitoring result of a real-time task state and consumption delay data in the streaming processing framework; after the capacity expansion operation is executed on the stream processing framework, the real-time task is started again, so that the real-time task starts to process data from a corresponding second consumption data position when the real-time task is started; wherein the first offline batch task is used for processing data between the first consumption data position and the second consumption data position. According to the invention, an integrated data warehouse of real-time stream and offline stream can be realized, and the concurrency and accuracy of data are improved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, specifically to the field of data warehouse technology, and in particular to a data processing method, device, equipment and medium for a stream processing framework. Background Art

[0002] With the development of the Internet, a large amount of data has surged. How to use and analyze this data, calculate and store it reasonably and efficiently, and lay a solid foundation for data decision-making and other behaviors has become a prominent problem to be solved.

[0003] At present, data warehouse construction is usually based on the stream processing framework to achieve data calculation and storage. However, the existing data warehouse construction solutions are still imperfect and cannot meet the requirements of low latency and high accuracy of data. Summary of the invention

[0004] The present application provides a data processing method, apparatus, device and medium for a stream processing framework, so as to solve the problem that the prior art cannot meet the requirements of low latency and high accuracy of data.

[0005] In a first aspect, the present application provides a data processing method for a stream processing framework, comprising:

[0006] In response to a first task control instruction, pausing a real-time task of a stream processing framework, obtaining a first consumption data location corresponding to the real-time task when the real-time task is suspended, and starting a first offline batch task, wherein the first task control instruction is issued based on a real-time monitoring result of a real-time task status and consumption delay data in the stream processing framework;

[0007] After performing the capacity expansion operation on the stream processing framework, the real-time task is started again, so that the real-time task starts processing data from the second consumption data position corresponding to the start-up;

[0008] The first offline batch task is used to process data between the first consumption data location and the second consumption data location.

[0009] In a second aspect, the present application provides a data processing device for a stream processing framework, comprising:

[0010] a first task control instruction processing module, configured to, in response to a first task control instruction, pause a real-time task of a stream processing framework, obtain a first data consumption position corresponding to the real-time task when paused, and start a first offline batch task, wherein the first task control instruction is issued based on a real-time monitoring result of a real-time task status and consumption delay data in the stream processing framework;

[0011] An expansion module, used to restart the real-time task after performing the expansion operation on the stream processing framework, so that the real-time task starts processing data from the second data consumption position corresponding to the start-up;

[0012] The first offline batch task is used to process data between the first data consumption position and the second data consumption position.

[0013] In a third aspect, the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the data processing method for a stream processing framework as described in any one of the embodiments of the present application is implemented.

[0014] In a fourth aspect, the present application further provides a computer-readable storage medium on which a computer program is stored, which, when executed by a processor, implements a data processing method for a stream processing framework as described in any one of the embodiments of the present application.

[0015] In a fifth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements a data processing method for a stream processing framework as described in any one of the embodiments of the present application.

[0016] The data processing method, device, equipment and medium for a stream processing framework provided in the present application monitor the real-time task status and consumption delay data in the stream processing framework, issue a first task control instruction according to the monitoring result, suspend the real-time task, obtain the first consumption data position corresponding to the real-time task when it is suspended, and start the first offline batch task. At the same time, the stream processing framework is expanded, and then the real-time task is restarted so that the real-time task starts processing data from the second consumption data position corresponding to the startup, and the first offline batch task processes the data between the first consumption data position and the second consumption data position. In this way, a data warehouse architecture integrating real-time streams and offline streams is realized to solve the consumption delay problem and improve the concurrency and processing efficiency of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 A flowchart of a data processing method for a stream processing framework provided in an embodiment of the present application;

[0019] Figure 2 A technical architecture diagram of the data processing method provided in the embodiment of the present application;

[0020] Figure 3 A schematic diagram of dimension table synchronization design in a data processing method provided in an embodiment of the present application;

[0021] Figure 4 A schematic diagram of a data storage design in a data processing method provided in an embodiment of the present application;

[0022] Figure 5 A schematic diagram of a cleaning conversion design in a data processing method provided in an embodiment of the present application;

[0023] Figure 6 A schematic diagram of a light aggregation design in a data processing method provided in an embodiment of the present application;

[0024] Figure 7 A schematic diagram of a highly aggregated design in a data processing method provided in an embodiment of the present application;

[0025] Figure 8 A schematic diagram of a data extraction design in a data processing method provided in an embodiment of the present application;

[0026] Fig. 9 A schematic diagram of the structure of a data processing device for a stream processing framework provided in an embodiment of the present application;

[0027] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0028] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0029] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only the parts related to the present application, rather than all structures, are shown in the accompanying drawings.

[0030] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance. The acquisition, storage, use, processing, etc. of data in the technical solution of this application comply with the relevant provisions of national laws and regulations. It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. The purpose is only to illustrate the feasibility of the implementation of the technical solution of this application, but it does not mean that the applicant has or must use the relevant content of the solution.

[0031] Figure 1 The flowchart of the data processing method for the stream processing framework provided in the embodiment of the present application. The method can be executed by a data processing device for the stream processing framework, which can be implemented in software and / or hardware, preferably configured in an electronic device, such as a computer device or a server. Figure 1 As shown, the method specifically includes:

[0032] S101. In response to a first task control instruction, suspend a real-time task of a stream processing framework, obtain a first consumption data position corresponding to the real-time task when suspended, and start a first offline batch task, wherein the first task control instruction is issued based on a real-time monitoring result of a real-time task status and consumption delay data in the stream processing framework.

[0033] S102. After the stream processing framework is expanded, the real-time task is started again, so that the real-time task starts processing data from the second consumption data position corresponding to the start-up; wherein the first offline batch task is used to process data between the first consumption data position and the second consumption data position.

[0034] Among them, the stream processing framework refers to a framework for real-time data processing, such as the Flink framework. In the prior art, data warehouses are usually built based on the stream processing framework to store and calculate data in real time, and data can be obtained from the database in real time in response to user query requests. However, the existing data warehouse construction technology solutions cannot meet the needs of large-scale data warehouses for the integration of real-time tasks and offline tasks, resulting in serious delays and affecting the efficiency of data storage and query.

[0035] In an embodiment of the present application, the stream processing framework is task monitored, specifically, the real-time task status and consumption delay data in the stream processing framework are monitored. Among them, the real-time task status refers to the processing speed and delay of the real-time task of the data stream, and the consumption delay data refers to the data consumption efficiency and delay of different layers in the stream processing framework, as well as the processing delay of user query requests. The monitoring results are used to reflect whether there are delays or exceptions in the tasks in the stream processing framework. During implementation, the judgment threshold for the real-time task status and consumption delay data can be pre-set, and when the monitoring result exceeds the threshold, it can be judged that a delay or anomaly has occurred. According to the monitoring results, the real-time tasks and offline tasks in the stream processing framework can be controlled, thereby solving the delay problem and improving efficiency.

[0036] Specifically, when it is determined according to the monitoring results that a task delay or anomaly occurs, a first task control instruction can be sent to the stream processing framework, and the real-time task of the stream processing framework can be suspended through the first task control instruction, and the first offline batch task can be started. At the same time, the first consumption data position corresponding to the real-time task when it is suspended is also obtained. The first consumption data position refers to the position of the data being consumed by the real-time task when it is suspended, for example, the position can be represented by a timestamp. Then, the stream processing framework is expanded and the expansion operation is performed. The specific expansion method includes, for example, adjusting the parameters of the stream processing framework, or adjusting the parameters such as CPU, memory or thread to achieve it. It can also refer to the execution method for expansion in the prior art, and this application does not make any limitation on this. After executing the expansion operation, the real-time task can be started again so that the real-time task starts processing data from the second consumption data position corresponding to the startup. The data between the first consumption data position and the second consumption data position is processed by the first offline batch task that is started. Therefore, on the one hand, the data delay problem can be alleviated by expanding the capacity, and on the other hand, while expanding the capacity, the first offline batch task is started to batch process the historical tasks during the suspension of the real-time task, so as not to affect the timeliness of the real-time data and avoid data delay in time. In this way, data processing is carried out in both real-time and offline lines to ensure the accuracy and timeliness of the data.

[0037] The technical solution of the embodiment of the present application monitors the real-time task status and consumption delay data in the stream processing framework, issues a first task control instruction according to the monitoring result, suspends the real-time task, obtains the first consumption data position corresponding to the real-time task when it is suspended, and starts the first offline batch task. At the same time, the stream processing framework is expanded, and then the real-time task is restarted so that the real-time task starts processing data from the second consumption data position corresponding to the startup, while the first offline batch task processes the data between the first consumption data position and the second consumption data position. In this way, a data warehouse architecture integrating real-time streams and offline streams is realized to solve the consumption delay problem and improve the concurrency and processing efficiency of data.

[0038] In addition, in an embodiment of the present application, the real-time task and the first offline batch task are also used to load the configuration data of the configuration system, wherein the configuration data includes at least the table structure and the data processing logic information. The configuration data in the configuration system can be pre-configured and changed at any time based on demand, involving changes to the logic of data processing. The real-time task and the first offline batch task both obtain the configuration data from the configuration system and perform data processing based on the configuration data, which can ensure that the real-time stream and the offline batch task use the same set of logic and mechanism to process the data, thereby realizing a data warehouse that integrates streams and batches, and ensuring the accuracy of the data. For example, when the real-time task is started again, the configuration data is first obtained from the configuration system, and then the real-time task performs data processing based on the configuration data. The first offline batch task obtains the configuration data from the configuration system after startup, and then performs data processing based on the configuration data.

[0039] In one embodiment, the method of the embodiment of the present application further includes:

[0040] In response to a second task control instruction, a second offline batch task is started so that the second offline batch task obtains the changed configuration data from the configuration system and refreshes the data of the specified historical time period based on the changed configuration data; wherein the second task control instruction is issued when the configuration data of the configuration system is changed.

[0041] Specifically, since it is possible to change the configuration data at any time according to different needs, in order to ensure the timeliness and accuracy of data processing in the embodiment of the present application, after the configuration data is changed, the real-time task continues to process the real-time data based on the latest configuration data. For the data of the specified historical time period, the second offline batch task is started by the control flow processing framework through the second task control instruction, and the second offline batch task obtains the changed configuration data from the configuration system, and flushes the data of the specified historical time period based on the changed configuration data, that is, the historical data is processed based on the new configuration data. In this way, not only the timeliness of data processing can be guaranteed by the real-time task, but also the accuracy of the data can be ensured by the flushing of the historical task by the second offline batch task.

[0042] In another embodiment, the method of the embodiment of the present application further includes:

[0043] In response to the third task control instruction, a third offline batch task is started, so that the third offline batch task processes the task of calculating the index with the set time span or the task of calculating the deduplication index based on the latest configuration data of the configuration system.

[0044] Specifically, when it is necessary to calculate an indicator with a longer time span, such as calculating the indicator statistics of 1 hour or 1 day, or when it is necessary to calculate some indicators such as deduplication, it is not suitable to accumulate the 1-minute calculation results of the real-time stream for the indicator tasks with a large time span and the deduplication indicator tasks, which affects the calculation efficiency. Therefore, the embodiment of the present application starts the third offline batch task, and allows the third offline batch task to perform offline batch processing on the tasks with large time span indicator calculations or the tasks of calculating deduplication indicators based on the latest configuration data of the configuration system, and output the calculation results to the offline table, thereby improving the efficiency of data calculation and processing, and at the same time does not affect the timeliness of the real-time task.

[0045] Figure 2 This is a technical architecture diagram of the data processing method provided in the embodiment of the present application. Figure 2 As shown, the stream processing framework of the embodiment of the present application is a Flink framework, which includes a dimension table layer (DIM layer), a source data layer (ODS layer), a detail data layer (DWD layer), a light aggregation layer (DWS layer) and an application data layer (ADS layer). The data of each layer is stored in an open table format Iceberg.

[0046] Specifically, the embodiment of the present application constructs a real-time and offline integrated data warehouse architecture based on Flink and Iceberg. It is generally divided into the access layer, computing layer, data warehouse storage layer and application layer. Among them, the access layer takes over the source data of the ODS layer and uses the Kafka message queue to receive the real-time data reported by the SDK. Flink consumes the data in Kafka in real time and stores it as ODS layer data in Iceberg.

[0047] The computing layer and data warehouse storage layer are responsible for processing and computing data, which can be divided into Flink cleaning conversion, Flink light aggregation and Flink high aggregation. Flink cleaning conversion refers to reading source data from Kafka in the ODS layer, filtering, translating, formatting, field expansion, field filling, stream JOIN merging and other operations, and outputting them to the downstream Iceberg message queue as the data of the DWD detail data layer after processing. Flink light aggregation refers to reading detail data from Iceberg in the DWD layer, lightly aggregating according to the fine-grained dimensions of the business, and performing field translation and conversion with the dimension table configuration of the DIM layer, and outputting the aggregated data to the downstream Iceberg as the DWS layer data. The dimension table configuration data stored in the DIM layer is based on user dimension data and is obtained through data synchronization through Flink CDC (Change Data Capture) technology. Flink high aggregation refers to a Flink task used to consume the result data of the DWS layer, and highly aggregate based on the coarse-grained dimension, and output the aggregated data to the downstream Iceberg as the ADS layer data.

[0048] The application layer mainly applies data to scenarios such as real-time dashboards, business monitoring, online algorithm learning, offline algorithm learning, and data services. Among them, the real-time dashboard mainly connects to the data query of the DWS layer and the ADS layer. Business monitoring mainly connects to the lightly aggregated data of the DWS layer. Online and offline learning mainly connects to the detailed data of the DWD layer.

[0049] Figure 3 Schematic diagram of dimension table synchronization design in the data processing method provided in the embodiment of the present application. Figure 3 As shown, the dimension table synchronization design in the embodiment of the present application is mainly aimed at how to synchronize dimension data efficiently and incrementally. Specifically, using Flink CDC technology, two Source operators are used to read the full and incremental BinLog logs of the business-side Mysql dimension data and Mongodb dimension data, respectively, to achieve data synchronization. At the same time, since Mysql is structured data and MongoDB is unstructured data, it is also necessary to use the Trans conversion operator to heterogeneously merge the data of two different data object types and perform dimension conversion, and finally output it to the downstream Iceberg dimension table through the Sink operator to obtain the Iceberg dimension conversion data.

[0050] In addition, the embodiments of the present application support millisecond-level synchronization of added, deleted, and modified data. For example, when data is deleted, the rollback flow is used to delete the historical data first and then write the new data in, so as to avoid duplication of data. At the same time, this task will synchronize the full amount of historical data when it is first started, and the subsequent synchronization can be performed incrementally to save synchronization computing resources.

[0051] Figure 4 This is a schematic diagram of the data warehousing design in the data processing method provided in the embodiment of the present application. Figure 4 As shown, Flink's real-time stream consumes data in the Kafka message queue in real time and adds a receiving timestamp, which can be used to determine the difference between the actual data generation time and the reception time, so as to evaluate the data latency based on the difference. At the same time, the IP address of the receiving machine can also be added for subsequent analysis of the machine's data reception. Flink data warehousing requires the Source operator to read data, the Trans operator to perform object conversion, and the Sink operator to output data. No additional processing is done on the original data, and it is stored in the ODS layer table of the Iceberg business data source.

[0052] In addition, Flink tasks will be monitored and the results of task monitoring will be reported to Granafa (an open source visualization tool) and ElasticSearch (an open source full-text search engine, ES for short) for storage and display. At the same time, real-time result monitoring of the result data entering the warehouse will be added to monitor the number of data entries entered, the timeliness of the entry, etc., and real-time alarms will be issued according to traffic fluctuations to monitor data quality and timeliness.

[0053] Figure 5 This is a schematic diagram of the cleaning conversion design in the data processing method provided in the embodiment of the present application. Figure 5As shown in the figure, the data of the ODS layer is stored in Iceberg, and the Flink real-time stream will have two Source operators, namely Source1 operator and Source2 operator. Among them, the Source1 operator is used to read the configuration data of the configuration system, which mainly stores the table structure, value verification logic, dimension expansion logic, etc. At the same time, Source1 also broadcasts the configuration data to the downstream operators, so that the downstream operators can also process based on the same set of rules to ensure the accuracy of the data. The Source2 operator is used to read the real-time stream business data source of the ODS layer. After receiving the real-time business data source and configuration data, the cleaning operator performs data type verification and value verification, performs dimension expansion operations on the json field, and outputs the cleaned normal data to the downstream Iceberg DWD detail table, and stores the abnormal data in the backup table for data verification. At the same time, when the business logic changes, the Flink real-time stream can consume and process data with the latest logic.

[0054] In the design of cleaning conversion, the embodiment of the present application realizes the integrated design of stream and batch. Task monitoring is performed for Flink cleaning conversion, and the real-time task status and consumption delay in Flink are monitored in real time. When the following situations occur, this monitoring task will initiate instructions to control the start and stop of real-time tasks and offline batch tasks:

[0055] (1) When a real-time task is abnormal and the consumption delay is too large (for example, the offset is greater than 1000W), the monitoring task will send instructions to Flink to suspend the real-time task and expand Flink's parallelism and resources by 30%, that is, to perform a capacity expansion operation. After the capacity expansion, the real-time task is started again, so that the real-time task starts consuming from the latest offset, which does not affect the timeliness of the real-time data. At the same time, an offline batch task will be started to consume the data of this offset segment with a large delay, thereby avoiding excessive data delay. The offline batch task will load the configuration data of the upstream rule configuration system and the business behavior data of the ODS layer in batches, so as to complete the historical data as soon as possible. In this way, the two real-time and offline SQLs with the same calculation logic process the data of different upstream offsets and store them in the database, ensuring the accuracy and timeliness of the data.

[0056] (2) When the computing logic changes, the rules of the rule configuration system will also change, and the real-time stream will automatically obtain the latest computing rules to process the data. At the same time, you can issue instructions to Flink through task monitoring to let it automatically start an offline batch task to process the data, that is, to flush the data of the specified historical time period. In this way, the real-time and offline SQLs with the same computing logic process data in different time periods and store them in the database, ensuring the accuracy and timeliness of the data.

[0057] (3) The real-time stream will store and display the performance indicators of each operator stage (such as CPU consumption, memory consumption, disk consumption and IO consumption, etc.), business indicators (such as the amount of indicator 1, the amount of indicator 2, etc.), and log data (such as task info log, warn log, error log) in Granafa and ElasticSearch, making it convenient for the market to observe and compare the changes in indicator fluctuations and for subsequent task anomaly troubleshooting and data reconciliation.

[0058] Figure 6 This is a schematic diagram of a light aggregation design in a data processing method provided in an embodiment of the present application. Figure 6 As shown in the figure, three operators are started in Flink light aggregation, namely Source1 operator, Source2 operator and Source3 operator. Source1 operator obtains the configuration data of the configuration system, Source2 operator and Source3 operator consume the dimension table data of the DIM layer and the detail table data of the DWD layer in Iceberg respectively. Specifically, multi-stream association is performed through the following steps:

[0059] (1) The Source1 operator is mainly responsible for reading the configuration data of the indicator configuration system through the API interface, which mainly includes the dimensions of each business, indicator aggregation rules and storage methods. After reading the configuration, it broadcasts the configuration data to the downstream operators and stores it in Flink's State1 (Flink's intermediate calculation result storage structure) for sharing by other operators.

[0060] (2) The Source2 operator is mainly responsible for reading Iceberg dimension data, that is, DIM layer data, which is stored in Flink's State2 and waits for use by the downstream Join operator.

[0061] (3) The Source3 operator is mainly responsible for reading Iceberg detailed data, that is, DWD layer data. After the data flows to the Join operator, it will be stored in Flink's State3. At this time, the dimension Name value in State2 will be matched according to the dimension ID. The data is output to the next Calc aggregation operator, which obtains the calculation rules in State1 corresponding to the current business, and then aggregates with the results of the Join operator. For example, a 1-minute aggregation window is opened to aggregate the corresponding 2 indicators and 3 dimensions. Finally, the results are output to the downstream Iceberg light aggregation layer DWS layer storage through the Sink operator.

[0062] In the light aggregation design, the embodiment of the present application realizes the integrated design of stream and batch. Task monitoring is performed for Flink light aggregation, and the real-time task status and consumption delay in Flink are monitored in real time. When the following situations occur, this monitoring task will initiate instructions to control the start and stop of real-time tasks and offline batch tasks:

[0063] (1) When a real-time task is abnormal and the consumption delay is too large (for example, the offset is greater than 1000W), the monitoring task will send instructions to Flink to suspend the real-time task and expand Flink's parallelism and resources by 30%, that is, to perform a capacity expansion operation. After the capacity expansion, the real-time task is started again, so that the real-time task starts consuming from the latest offset, which does not affect the timeliness of the real-time data. At the same time, an offline batch task will be started to consume the data of this offset segment with a large delay, thereby avoiding excessive data delay. The offline batch task will load the configuration data, dimension data and detailed data of the upstream rule configuration system in batches, which plays a role in completing historical data as soon as possible and ensuring the accuracy of indicator calculations. In this way, the two SQLs with the same calculation logic, real-time and offline, process data from different upstream offsets and store them in the database, ensuring the accuracy and timeliness of the data.

[0064] (2) When the computing logic changes, the rules of the rule configuration system will also change, and the real-time stream will automatically obtain the latest computing rules to process the data. At the same time, you can issue instructions to Flink through task monitoring to let it automatically start an offline batch task to process the data, that is, to flush the data of the specified historical time period. In this way, the real-time and offline SQLs with the same computing logic process data in different time periods and store them in the database, ensuring the accuracy and timeliness of the data.

[0065] (3) When it is necessary to calculate indicators with a longer time span, such as calculating indicator statistics for 1 hour or 1 day, and when it is necessary to calculate some indicators such as deduplication (deduplication indicators are not suitable for accumulation from real-time 1-minute calculation results), you can start an offline batch task to read the indicator configuration system data and lightly aggregated data, and then calculate and output them to the offline table. This can improve the efficiency of data calculation without affecting the timeliness of real-time tasks, ensuring the accuracy and consistency of offline indicator calculations.

[0066] (4) The real-time stream will store and display the performance indicators of each operator stage (such as CPU consumption, memory consumption, disk consumption and IO consumption, etc.), business indicators (such as the amount of indicator 1, the amount of indicator 2, etc.), and log data (such as task info log, warn log, error log) in Granafa and ElasticSearch, making it convenient for the market to observe and compare the changes in indicator fluctuations and for subsequent task anomaly troubleshooting and data reconciliation.

[0067] Figure 7 This is a schematic diagram of a highly aggregated design in the data processing method provided in the embodiment of the present application. Figure 7 As shown, similarly, the configuration data of the indicator configuration system is obtained through the Source operator. The Flink real-time task consumes the data of the DWS layer in the upstream Iceberg through the real-time stream, reads these behavior data through the Source operator, aggregates them using a 1-minute window through the Calc operator, and outputs the coarse-grained aggregated data to the downstream ADS layer through the Sink operator. If the real-time task encounters a logic change, the batch task with the same code logic can flush the data and aggregate the data at the day granularity, and output it to the minute-level table or day-level table corresponding to the ADS layer of Iceberg (converting fine-grained data to coarse-grained data). At the same time, the monitoring of the Flink task will be reported to Granafa and ElasticSearch for storage and display.

[0068] In the highly aggregated design, the embodiment of the present application realizes the integrated design of stream and batch. Task monitoring is performed for Flink highly aggregated, and the real-time task status and consumption delay in Flink are monitored in real time. When the following situations occur, this monitoring task will initiate instructions to control the start and stop of real-time tasks and offline batch tasks:

[0069] (1) When a real-time task is abnormal and the consumption delay is too large (for example, the offset is greater than 1000W), the monitoring task will send instructions to Flink to suspend the real-time task and expand Flink's parallelism and resources by 30%, that is, to perform a capacity expansion operation. After the capacity expansion, the real-time task is started again, so that the real-time task starts consuming from the latest offset, which does not affect the timeliness of the real-time data. At the same time, an offline batch task will be started to consume the data of this offset segment with a large delay, thereby avoiding excessive data delay. The offline batch task will load the configuration data of the upstream rule configuration system and the business behavior data of the ODS layer in batches, so as to complete the historical data as soon as possible. In this way, the two real-time and offline SQLs with the same calculation logic process the data of different upstream offsets and store them in the database, ensuring the accuracy and timeliness of the data.

[0070] (2) When the computing logic changes, the rules of the rule configuration system will also change, and the real-time stream will automatically obtain the latest computing rules to process the data. At the same time, you can issue instructions to Flink through task monitoring to let it automatically start an offline batch task to process the data, that is, to flush the data of the specified historical time period. In this way, the real-time and offline SQLs with the same computing logic process data in different time periods and store them in the database, ensuring the accuracy and timeliness of the data.

[0071] (3) When it is necessary to calculate indicators with a longer time span, such as calculating the indicator statistics of 1 hour or 1 day, and when it is necessary to calculate some indicators such as deduplication (deduplication indicators are not suitable for accumulation from real-time 1 minute calculation results), it is more appropriate to start an offline batch task to read the indicator configuration system data and highly aggregated data, and then calculate and output to the offline table. At the same time, it does not affect the timeliness of the real-time task, and ensures the accuracy and consistency of offline indicator calculation.

[0072] (4) The real-time stream will store and display the performance indicators of each operator stage (such as CPU consumption, memory consumption, disk consumption and IO consumption, etc.), business indicators (such as the amount of indicator 1, the amount of indicator 2, etc.), and log data (such as task info log, warn log, error log) in Granafa and ElasticSearch, making it convenient for the market to observe and compare the changes in indicator fluctuations and for subsequent task anomaly troubleshooting and data reconciliation.

[0073] Figure 8 The data extraction design diagram in the data processing method provided in the embodiment of the present application is mainly aimed at how to extract DWD layer data, DWS layer data or ADS layer data to achieve the purpose of data application. Figure 8 As shown, for offline learning and online learning scenarios, downstream tasks can consume detailed data from the DWD layer in Iceberg to perform learning calculations, because this layer has relatively detailed basic information and label information. For scenarios such as business monitoring, real-time dashboards, data analysis, and data services, the DWS layer and ADS layer can be used to import data stored in Clickhouse through real-time tasks for millisecond-level queries to speed up the query efficiency of the data provided by the entire data warehouse. The import tool can read data through multiple consumer ends that support high concurrency and write them to Clickhouse separately. Instead of going through the Clickhouse proxy node, it is directly written to the corresponding coarse node after Hash bucketing, which improves the write throughput and is stored in different tables. Of course, the monitoring of the entire data extraction process will be reported to Granafa and ElasticSearch for storage and display.

[0074] Next, the purposes of different data applications are introduced:

[0075] (1) Offline learning: This means that the algorithm needs to obtain the feature data of the target object offline and conduct model learning at least one day in advance. The advantage is that the amount of historical data is large, the learning time is sufficient, and it does not affect online business. The image features of the target object can be obtained from the historical data, and ultimately advertising recommendations and other services can be provided to the target object more accurately. The data that needs to be extracted here is read from Iceberg in the DWD detail layer, which contains very detailed target feature data.

[0076] (2) Online learning: This means that the algorithm needs to obtain the feature data of the target object in real time and conduct real-time learning. The benefit is that it can capture the current feature status of the target object in real time and make real-time recommendations. However, the amount of data should not be too large, otherwise it will affect the timeliness of online business. The data that needs to be extracted here is read from Iceberg in the DWD detail layer, which contains very detailed feature detail data.

[0077] (3) Data analysis: This means querying indicator data by providing the editing function of the SQL window, which allows for more flexible data analysis at each level and convenient tracking of indicators and dimensions to troubleshoot problems.

[0078] (4) Business monitoring: This means monitoring the fluctuation of business indicators to detect data problems or service problems. For example, you need to monitor whether the exposure of a business has doubled or doubled, which reflects some problems that may exist in the business or technical services, and you need to issue warnings and fix the problems in a timely manner.

[0079] (5) Real-time dashboard: refers to a real-time BI dashboard that queries the data in Clickhouse in real time and displays it on a large screen. This can improve the timeliness of the query and also allow real-time viewing of data changes so that development or product response measures can be taken in real time.

[0080] (6) Data service: refers to providing data services to relevant personnel on demand by providing interfaces. Of course, some interface frequency limiting functions are also provided here to prevent service crashes caused by excessive requests. Data can also be queried remotely, achieving the purpose of flexible and multi-purpose use of data.

[0081] In summary, the technical solution of the embodiment of the present application provides a complete data warehouse construction method for medium or large data construction, including how to design a real-time / offline stream-batch integrated data warehouse architecture based on Flink and Iceberg. First, the data warehouse is layered in detail, namely, it is divided into ODS source data layer, DIM dimension table layer, DWD detail data layer, DWS light aggregation layer and ADS application data layer. At the same time, from the data link level, it is divided into access layer, computing layer, intermediate result layer, storage layer and display layer. The data warehouse layering aims to decouple data modules, reduce repeated development costs, improve report usage efficiency, increase data utilization, and improve data development efficiency. The use of Flink's unified stream-batch architecture can reduce code maintenance and development costs.

[0082] Fig. 9 A schematic diagram of the structure of a data processing device for a stream processing framework provided in an embodiment of the present application. Fig. 9 As shown, the device 90 comprises:

[0083] A first task control instruction processing module 901 is used to, in response to a first task control instruction, suspend the real-time task of the stream processing framework, obtain a first data consumption position corresponding to the real-time task when suspended, and start a first offline batch task, wherein the first task control instruction is issued based on a real-time monitoring result of the real-time task status and consumption delay data in the stream processing framework;

[0084] The expansion module 902 is used to restart the real-time task after performing the expansion operation on the stream processing framework, so that the real-time task starts processing data from the second data consumption position corresponding to the start-up;

[0085] The first offline batch task is used to process data between the first data consumption position and the second data consumption position.

[0086] In some embodiments, the real-time task and the first offline batch task are also used to load configuration data of a configuration system, wherein the configuration data at least includes a table structure and data processing logic information.

[0087] In some embodiments, the apparatus further comprises:

[0088] a second task control instruction processing module, configured to start a second offline batch task in response to the second task control instruction, so that the second offline batch task obtains the changed configuration data from the configuration system, and refreshes the data of the specified historical time period based on the changed configuration data;

[0089] The second task control instruction is issued when the configuration data of the configuration system is changed.

[0090] In some embodiments, the apparatus further comprises:

[0091] The third task control instruction processing module is used to start a third offline batch task in response to a third task control instruction, so that the third offline batch task processes the task of calculating an indicator with a set time span or the task of calculating a deduplication indicator based on the latest configuration data of the configuration system.

[0092] In some embodiments, the stream processing framework is a Flink framework; the Flink framework includes a dimension table layer, a source data layer, a detail data layer, a light aggregation layer and an application data layer.

[0093] In some embodiments, the light aggregation layer is used to lightly aggregate data based on the detail data of the detail data layer and the dimension table data of the dimension table layer.

[0094] In some embodiments, the data of the dimension table layer, source data layer, detail data layer, light aggregation layer and application data layer are stored in an open table format Iceberg.

[0095] The data processing device for a stream processing framework provided in an embodiment of the present application can be used to execute the technical solution of the data processing method for a stream processing framework in the above-mentioned embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0096] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the first task control instruction processing module 901 can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The function of the first task control instruction processing module 901. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0097] Fig.10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.10As shown, the electronic device may include: a transceiver 121 , a processor 122 , and a memory 123 .

[0098] The processor 122 executes the computer execution instructions stored in the memory, so that the processor 122 executes the scheme in the above embodiment. The processor 122 can be a general-purpose processor, including a central processing unit CPU, a network processor (network processor, NP), etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0099] The memory 123 is connected to the processor 122 via a system bus and completes communication between them. The memory 123 is used to store computer program instructions.

[0100] The transceiver 121 may be used to obtain tasks to be executed and configuration information of the tasks to be executed.

[0101] The system bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The transceiver is used to realize the communication between the database access device and other computers (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM) and may also include non-volatile memory.

[0102] The electronic device provided in the embodiment of the present application may be the terminal device of the above embodiment.

[0103] An embodiment of the present application also provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the technical solution of the data processing method for a stream processing framework in the above embodiment.

[0104] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, it can implement the technical solution of the data processing method for the stream processing framework in the above embodiment.

[0105] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present application, and the programming language includes an object-oriented programming language, such as Java, Smalltalk, C++, and also includes a conventional procedural programming language, such as "C" language or similar programming language. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0106] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A data processing method for a stream processing framework, characterized in that: include: In response to a first task control instruction, pausing a real-time task of a stream processing framework, obtaining a first consumption data location corresponding to the real-time task when the real-time task is suspended, and starting a first offline batch task, wherein the first task control instruction is issued based on a real-time monitoring result of a real-time task status and consumption delay data in the stream processing framework; After performing the capacity expansion operation on the stream processing framework, the real-time task is started again, so that the real-time task starts processing data from the second consumption data position corresponding to the start-up; The first offline batch task is used to process data between the first consumption data location and the second consumption data location.

2. The method according to claim 1, characterized in that The real-time task and the first offline batch task are also used to load configuration data of a configuration system, wherein the configuration data at least includes a table structure and data processing logic information.

3. The method according to claim 2, characterized in that Also includes: In response to the second task control instruction, start a second offline batch task, so that the second offline batch task obtains the changed configuration data from the configuration system, and refreshes the data of the specified historical time period based on the changed configuration data; The second task control instruction is issued when the configuration data of the configuration system is changed.

4. The method according to claim 2, characterized in that: Also includes: In response to the third task control instruction, a third offline batch task is started, so that the third offline batch task processes the task of calculating the index with the set time span or the task of calculating the deduplication index based on the latest configuration data of the configuration system.

5. The method according to claim 1, characterized in that The stream processing framework is a Flink framework; the Flink framework includes a dimension table layer, a source data layer, a detail data layer, a light aggregation layer and an application data layer.

6. The method according to claim 5, characterized in that The light aggregation layer is used to perform light aggregation on data based on the detail data of the detail data layer and the dimension table data of the dimension table layer.

7. The method according to claim 5, characterized in that The data of the dimension table layer, source data layer, detail data layer, light aggregation layer and application data layer are stored in an open table format Iceberg.

8. A data processing device for a stream processing framework, characterized in that: include: a first task control instruction processing module, configured to, in response to a first task control instruction, pause a real-time task of a stream processing framework, obtain a first data consumption position corresponding to the real-time task when paused, and start a first offline batch task, wherein the first task control instruction is issued based on a real-time monitoring result of a real-time task status and consumption delay data in the stream processing framework; An expansion module, used to restart the real-time task after performing the expansion operation on the stream processing framework, so that the real-time task starts processing data from the second data consumption position corresponding to the start-up; The first offline batch task is used to process data between the first data consumption position and the second data consumption position.

9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the data processing method for a stream processing framework according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the data processing method for a stream processing framework according to any one of claims 1 to 7.

11. A computer program product, characterized in that It comprises a computer program, which, when executed by a processor, implements the data processing method for a stream processing framework according to any one of claims 1 to 7.

Images