Streaming data processing method, system and equipment based on Flink and medium

Abstract

The invention provides a streaming data processing method, system and device based on Flink and a medium, and belongs to the technical field of data processing.The method comprises the following steps that data streams are received from different data sources, a connection pool is established according to the types of the data sources, and data in the data streams are processed; dividing the data stream into a plurality of windows for processing based on event time, storing the processing state of the data stream in an increment form, and performing partition management; task faults in the data processing process are detected, task restarting and load balancing are automatically carried out, meanwhile, data stream processing delay is monitored, and computing resources are dynamically adjusted when the delay exceeds the standard; and monitoring the state of the data stream in real time, carrying out visual display, and giving an alarm when the data stream is abnormal. According to the method, mass real-time data streams from different data sources are received and preprocessed, so that efficient and stable processing is realized; and the stability and the real-time performance of the system are guaranteed by means of mechanisms such as fault detection and load balancing.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and specifically relates to a streaming data processing method, system, device and medium based on Flink. Background Art

[0002] With the development of the Internet and the Internet of Things, the demand for processing real-time data streams is growing. For example, the data processing platform of a certain type of enterprise needs to process massive data streams from sensors, user behavior logs, financial transactions, and other scenarios, so as to conduct real-time analysis and decision-making. The traditional batch processing method has high processing latency and poor real-time performance. It is difficult to cope with the real-time processing needs of massive data streams, and it cannot meet the business needs of modern application scenarios.

[0003] Flink is the abbreviation of Apache Flink. It is a distributed data processing engine that can process real-time streaming data and batch data, can process unbounded data streams, and has the advantages of low latency, high throughput and fault tolerance. It can provide a certain degree of state management and fault tolerance. However, in the processing of large-scale real-time data streams, Flink still has data partition imbalance, complex state management, and fault recovery bottlenecks, which cannot meet the processing requirements of large-scale real-time data streams. First, if the amount of data in some partitions is much larger than that in other partitions, it will cause the processing nodes where these partitions are located to be overloaded, while other nodes are idle, which not only reduces the overall throughput of the data processing platform, but also may cause increased processing delays. Secondly, in Flink's stream processing framework, full state storage is usually used, which not only takes up a lot of storage space, but also increases the network transmission overhead during state recovery. Thirdly, traditional fault recovery solutions usually rely on full snapshots or log replay, which are insufficient in terms of response time, resource consumption and consistency guarantee. Especially in the processing of large-scale real-time data streams, the long fault recovery time leads to data loss or increased processing delays. Summary of the invention

[0004] In a first aspect, an embodiment of the present application provides a streaming data processing method based on Flink, comprising the following steps: S1. Receive data streams from different data sources, establish a connection pool of corresponding size for each data source according to the data source type, and decode, format and preliminarily clean the data in the data stream; S2. Divide the data stream into several windows for processing based on event time, store the data stream processing status in incremental form, and manage the data stream processing status by partition; S3. Detect task failures during data processing, automatically restart tasks and load balance, monitor data stream processing delays, and dynamically adjust computing resources when delays exceed the limit; S4. Monitor the data flow status in real time and display it visually, and issue an alarm when the data flow is abnormal.

[0005] Furthermore, the specific steps of step S1 are as follows: S11. Determine the data source type of the data processing platform, and create a connection pool for each type of data source according to the processing requirements, and create the required number of processing threads; S12. Identify the data format of each type of data source, and select an appropriate encoder for each type of data to convert the format, converting the data source to a unified standard data format of the data processing platform; S13. Clean the data stream that has completed the format conversion, identify and filter out null values, abnormal values, and data that does not conform to the format according to the data source type.

[0006] Furthermore, the specific steps of step S2 are as follows: S21. Determine the current time period; If the current time period is a peak period, proceed to step S22; If the current time period is a low period, proceed to step S23; If the current time period is a normal time period, proceed to step S24; S22. The preset window size is adjusted according to the preset increase, and the process proceeds to step S24; S23. The preset window size is adjusted according to the preset reduction amplitude, and the process proceeds to step S24; S24. Divide the data stream into windows according to the preset window size according to the trigger event time of each type of data source, and set the window watermark according to the maximum delay time of each type of data source; S25. Assign the corresponding processing thread in the connection pool to the data in each type of window in the data stream for processing, and wait for data events according to the window watermark; S26. Record the difference in the number of events during the window sliding process of data stream processing, and store them in partitions by stream.

[0007] Furthermore, the specific steps of step S3 are as follows: S31 monitors the execution tasks of the computing nodes where the processing threads of various types of data sources are located. When a computing node failure is detected, an available computing node is found for failure recovery; S32. Monitor the processing delay of the type data source, and make dynamic resource adjustments when the processing delay exceeds the set delay threshold upper limit.

[0008] Furthermore, the specific steps of step S31 are as follows: S311. Record the event time of successful execution of processing threads of each type of data source; S312. Perform heartbeat monitoring on the computing nodes where the processing threads of various data sources are located; If the heartbeat monitoring is normal, proceed to step S313; If the heartbeat monitoring is abnormal, go to step S313; S313. Analyze the abnormal logs of the computing nodes where the processing threads of various data sources are located to determine whether insufficient resources are identified; If yes, go to step S314; If not, proceed to step S32; S314. Determine the current computing node as a faulty computing node, and select one computing node from the computing nodes with a load rate higher than a threshold as a target computing node; S315. Migrate the processing thread of the failed computing node to the target computing node, and restore the data stream processing according to the last saved successful execution event time.

[0009] Furthermore, the specific steps of step S32 are as follows: S321. Count the event delays of various types of data sources in the statistical data flow and calculate the average delay time, where the event delay includes flow delay and processing delay; S322. Monitor the real-time processing time of events of various types of data sources in the data stream to determine whether it is greater than the average delay time and the duration is greater than the set time threshold; If yes, go to step S323; If not, proceed to step S4; S323. Increase the number of computing nodes where the event corresponding processing thread is located.

[0010] Further, step S4 is specifically as follows: S41. Statistics on the processing status of each type of data source, the processing status includes delay, rate, and event data; S42. Generate a chart of the processing status of each type of data source in the data stream and display it in a graphical interface; S43. Generate alarm information for abnormal situations in the data flow processing process and notify the operation and maintenance personnel in a preset manner.

[0011] In a second aspect, an embodiment of the present application further provides a streaming data processing system based on Flink, including: The data source management module is used to receive data streams from different data sources, establish a connection pool of corresponding size for each data source according to the data source type, and decode, format and preliminarily clean the data in the data stream; The data window and processing state management module is used to divide the data stream into several windows for processing based on event time, store the data stream processing state in incremental form, and manage the data stream processing state by partition; Fault recovery and delay monitoring module, which is used to detect task failures during data processing, automatically restart tasks and balance loads, monitor data stream processing delays, and dynamically adjust computing resources when delays exceed the limit; The real-time monitoring module is used to monitor the data flow status in real time and display it visually, as well as to issue an alarm when the data flow is abnormal.

[0012] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the Flink-based streaming data processing method as described in the first aspect are implemented.

[0013] In a fourth aspect, an embodiment of the present application further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the Flink-based streaming data processing method as described in the first aspect are implemented.

[0014] It can be seen from the above technical solutions that the present invention has the following advantages: The Flink-based streaming data processing method, system, device and medium provided in this application receive data streams from different data sources, perform preprocessing such as decoding, formatting and cleaning, perform window division and incremental state storage based on event time, and realize efficient, stable and reliable processing of massive real-time data streams. At the same time, the stability and real-time performance of the system are ensured through fault detection, automatic restart, load balancing and dynamic resource adjustment. The maintainability and reliability of the system are improved through real-time monitoring and visual display. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0016] Figure 1 It is a flow chart of the streaming data processing method based on Flink of the present invention.

[0017] Figure 2 It is a schematic diagram of a Flink-based streaming data processing system of the present invention. DETAILED DESCRIPTION

[0018] In the specific steps of the streaming data processing method based on Flink, which will be described in detail below, various embodiments of the present disclosure will be described more comprehensively. The present disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of the present disclosure to the specific embodiments disclosed herein, but the present disclosure should be understood to cover all adjustments, equivalents and / or alternatives that fall within the spirit and scope of the various embodiments of the present disclosure.

[0019] For example, with the continuous evolution of Internet technology and IoT applications, the demand for processing real-time data streams is rising sharply. Take a certain enterprise as an example, its data processing platform needs to cope with massive data torrents from multiple scenarios such as sensor monitoring, user behavior tracking, and financial transaction records to achieve real-time data analysis and business decisions. However, the traditional batch processing mode has significant processing delays and insufficient real-time response capabilities, and is no longer competent for the current real-time processing tasks of massive data streams, let alone the stringent requirements of modern business scenarios for immediacy.

[0020] Apache Flink, or Flink for short, is a distributed data processing engine that has the ability to process both real-time streaming data and batch data, and is particularly good at processing unbounded data streams. Flink is known for its low latency, high throughput, and strong fault tolerance, while also providing a certain degree of state management and fault tolerance. However, Flink still faces many challenges when dealing with large-scale real-time data streams. Data partition imbalance is one of them: when some partitions carry much more data than other partitions, the processing nodes where these overloaded partitions are located will be under tremendous pressure, while other nodes may be idle, which not only reduces the throughput of the entire data processing platform, but may also cause an increase in processing delays.

[0021] In addition, Flink's stream processing framework usually adopts a full state storage strategy, which not only occupies a huge amount of storage space resources, but also increases the burden of network transmission during state recovery. Furthermore, traditional fault recovery methods, such as full snapshots or log replay, have obvious shortcomings in terms of recovery time, resource consumption, and data consistency assurance. Especially when processing large-scale real-time data streams, the extension of fault recovery time may lead to further data loss or processing delays.

[0022] In view of the above problems, this embodiment provides a streaming data processing method based on Flink, which can achieve higher processing efficiency and better fault tolerance in streaming computing by optimizing Flink's state management, data partitioning strategy and fault recovery mechanism.

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] See also Figure 1 FIG. 1 is a flowchart of a streaming data processing method based on Flink in a specific embodiment, and the method includes the following steps: S1. Receive data streams from different data sources, establish a connection pool of corresponding size for each data source according to the data source type, and decode, format and preliminarily clean the data in the data stream; It should be noted that by establishing a connection pool and processing threads, the parallelism of data reception and processing is improved; by decoding, formatting and cleaning data, the quality of data for subsequent processing is ensured; S2. Divide the data stream into several windows for processing based on event time, store the data stream processing status in incremental form, and manage the data stream processing status by partition; It should be noted that by dividing windows based on event time, the fluctuation of data streams can be adapted; by incremental state storage and partition management, processing efficiency and fault tolerance can be improved; S3. Detect task failures during data processing, automatically restart tasks and load balance, monitor data stream processing delays, and dynamically adjust computing resources when delays exceed the limit; It should be noted that the stability of the system is ensured through fault detection and automatic restart; the real-time performance and throughput of the system are improved through load balancing and dynamic resource adjustment; S4. Monitor the data flow status in real time and display it visually, and issue an alarm when the data flow is abnormal; It should be noted that real-time monitoring and visual display improve the maintainability and reliability of the system; by generating alarm information, operation and maintenance personnel are notified in time to handle abnormal situations.

[0025] This embodiment receives data streams from different data sources and performs pre-processing such as decoding, formatting, and cleaning to provide high-quality data for subsequent processing. At the same time, it performs window division, incremental state storage, and partition management based on event time, effectively improving data processing efficiency and fault tolerance. Fault detection and dynamic resource adjustment ensure the stability and real-time performance of the system.

[0026] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to fully illustrate the specific implementation process in this embodiment, another streaming data processing method based on Flink is provided, and the method includes the following steps: S1. Receive data streams from different data sources, establish a connection pool of corresponding size for each data source according to the data source type, and decode, format and preliminarily clean the data in the data stream; the specific steps of step S1 are as follows: S11. Determine the data source type of the data processing platform, and create a connection pool for each type of data source according to the processing requirements, and create the required number of processing threads; For example, taking an e-commerce platform as an example, data sources may include Kafka (for user behavior logs), MySQL (for transaction records), and HDFS (for historical data backup); 100 threads are configured for the Kafka connection pool and 50 connections are configured for the MySQL connection pool to ensure data access capabilities under high concurrency; S12. Identify the data format of each type of data source, and select an appropriate encoder for each type of data to convert the format, converting the data source to a unified standard data format of the data processing platform; For example, user behavior logs are stored in Kafka in JSON format and converted to the internal format through a JSON decoder; transaction records are stored in MySQL in Avro format and converted through an Avro decoder; S13. Clean the data stream after format conversion, identify and filter out null values, abnormal values, and data that does not conform to the format according to the data source type; For example, the incoming user behavior logs are cleaned to filter out invalid and abnormal data, such as null values, format mismatches, abnormal values, etc.; the transaction records are cleaned to filter out invalid transaction records, such as repeated transactions, abnormal transactions, etc.; It should be noted that by creating a connection pool and processing thread for each data source type, data parallel processing is achieved, improving processing efficiency; format conversion and preliminary cleaning ensure data consistency and accuracy; S2. Divide the data stream into several windows based on event time for processing, store the data stream processing status in incremental form, and manage the data stream processing status by partitions; the specific steps of step S2 are as follows: S21. Determine the current time period; If the current time period is a peak period, proceed to step S22; If the current time period is a low period, proceed to step S23; If the current time period is a normal time period, proceed to step S24; S22. The preset window size is adjusted according to the preset increase, and the process proceeds to step S24; S23. The preset window size is adjusted according to the preset reduction amplitude, and the process proceeds to step S24; S24. Divide the data stream into windows according to the preset window size according to the trigger event time of each type of data source, and set the window watermark according to the maximum delay time of each type of data source; S25. Assign the corresponding processing thread in the connection pool to the data in each type of window in the data stream for processing, and wait for data events according to the window watermark; For example, user behavior logs are segmented according to event time, such as user click time, and the sliding window is set to 10 minutes, sliding once every 5 minutes; transaction records are segmented according to transaction time, and the rolling window is set to every hour; Dynamically adjust the window watermark threshold based on the user behavior log ID delay. For example, when it is found that most log delays do not exceed 30 seconds, set the window shade threshold to 30 seconds to reduce errors caused by out-of-order events. For example, the peak period is from 8pm to 10pm, the trough period is from 2am to 8am, and the rest of the period is normal period. During the peak period, the rate of user behavior logs increases significantly, so the window size is automatically increased to 20 minutes to relieve computing pressure; during the trough period, the window size is automatically reduced to 5 minutes to improve computing efficiency. S26. Record the difference in the number of events during the window sliding process of data stream processing, and store them in partitions according to the stream; For example, for the processing of user behavior logs, the system only saves the incremental changes in the number of user clicks in each window to reduce the consumption of state storage resources; and partitions the user behavior logs by user ID, and each partition manages its state data independently; for example, data with user IDs of 1-1000 is stored in partition 1, and data with user IDs of 1001-2000 is stored in partition 2 to ensure consistency in cross-partition transmission; It should be noted that adjusting the window size according to the time period can better adapt to the fluctuation of the data flow and improve processing efficiency. At the same time, window segmentation and watermark setting based on event time ensure the integrity and accuracy of the data. Partitioned storage improves the efficiency and fault tolerance of state management. S3. Detect task failures during data processing, automatically restart tasks and load balance, monitor data stream processing delays, and dynamically adjust computing resources when delays exceed the limit; the specific steps of step S3 are as follows: S31 monitors the execution tasks of the computing nodes where the processing threads of various types of data sources are located. When a computing node failure is detected, an available computing node is found for failure recovery; S32. Monitor the processing delay of the type data source and make dynamic resource adjustments when the processing delay exceeds the set delay threshold upper limit; It should be noted that the stability and real-time performance of the system are ensured through fault recovery and dynamic resource adjustment. When a fault occurs, the processing task can be quickly restored to avoid data loss or processing delay; at the same time, resources are dynamically adjusted according to processing delays to improve the overall performance of the system; S4. Real-time monitoring of data flow status and visualization, as well as alarms when data flow is abnormal; Step S4 is as follows: S41. Statistics on the processing status of each type of data source, the processing status includes delay, rate, and event data; S42. Generate a chart of the processing status of each type of data source in the data stream and display it in a graphical interface; S43. Generates alarm information for abnormal situations in the data flow processing process and notifies the operation and maintenance personnel in a preset manner; It should be noted that by real-time monitoring and visual display of data flow status, as well as generating alarm information to notify operation and maintenance personnel, comprehensive monitoring and management of the data processing process is achieved, improving the maintainability and reliability of the system.

[0027] In an embodiment of the present invention, based on step S31 and step S32, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0028] The specific steps of step S31 are as follows: S311. Record the event time of successful execution of processing threads of each type of data source; S312. Perform heartbeat monitoring on the computing nodes where the processing threads of various data sources are located; If the heartbeat monitoring is normal, proceed to step S313; If the heartbeat monitoring is abnormal, go to step S313; For example, when the heartbeat of a computing node that processes user behavior logs is found to have timed out for 30 seconds, it is determined to be a fault; S313. Analyze the abnormal logs of the computing nodes where the processing threads of various data sources are located to determine whether insufficient resources are identified; If yes, go to step S314; If not, proceed to step S32; S314. Determine the current computing node as a faulty computing node, and select one computing node from the computing nodes with a load rate higher than a threshold as a target computing node; Exemplarily, the failed task is automatically restarted and scheduled to a target computing node with a lighter load; for example, the task on the failed computing node is scheduled to a target computing node with an idle rate of 80%; S315. Migrate the processing thread of the failed computing node to the target computing node, and restore the data stream processing according to the last saved successful execution event time; It should be noted that through heartbeat monitoring and abnormal log analysis, faulty nodes and insufficient resources can be accurately identified; by migrating the processing tasks of the faulty nodes to the target nodes and resuming data flow processing, the continuity and reliability of the system are ensured; The specific steps of step S32 are as follows: S321. Count the event delays of various types of data sources in the statistical data flow and calculate the average delay time, where the event delay includes flow delay and processing delay; For example, statistics show that the average delay from reading user behavior logs from Kafka to processing is 50 milliseconds, and the average delay from reading transaction records from MySQL to processing is 100 milliseconds; S322. Monitor the real-time processing time of events of various types of data sources in the data stream to determine whether it is greater than the average delay time and the duration is greater than the set time threshold; If yes, go to step S323; If not, proceed to step S4; S323. Increase the number of computing nodes where the event corresponding processing thread is located; For example, when it is found that the processing delay of the user behavior log exceeds 100 milliseconds, the number of computing nodes processing the data is automatically increased to distribute the load; It should be noted that by monitoring the event delays of various types of data sources in the data stream and dynamically adjusting the number of computing nodes, the processing delay is effectively reduced and the real-time performance and throughput of the system are improved.

[0029] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0030] like Figure 2As shown, the following is an embodiment of a Flink-based streaming data processing system provided by an embodiment of the present disclosure. The system and the Flink-based streaming data processing methods of the above-mentioned embodiments belong to the same inventive concept. For details not fully described in the embodiment of the Flink-based streaming data processing system, reference can be made to the above-mentioned embodiment of the Flink-based streaming data processing method.

[0031] The system includes: The data source management module is used to receive data streams from different data sources, establish a connection pool of corresponding size for each data source according to the data source type, and decode, format and preliminarily clean the data in the data stream; The data window and processing state management module is used to divide the data stream into several windows for processing based on event time, store the data stream processing state in incremental form, and manage the data stream processing state by partition; Fault recovery and delay monitoring module, which is used to detect task failures during data processing, automatically restart tasks and balance loads, monitor data stream processing delays, and dynamically adjust computing resources when delays exceed the limit; The real-time monitoring module is used to monitor the data flow status in real time and display it visually, as well as to issue an alarm when the data flow is abnormal.

[0032] This embodiment achieves efficient, stable and reliable processing of massive real-time data streams by integrating functional modules of data source management, data window and processing status management, fault recovery and delay monitoring, and real-time monitoring.

[0033] The streaming data processing method based on Flink provided in the embodiment of the present application can be applied to electronic devices. It can be understood by those skilled in the art that the electronic device structure involved in the embodiment of the present invention does not constitute a limitation on the electronic device, and the electronic device may include more or less components than shown, or combine certain components, or arrange different components. In an embodiment of the present invention, the electronic device includes but is not limited to a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of the present application described and / or required herein.

[0034] The electronic device may include a processor, an external memory interface, an internal memory, a universal serial bus (USB) interface, a charging management module, a power management module, a battery, a wireless communication module, an audio module, a speaker, a microphone, a sensor module, buttons, a camera, a display, and a SIM card interface, etc.

[0035] It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0036] The processor may include one or more processing units, for example, the processor may include a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated into one or more processors.

[0037] The processor can be the nerve center and command center of the electronic device. The controller can generate an operation control signal according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0038] A memory may also be provided in the processor for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. The memory may store instructions or data that the processor has just used or is cyclically used. If the processor needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor, and thus improves system efficiency.

[0039] The above-mentioned electronic device implements the streaming data processing method based on Flink of the present application, receives data streams from different data sources, establishes a connection pool of corresponding size for each data source according to the data source type, and decodes, formats and preliminarily cleans the data in the data stream; divides the data stream into several windows for processing based on event time, stores the data stream processing status in an incremental form, and manages the data stream processing status by partitions; detects task failures in the data processing process, automatically restarts tasks and load balances, monitors data stream processing delays, and dynamically adjusts computing resources when the delay exceeds the standard; monitors the data stream status in real time and displays it visually, as well as issues an alarm when the data stream is abnormal, so as to achieve efficient and stable processing by receiving and preprocessing massive real-time data streams from different data sources; and uses fault detection, load balancing and other mechanisms to ensure system stability and real-time performance; at the same time, real-time monitoring and visual display improve the maintainability and reliability of the system.

[0040] The storage medium provided in the present application stores a program product capable of implementing a streaming data processing method based on Flink.

[0041] Flink-based streaming data processing methods include: Receive data streams from different data sources, establish a connection pool of corresponding size for each data source according to the data source type, and decode, format and preliminarily clean the data in the data stream; Divide the data stream into several windows for processing based on event time, store the data stream processing status in incremental form, and manage the data stream processing status in partitions; Detect task failures during data processing, automatically restart tasks and balance loads, monitor data stream processing delays, and dynamically adjust computing resources when delays exceed the limit; Monitor data flow status in real time and display it visually, and issue alarms when data flow is abnormal.

[0042] In some possible implementations, the Flink-based streaming data processing method of the present disclosure may be implemented in the form of a program product, which includes a program code. When the program product runs on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary implementations of the present disclosure described in the above “Exemplary Method” section of this specification.

[0043] The storage medium of the present disclosure can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0044] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A streaming data processing method based on Flink, characterized in that: The steps include: S1. Receive data streams from different data sources, establish a connection pool of corresponding size for each data source according to the data source type, and decode, format and preliminarily clean the data in the data stream; S2. Divide the data stream into several windows for processing based on event time, store the data stream processing status in incremental form, and manage the data stream processing status by partition; S3. Detect task failures during data processing, automatically restart tasks and load balance, monitor data stream processing delays, and dynamically adjust computing resources when delays exceed the limit; S4. Monitor the data flow status in real time and display it visually, and issue an alarm when the data flow is abnormal.

2. The streaming data processing method based on Flink according to claim 1, characterized in that: The specific steps of step S1 are as follows: S11. Determine the data source type of the data processing platform, and create a connection pool for each type of data source according to the processing requirements, and create the required number of processing threads; S12. Identify the data format of each type of data source, and select an appropriate encoder for each type of data to convert the format, converting the data source to a unified standard data format of the data processing platform; S13. Clean the data stream that has completed the format conversion, identify and filter out null values, abnormal values, and data that does not conform to the format according to the data source type.

3. The streaming data processing method based on Flink according to claim 2 is characterized in that: The specific steps of step S2 are as follows: S21. Determine the current time period; If the current time period is a peak period, proceed to step S22; If the current time period is a low period, proceed to step S23; If the current time period is a normal time period, proceed to step S24; S22. The preset window size is adjusted according to the preset increase, and the process proceeds to step S24; S23. The preset window size is adjusted according to the preset reduction amplitude, and the process proceeds to step S24; S24. Divide the data stream into windows according to the preset window size according to the trigger event time of each type of data source, and set the window watermark according to the maximum delay time of each type of data source; S25. Assign the corresponding processing thread in the connection pool to the data in each type of window in the data stream for processing, and wait for data events according to the window watermark; S26. Record the difference in the number of events during the window sliding process of data stream processing, and store them in partitions by stream.

4. The streaming data processing method based on Flink according to claim 3 is characterized in that: The specific steps of step S3 are as follows: S31 monitors the execution tasks of the computing nodes where the processing threads of various types of data sources are located. When a computing node failure is detected, an available computing node is found for failure recovery; S32. Monitor the processing delay of the type data source, and make dynamic resource adjustments when the processing delay exceeds the set delay threshold upper limit.

5. The streaming data processing method based on Flink according to claim 4 is characterized in that: The specific steps of step S31 are as follows: S311. Record the event time of successful execution of processing threads of each type of data source; S312. Perform heartbeat monitoring on the computing nodes where the processing threads of various data sources are located; If the heartbeat monitoring is normal, proceed to step S313; If the heartbeat monitoring is abnormal, go to step S313; S313. Analyze the abnormal logs of the computing nodes where the processing threads of various data sources are located to determine whether insufficient resources are identified; If yes, go to step S314; If not, proceed to step S32; S314. Determine the current computing node as a faulty computing node, and select one computing node from the computing nodes with a load rate higher than a threshold as a target computing node; S315. Migrate the processing thread of the failed computing node to the target computing node, and restore the data stream processing according to the last saved successful execution event time.

6. The streaming data processing method based on Flink according to claim 5, characterized in that: The specific steps of step S32 are as follows: S321. Count the event delays of various types of data sources in the statistical data flow and calculate the average delay time, where the event delay includes flow delay and processing delay; S322. Monitor the real-time processing time of events of various types of data sources in the data stream to determine whether it is greater than the average delay time and the duration is greater than the set time threshold; If yes, go to step S323; If not, proceed to step S4; S323. Increase the number of computing nodes where the event corresponding processing thread is located.

7. The streaming data processing method based on Flink according to claim 4 is characterized in that: Step S4 is specifically as follows: S41. Statistics on the processing status of each type of data source, the processing status includes delay, rate, and event data; S42. Generate a chart of the processing status of each type of data source in the data stream and display it in a graphical interface; S43. Generate alarm information for abnormal situations in the data flow processing process and notify the operation and maintenance personnel in a preset manner.

8. A streaming data processing system based on Flink, characterized in that: include: The data source management module is used to receive data streams from different data sources, establish a connection pool of corresponding size for each data source according to the data source type, and decode, format and preliminarily clean the data in the data stream; The data window and processing state management module is used to divide the data stream into several windows for processing based on event time, store the data stream processing state in incremental form, and manage the data stream processing state by partition; Fault recovery and delay monitoring module, which is used to detect task failures during data processing, automatically restart tasks and balance loads, monitor data stream processing delays, and dynamically adjust computing resources when delays exceed the limit; The real-time monitoring module is used to monitor the data flow status in real time and display it visually, as well as to issue an alarm when the data flow is abnormal.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the Flink-based streaming data processing method are implemented as described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the Flink-based streaming data processing method are implemented as described in any one of claims 1 to 7.

Images