Optimization method and system for improving streaming data processing efficiency

Abstract

The invention discloses an optimization method and system for improving streaming data processing efficiency, and the method comprises the following steps: S1, providing a tool sdk packet in a streaming channel, monitoring the streaming data transmitted in the streaming channel through the tool sdk packet, and judging whether the streaming data exists in the streaming channel for transmission or not in a period; s2, if the condition that no streaming data exists in one period is not met, processing is not carried out; s3, if the condition that no streaming data exists in one period is met, it is determined that the period is a period needing special processing, and a special processing mark is newly added in the memory; and the tool sdk packet is triggered to send a heartbeat packet into the stream channel, so that the stream channel can normally process the stream data. The flow channel state can be prevented from being converted from a normal state to a blocked state, so that the processing delay of partial data is reduced, and the response time is prolonged.

Description

Technical Field

[0001] The present invention relates to the field of computer communication technology, and in particular to an optimization method and system for improving stream data processing efficiency. Background Art

[0002] Stream processing is an efficient way to process data streams, which has many significant benefits. Stream processing allows data to be processed immediately when it is generated, rather than waiting for the accumulation of batch data. This enables real-time analysis and response to data, and is suitable for application scenarios that require fast response, such as financial transaction monitoring, social media analysis, etc. At the same time, because stream processing can process data instantly, it usually has lower latency. This enables applications to make decisions faster and improve user experience.

[0003] Because streaming processing uses streaming channels to transmit streaming data, when there is data transmission in the streaming channel, streaming data processing can be performed quickly, and there is usually a steady stream of data transmission. But what if there is no data transmission for a period of time? At this time, the entire streaming channel will be in a blocked state. When data transmission occurs occasionally, the streaming channel will change from a blocked state to a normal state. At this time, there will be a large delay in data processing. Therefore, due to the uncertainty of streaming data, the delay fluctuation of streaming data processing is large. At present, most streaming systems have no suitable method to deal with it. Summary of the invention

[0004] To overcome the above problems, the purpose of the present invention is to provide an optimization method for improving the efficiency of stream data processing, which can prevent the stream channel state from changing from a normal state to a blocked state, thereby reducing the processing delay of some data and improving the response time.

[0005] The present invention is implemented by the following scheme: an optimization method for improving the efficiency of stream data processing, the optimization method comprising the following steps:

[0006] Step S1: Provide a tool SDK package in the streaming channel, the tool SDK package monitors the streaming data transmitted in the streaming channel, and determines whether there is streaming data for transmission in the streaming channel within a cycle;

[0007] Step S2: If the condition that there is no flow data within a cycle is not met, no processing is performed;

[0008] Step S3: If the condition that there is no flow data within a cycle is met, the cycle is deemed to require special processing, and a special processing flag is added to the memory; and the tool SDK package is triggered to send a heartbeat packet to the flow channel to keep the flow channel able to process flow data normally.

[0009] Furthermore, the adding of a special processing flag in the memory is further as follows: the tool SDK package sets a flag spec_flag to 1 in the internal cache, indicating that there is no flow data in the flow channel in this cycle.

[0010] Furthermore, the triggering of sending a heartbeat packet to the flow channel by the tool SDK package is further as follows: an asynchronous thread is automatically started inside the SDK, and the thread automatically monitors the spec_flag flag. When the flag is 1, a heartbeat packet is simulated to be sent to the flow channel.

[0011] The present invention also provides an optimization system for improving the efficiency of stream data processing, the optimization system comprising a monitoring module, a normal processing module, and an optimization processing module;

[0012] The monitoring module: provides a tool SDK package in the streaming channel, and the tool SDK package monitors the streaming data transmitted in the streaming channel, and determines whether there is streaming data for transmission in the streaming channel within a cycle;

[0013] The normal processing module: if the condition that there is no flow data within a cycle is not met, no processing is performed;

[0014] The optimization processing module: if the condition that there is no flow data within a cycle is met, the cycle is deemed to be a cycle requiring special processing, and a special processing flag is added to the memory; and the tool SDK package is triggered to send a heartbeat packet to the flow channel to keep the flow channel able to process flow data normally.

[0015] Furthermore, the adding of a special processing flag in the memory is further as follows: the tool SDK package sets a flag spec_flag to 1 in the internal cache, indicating that there is no flow data in the flow channel in this cycle.

[0016] Furthermore, the triggering of sending a heartbeat packet to the flow channel by the tool SDK package is further as follows: an asynchronous thread is automatically started inside the SDK, and the thread automatically monitors the spec_flag flag. When the flag is 1, a heartbeat packet is simulated to be sent to the flow channel.

[0017] The beneficial effects of the present invention are as follows: by adding a heartbeat function to the streaming channel and performing periodic statistical operations on the data in the streaming channel, when there is occasional lack of streaming data transmission in the future, the heartbeat function can be used to maintain the flow of the streaming channel and prevent the streaming channel state from changing from a normal state to a blocked state, thereby reducing the processing delay of some data and improving the response time. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic flow chart of the method of the present invention.

[0019] Figure 2 It is a system principle block diagram of the present invention. DETAILED DESCRIPTION

[0020] The present invention will be further described below in conjunction with the accompanying drawings.

[0021] See also Figure 1 As shown, an optimization method for improving stream data processing efficiency of the present invention comprises the following steps:

[0022] Step S1: Provide a tool SDK package in the streaming channel, the tool SDK package monitors the streaming data transmitted in the streaming channel, and determines whether there is streaming data for transmission in the streaming channel within a cycle;

[0023] Step S2: If the condition that there is no flow data within a cycle is not met, no processing is performed;

[0024] Step S3: If the condition that there is no flow data within a cycle is met, the cycle is deemed to require special processing, and a special processing flag is added to the memory; and the tool SDK package is triggered to send a heartbeat packet to the flow channel to keep the flow channel able to process flow data normally.

[0025] The present invention will be further described below in conjunction with a specific embodiment:

[0026] An optimization method to improve the efficiency of stream data processing,

[0027] 1. Provide a tool SDK package in the streaming channel, which monitors the streaming data transmitted in the streaming channel and determines whether there is streaming data for transmission in the streaming channel within a cycle;

[0028] For example: 1. Assume that there is a Java project that consumes data through streaming channels. External data sources write data into the message middleware [such as Kafka]. This project reads messages from Kafka and sends streaming data to other data sources through streaming channels. An SDK package is provided. The SDK package is developed in Java and can be integrated into the project through Maven tools.

[0029] 2. If the condition that there is no flow data within a cycle is not met, no processing will be performed;

[0030] 3. If the condition of no flow data within a cycle is met, the cycle is considered to be a cycle that requires special processing, and a special processing flag is added to the memory; and the tool SDK package is triggered to send a heartbeat packet to the flow channel to keep the flow channel able to process the flow data normally. Special processing is to use the heartbeat function package to maintain the flow of the flow channel, prevent the flow channel state from changing from normal to blocked, and thus reduce the processing delay of some data.

[0031] The special processing flag added in the memory described in 3.1 is further as follows: the tool SDK package sets a flag spec_flag to 1 in the internal cache, indicating that there is no flow data in the flow channel during this cycle.

[0032] 3.2 The triggering of sending a heartbeat packet to the flow channel by the tool SDK package is further as follows: an asynchronous thread is automatically started inside the SDK, and the thread automatically monitors the spec_flag flag. When the flag is 1, a heartbeat packet is simulated to be sent to the flow channel.

[0033] For example: 3. After the project is started, the SDK will monitor the data in the stream channel. If there is data flow, it will not be processed. Within a cycle, for example, there is no stream data within 5 seconds, the SDK will set a flag in the internal cache, spec_flag is 1, indicating that there is no stream data in the stream channel within this cycle, and the SDK will automatically start an asynchronous thread inside. The thread will automatically monitor the spec_flag flag in step 3. When the flag is 1, it means that there is no data in the stream channel at this time. It will simulate sending a heartbeat packet to the stream channel to keep the stream channel able to process the stream data normally without causing blocking. Usually, message producers send messages to the channel, and message consumers read messages from the channel for consumption. When the consumer of the stream channel consumes the heartbeat packet data in step 4, it only needs to receive the heartbeat packet without any business processing.

[0034] See also Figure 2 As shown, the present invention also provides an optimization system for improving the efficiency of stream data processing, the optimization system includes a monitoring module, a normal processing module, and an optimization processing module;

[0035] The monitoring module: provides a tool SDK package in the streaming channel, and the tool SDK package monitors the streaming data transmitted in the streaming channel, and determines whether there is streaming data for transmission in the streaming channel within a cycle;

[0036] For example: 1. Assume that there is a Java project that consumes data through streaming channels. External data sources write data into the message middleware [such as Kafka]. This project reads messages from Kafka and sends streaming data to other data sources through streaming channels. An SDK package is provided. The SDK package is developed in Java and can be integrated into the project through Maven tools.

[0037] The normal processing module: if the condition that there is no flow data within a cycle is not met, no processing is performed;

[0038] The optimization processing module: if the condition that there is no flow data within a cycle is met, the cycle is considered to be a cycle that requires special processing, and a special processing flag is added to the memory; and the tool SDK package is triggered to send a heartbeat packet to the flow channel to keep the flow channel able to process the flow data normally. Special processing is to use the heartbeat function package to maintain the flow of the flow channel, prevent the flow channel state from changing from normal to blocked state, and thus reduce the processing delay of some data.

[0039] The adding of a special processing flag in the memory is further as follows: the tool SDK package sets a flag spec_flag to 1 in the internal cache, indicating that there is no flow data in the flow channel in this cycle.

[0040] The triggering tool SDK package to send a heartbeat packet to the flow channel is further as follows: an asynchronous thread is automatically started inside the SDK, and the thread automatically monitors the spec_flag flag. When the flag is 1, a heartbeat packet is simulated to be sent to the flow channel.

[0041] 3. After the project is started, the SDK will monitor the data in the stream channel. If there is data flow, it will not be processed. Within a cycle, for example, if there is no stream data within 5 seconds, the SDK will set a flag in the internal cache, spec_flag is 1, indicating that there is no stream data in the stream channel within this cycle. The SDK will automatically start an asynchronous thread inside, which will automatically monitor the spec_flag flag in step 3. When the flag is 1, it means that there is no data in the stream channel at this time. It will simulate sending a heartbeat packet to the stream channel to keep the stream channel able to process the stream data normally without causing blocking. Usually, message producers send messages to the channel, and message consumers read messages from the channel for consumption. When the consumer of the stream channel consumes the heartbeat packet data in step 4, it only needs to receive the heartbeat packet without any business processing.

[0042] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.

Claims

1. An optimization method for improving stream data processing efficiency, characterized in that: The optimization method comprises the following steps: Step S1: Provide a tool SDK package in the streaming channel, the tool SDK package monitors the streaming data transmitted in the streaming channel, and determines whether there is streaming data for transmission in the streaming channel within a cycle; Step S2: If the condition that there is no flow data within one cycle is not met, no processing is performed; Step S3: If the condition that there is no flow data within a cycle is met, the cycle is identified as a cycle that requires special processing, and a special processing flag is added to the memory; And trigger the tool SDK package to send a heartbeat packet to the stream channel to keep the stream channel able to process stream data normally.

2. The optimization method for improving stream data processing efficiency according to claim 1, characterized in that: The adding of a special processing flag in the memory is further as follows: the tool SDK package sets a flag spec_flag to 1 in the internal cache, indicating that there is no flow data in the flow channel in this cycle.

3. The optimization method for improving stream data processing efficiency according to claim 2, characterized in that: The triggering tool SDK package to send a heartbeat packet to the flow channel is further as follows: an asynchronous thread is automatically started inside the SDK, and the thread automatically monitors the spec_flag flag. When the flag is 1, a heartbeat packet is simulated to be sent to the flow channel.

4. An optimization system for improving stream data processing efficiency, characterized in that: The optimization system includes a monitoring module, a normal processing module, and an optimization processing module; The monitoring module: provides a tool SDK package in the streaming channel, and the tool SDK package monitors the streaming data transmitted in the streaming channel to determine whether there is streaming data for transmission in the streaming channel within a cycle; The normal processing module: if the condition that there is no flow data within a cycle is not met, no processing is performed; The optimization processing module: if the condition that there is no flow data within a cycle is met, the cycle is determined to be a cycle requiring special processing, and a special processing flag is added to the memory; And trigger the tool SDK package to send a heartbeat packet to the stream channel to keep the stream channel able to process stream data normally.

5. The optimization system for improving stream data processing efficiency according to claim 4, characterized in that: The adding of a special processing flag in the memory is further as follows: the tool SDK package sets a flag spec_flag to 1 in the internal cache, indicating that there is no flow data in the flow channel in this cycle.

6. The optimization system for improving stream data processing efficiency according to claim 5, characterized in that: The triggering tool SDK package to send a heartbeat packet to the flow channel is further as follows: an asynchronous thread is automatically started inside the SDK, and the thread automatically monitors the spec_flag flag. When the flag is 1, a heartbeat packet is simulated to be sent to the flow channel.

Images