High-throughput stream processing method and device
A high-throughput, stream processing technology, applied in the field of high-throughput stream processing methods and devices, can solve problems such as waste of hardware resources and low data efficiency, and achieve the effects of improving throughput and system stability
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Method used
Image
Examples
Embodiment 1
[0029] In order to improve throughput, improve computing resource utilization, and reduce operation and maintenance costs, this embodiment provides a high-throughput stream processing method. Please refer to figure 1 , the method includes the following steps:
[0030] S101, the data receiving thread judges whether the remaining buffer space of the current data memory cache is greater than a preset threshold, if so, caches the received data into the memory cache, otherwise, saves the received data to a local hard disk;
[0031] S102. The data processing thread acquires data from the memory cache, and performs calculation processing on the acquired data.
[0032] In this embodiment, the high-throughput stream processing method further includes: constructing a bolt model, the bolt model including: a data receiving bolt model, a data processing bolt model, and a data sending bolt model; the data receiving bolt model is used to receive The data message sent by the data sending bol...
Embodiment 2
[0046] In order to improve throughput, improve computing resource utilization, and reduce operation and maintenance costs, this embodiment provides a high-throughput flow processing method.
[0047] This embodiment describes the high-throughput stream processing method with a specific example. The current mr measurement report of a certain telecom operator city A requires that the data be processed in real time.
[0048] Such as Figure 4 As shown, this embodiment provides a flow chart of a high-throughput flow processing method, including the following steps:
[0049] S401. Open up a memory space inside the Worker as a unified memory cache space;
[0050] Create a memory space inside the Worker as a unified memory cache space for caching the data received by the bolt.
[0051] S402. Initialize the size of the unified memory cache space;
[0052] The size of the unified memory cache space is automatically initialized according to a certain ratio according to the memory size...
Embodiment 3
[0066] In order to improve throughput, improve computing resource utilization, and reduce operation and maintenance costs, this embodiment provides a high-throughput stream processing method.
[0067] This embodiment provides a high-throughput stream processing device, such as Figure 5 As mentioned above, it is a block diagram of a high-throughput stream processing device provided in this embodiment, including a data receiving module and a data processing module;
[0068] The data receiving module is used to receive data and determine whether the remaining buffer space of the current data memory cache is greater than a preset threshold, if so, cache the received data into the memory cache, otherwise, save the received data to the local hard drive;
[0069] The data processing module is configured to obtain data from the memory cache, and perform calculation processing on the obtained data.
[0070] The high-throughput stream processing device provided in this embodiment fur...
PUM
Login to View More Abstract
Description
Claims
Application Information
Login to View More