Supercharge Your Innovation With Domain-Expert AI Agents!

Stream data processing method and apparatus

A technology of streaming data and processing methods, applied in the network field, can solve the problems that the thread pool cannot improve processing efficiency, low processing efficiency, out of order, etc., and achieve the effect of solving low efficiency, solving out-of-order problems, and improving processing efficiency

Active Publication Date: 2016-07-27
DATANG SOFTWARE TECH
View PDF3 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] However, the above-mentioned traditional network management system uses a serial operation method inside the process, which will lead to low processing efficiency; for example, the execution time of process A and process B is 1 hour, when a piece of data , first send the piece of data to process A, and after process A completes the processing of the piece of data, then send the piece of data to process B, and then the processing of the next piece of data is still the same, process A or process B in Only one is performing calculations at the same time. If there are 100 pieces of data waiting to be processed, it will take 200 hours in total.
[0005] In addition, although some processing processes use the thread pool for calculation, the thread pool cannot improve the processing efficiency due to the contextuality of the alarm stream data; for example, there are alarm data D, E, and F in sequence, where D and F has a business relationship, so its relationship with D must be analyzed when processing F
Therefore, the time sequence must be strictly followed when processing alarm streaming data. The random concurrent processing of the thread pool will cause out-of-sequence problems, and increasing the sequence on the thread pool is no different from serial
Therefore, the thread pool is only suitable for unrelated data

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Stream data processing method and apparatus
  • Stream data processing method and apparatus
  • Stream data processing method and apparatus

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0041] refer to figure 1 , which shows a flow chart of the steps of a streaming data processing method according to the present invention, which may specifically include the following steps:

[0042] Step 101, copy and receive streaming data for the current process; wherein, the process copy is an operation entity with an independent operation thread;

[0043] The existing streaming data processing process adopts the method of serial operation, that is, after the process A completes the data processing, it is then handed over to the process B for processing, resulting in low processing efficiency. The present invention proposes to set several independent processes for the process copy, and achieve parallel execution between processes through decoupling between process copies.

[0044] For example, for process A, independent process copies (that is, computing entities with independent computing threads) A1, A2,..., An can be set according to data sources 1, 2, 3...n. A1 only p...

Embodiment 2

[0066] The streaming data processing method in this embodiment may further include the following optional technical solutions on the basis of the first embodiment above.

[0067] see figure 2 , showing a flow chart of steps of processing the streaming data by using the thread copied by the current process in the present invention, which may specifically include:

[0068] Step 201, using the current process copy to identify the source of the streaming data;

[0069] In an application example of the present invention, an adapter can be used to identify the source of the streaming data, and in addition, the adapter can also be used to complete the input and output work of the thread to the streaming data. Specifically, the input adapter can be used to receive streaming data, identify the source of the streaming data, and mark the identified source information on the data as an additional attribute. Moreover, after the thread finishes processing a piece of data, it can also use...

example

[0140] The example is as follows: Called in the main method.

[0141]

[0142]

[0143] The embodiment of the present invention provides various operation modes and parameter settings for the interface of the process to meet the needs of each process, such as high-speed file cache, congestion strategy and queuing strategy. The framework is responsible for the management of the entire thread life cycle, taking over input and output, and each process only needs to focus on its own logic.

[0144] In addition to data input, each process also needs other auxiliary tools, such as high-speed file cache as a tool, which is independent of the core framework. By providing the basic framework of file cache, simple development can be achieved to complete a high-speed file that is synchronized with the main library. A local cache is available for querying.

[0145] By setting the blocking strategy, it is possible to block the thread to save calculations when waiting, ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

Embodiments of the invention provide a stream data processing method and apparatus. The method specifically comprises the steps of receiving stream data for a current process copy, wherein the process copy is a calculation entity with independent calculation threads; processing the stream data by utilizing the threads of the current process copy, wherein the threads are in one-to-one correspondence with sources of the stream data; and transmitting the processed stream data to a next process copy by utilizing the threads of the current process copy, and processing the processed stream data by the next process copy until the stream data is processed by a final process copy. According to the embodiments of the invention, the stream data can be concurrently processed, so that the processing efficiency is improved.

Description

technical field [0001] The invention relates to the field of network technology, in particular to a method and device for processing streaming data. Background technique [0002] In recent years, with the rapid development of computer technology and Internet technology, network services and applications are gradually enriched, and the network has become an indispensable part of people's daily life. At the same time, with the continuous expansion of the network scale, more and more devices are managed by the network management system, and the amount of data to be processed is also increasing. higher requirement. [0003] The traditional network management system is divided into 2 to 3 processes according to the functional classification, and the processes communicate through the CORBA (Common Object Request Broker Architecture, Common Object Request Broker Architecture) middleware, and cooperate to complete the processing of streaming data. In addition, it is also possible ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F17/30
Inventor 商越李峰
Owner DATANG SOFTWARE TECH
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com