Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Data stream processing and classifying method, device and system

A technology for data processing and data flow, applied in the field of network systems, can solve the problems of inability to dynamically adjust data flow, inability to divide data flow statistics, and affect network system performance, so as to reduce data statistics, improve system performance, and reduce system performance. The effect of resource consumption

Pending Publication Date: 2021-07-16
ALIBABA GRP HLDG LTD
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, in this way, a large number of division calculations are required, and the data flow cannot be counted directly, and the statistics cannot be divided into data flows, and additional timer support is required.
If statistics need to be divided into data streams, each data stream needs to be counted separately. The data statistics are huge, and the statistical data needs to be compared horizontally to determine hot data streams. This will occupy a lot of computing resources and will Seriously affect the performance of the network system
[0006] Moreover, in the prior art, it is impossible to dynamically adjust the number of data streams allocated to the hardware according to the actual situation. Therefore, how to quickly and accurately identify hot data streams and allocate them to the hardware reasonably while occupying less data resources It has become a technical problem to be solved urgently

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data stream processing and classifying method, device and system
  • Data stream processing and classifying method, device and system
  • Data stream processing and classifying method, device and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0084] Embodiment 1 of the present invention provides a method for classifying data streams, the flow of which is as follows figure 1 shown, including the following steps:

[0085] S11. Periodically count the duration of a group of packets of a specified number in the forwarded data flow.

[0086] A data flow refers to a collection of all packets having the same quintuple, wherein the quintuple refers to: protocol number, source IP, destination IP, source port, and destination port. That is, a data flow refers to a set of all packets transmitted in the network that originate from the same IP address and port, have the same destination IP address and port, and use the same communication protocol. For example, all messages in the entire life cycle of a communication created between an APP and a remote server from the beginning to the end of the connection are a data stream.

[0087] Since a data flow has a large amount of data and lasts for a long time, in order to improve eff...

Embodiment 2

[0100] Embodiment 2 of the present invention provides a data stream processing method, the flow of which is as follows image 3 shown, including the following steps:

[0101] S21. Periodically count the duration of a group of packets of a specified number in the forwarded data flow.

[0102] Refer to the relevant description of step S11 in the first embodiment above, and details are not repeated here.

[0103] S22. Determine a first data stream meeting a set condition according to the duration of each data stream and the data processing capability of the hardware.

[0104] Refer to the relevant description of step S12 in the first embodiment above, and details are not repeated here.

[0105] S23. Allocate the first data stream to hardware for forwarding processing.

[0106] After the first data flow and the second data flow are determined in step S22, the allocation operation of the data flow can be performed according to whether the data flow is originally processed in sof...

Embodiment 3

[0111] Embodiment 3 of the present invention provides a specific implementation of the above-mentioned data flow classification method, which divides the first data flow and the second data flow based on the forwarding duration of a specified number of packets in the data flow, such as hotspot data flow and non-hot data flow, its flow is as follows Figure 4 As shown, the method includes the following steps:

[0112] S31. Periodically count the duration of a group of packets of a specified number in the forwarded data flow.

[0113] For specific steps, refer to the description of step S11, which will not be repeated here.

[0114] S32. Determine the processing ratio of the hardware according to the data processing capability of the hardware and in combination with the number of data streams to be processed.

[0115] In this step, the proportion that the hardware can process can be determined according to the data processing capability of the hardware combined with the number...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data stream processing and classifying method, device and system, and the processing method comprises the steps: carrying out the periodical statistics of the duration of a set of messages with a specified number in a forwarding data stream, determining a first data stream according with a set condition according to the duration of each data stream and the data processing capability of hardware, and distributing the first data stream to hardware for forwarding processing. The statistical result of the whole data flow is replaced by the statistics of part of the messages, so that the data statistics amount is reduced, the system resource consumption is greatly reduced, the memory usage amount is reduced, and the data flow meeting the condition, such as the hotspot data flow, can be quickly and accurately identified.

Description

technical field [0001] The present invention relates to the technical field of network systems, in particular to a data flow processing and classification method, device and system. Background technique [0002] With the rapid development of cloud computing and the Internet, there are more and more application scenarios for big data processing. At present, in the fields of finance, mobile data, network monitoring, and the Internet of Things, it is often necessary to process some massive, real-time, and fast-response data streams. Therefore, higher requirements are placed on the processing capability and processing speed of the network relay equipment. [0003] Both system software and hardware can forward and process all data traffic from the sender, but due to limited hardware resources, it is not enough to support the processing of large amounts of data. The data flow entries supported by the system software can be very large, but its processing speed It is not as good a...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): H04L12/851H04L12/859H04L47/2475
CPCH04L47/2441H04L47/2483H04L47/2475
Inventor 邹勇
Owner ALIBABA GRP HLDG LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com