Streaming data processing method and device, electronic device and storage medium
A technology of streaming data and processing methods, applied in the field of data processing, can solve problems such as the decline of the overall processing capacity
Pending Publication Date: 2020-01-21
BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF0 Cites 7 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0005] This application provides a streaming data processing method, device, electronic equipment, and storage medium to solve the problem of overall processing capacity decline caused by a short-term abnormality of a certain level of operator instances
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0076] Exemplary embodiments of the present application are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present application to facilitate understanding, and they should be regarded as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
[0077] In existing streaming computing, when the rate at which the system receives data is higher than the rate at which it processes data within a short period of time, the system usually performs backpressure processing. Among them, there may be the following two reasons for the back pressure: 1. The sudden increase in traffic exceeds the system processing capacity...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a streaming data processing method and device, an electronic device and a storage medium, and relates to the technical field of data processing. According to the specific implementation scheme, the method comprises: firstly obtaining an operation state of an instance corresponding to each data fragment in a first data storage container; wherein the first data storage container comprises a plurality of data fragments; storing the cache data of the data issuing component in the first data storage container according to the granularity of the data fragments, then determining the overflow data fragments according to the running state, the overflow data fragments being the data fragments corresponding to the instances with the abnormal running state, and finally cachingthe data in the overflow data fragments in the storage medium. According to the streaming data processing method provided by the invention, the data fragments corresponding to the instances of which the running states are the abnormal states are cached into the storage medium, so that the abnormal data fragments and the normal data fragments are separately processed, and the situation that the overall processing capacity is reduced due to short-term abnormality of a certain level of operator instances is avoided.
Description
technical field [0001] The present application relates to the technical field of data processing, and in particular to a streaming data processing method, device, electronic equipment, and storage medium. Background technique [0002] With the development of server data processing capabilities, streaming computing is widely used in large-scale distributed computing scenarios such as information flow, search and database building, and retrieval and billing. [0003] Among them, in stream computing, when the rate at which the system receives data is higher than the rate at which it processes data within a short period of time, the system usually performs backpressure processing. The reasons for the back pressure may be as follows: 1. The sudden increase in traffic exceeds the system processing capacity; 2. The short-term abnormality of a certain level of operator instances reduces the processing capacity. At present, the prior art does not distinguish between the above two si...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/2455G06F16/22G06F12/02
CPCG06F12/0246G06F16/2291G06F16/24568
Inventor 谢维柱邢越
Owner BEIJING BAIDU NETCOM SCI & TECH CO LTD