Streaming data processing method and device, equipment and medium

A technology of streaming data and processing methods, which is applied in the direction of electrical digital data processing, special data processing applications, digital data information retrieval, etc., can solve the problems of large memory resource consumption and affecting system processing performance, etc., to ensure rationality and accuracy performance, improve system performance, and reduce memory consumption

Active Publication Date: 2020-04-21
BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF25 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

When the amount of data is very large, this solution consumes a lot of memory resources and affects the overall processing performance of the system

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Streaming data processing method and device, equipment and medium
  • Streaming data processing method and device, equipment and medium
  • Streaming data processing method and device, equipment and medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0077] Exemplary embodiments of the present application are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present application to facilitate understanding, and they should be regarded as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

[0078] In the embodiment of the present application, the streaming data processing process may include two parts: a data preprocessing part and a data reading and writing part. The data preprocessing part is responsible for receiving the data stream in real time, and by dynamically maintaining the preset number of slots, it is determined in real time whether the new ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention discloses a streaming data processing method and device, equipment and a medium, and relates to the technical field of big data processing. The method comprises the steps that whether key fields, received in real time, in new data exist in a preset number of slots or not is determined, and the value of the preset number is larger than the set value of the required data size; if the new data does not exist and vacancies do not exist in the preset number of slots, replacing the last fields in the slots with the key fields, and determining the statistical attributes of the key fields in the slots according to the information currently carried by the new data and the statistical attributes of the last fields; and determining whether the new data belongs to demand data or not in real time according to the statistical attributes of the key fields. According to the embodiment of the invention, by dynamically maintaining the preset number of slot data, the memory consumption can be reduced under the condition of ensuring the top-k problem processing accuracy.

Description

technical field [0001] The embodiments of the present application relate to computer technology, in particular to big data processing, and in particular to a streaming data processing method, device, device and medium. Background technique [0002] In many statistical analysis systems or advertising systems, data streams are calculated in real time to solve the top-k problem for a certain data dimension. [0003] For this top-k problem, the current common solutions mainly include the following two types: [0004] (1) Directly through the first-in-first-out method, first-come-first-served, after the k slots are full, the data received later is directly discarded. This solution is only applicable to the scene where the key field key in the real-time data received earlier belongs to the key field key that frequently appears in the later stage, that is, this solution is applicable to a narrow range of scenarios, which can easily lead to the processing accuracy of the top-k prob...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/2455G06Q30/02
CPCG06F16/24568G06Q30/0277Y02D10/00
Inventor 陈鑫林江红高春旭叶峻
Owner BEIJING BAIDU NETCOM SCI & TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap