Index statistics method, system and device and storage medium

A statistical method and statistical system technology, applied in computing, special data processing applications, instruments, etc., can solve the problems of not supporting time window update one by one, large amount of calculation, and high delay
CN110471944APending Publication Date: 2019-11-19BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD +1
0 Cites 8 Cited by

Patent Information

Authority / Receiving Office
CN · China
Current Assignee / Owner
BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD
Publication Date
2019-11-19

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The invention discloses an index statistics method, system and device for a sliding window based on a big data processing framework and a storage medium. The index statistics method comprises the following steps: storing a plurality of parameters by using a state object in a mapWithstate operator, wherein the parameters comprise a time linked list, a minimum value of the time linked list, a maximum value of the time linked list and an index statistics value; enabling a big data processing framework to receive data, wherein each piece of data comprises to-be-counted index fields and corresponding timestamps, and a time chain table is used for storing the timestamps of the data according to a time sequence; taking the received latest data as the current data; and updating the time linked list, the minimum value, the maximum value and the index statistical value according to the timestamp of the current data, the to-be-counted index field of the current data and a preset time window value, wherein the index statistical value is obtained through multiplexing calculation. The calculated statistical value is reused, so that the calculation amount is reduced, the calculation speed is increased, the delay is reduced, and the statistical requirement of the service can be responded more quickly.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The present invention relates to the field of index statistics of big data, in particular to an index statistics method, system, device and storage medium based on a sliding window of a big data processing framework. Background technique

[0002] With the rapid development of big data processing technology, the business has higher and higher requirements on the timeliness of data. The traditional offline index calculation can no longer meet the business requirements. The rapid growth of demand has promoted the update and iteration of technology, and various big data can be processed in real time. Frameworks continue to emerge, and Flink (a big data processing framework) has received widespread attention for its ultra-short processing latency and high throughput, and has been applied in production environments.

[0003] In the field of indicator statistics, statistical indicators based on time windows are the most common form. Among various types of tim...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More