Mass data real-time monitoring method, system and device and storage medium

A real-time monitoring, massive data technology, applied in the field of big data processing, can solve problems such as waste of human resources, achieve the effect of avoiding waste and quickly adapting to changes in business needs

Pending Publication Date: 2020-07-21
BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD +1
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] The technical problem to be solved by the present invention is to overcome the defect of waste of human resources caused by highly repetitive work in the prior art in the scenario of real-time calculation of massive data with high concurrency and large flow rate, and when user needs are constantly changing. Provide a real-time monitoring method, system, device and storage medium for massive data that can realize different monitoring needs of users without repeated development and only with simple operations

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Mass data real-time monitoring method, system and device and storage medium
  • Mass data real-time monitoring method, system and device and storage medium
  • Mass data real-time monitoring method, system and device and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0071] like figure 1 As shown, the present embodiment provides a method for real-time monitoring of massive data, comprising the following steps:

[0072] Step 101. Configure a field pool for an access service, where the field pool includes several fields divided by the service.

[0073] In this embodiment, the configuration of the field pool is divided by the user according to the business. The more fields that can be reported by the business, the more suitable the scene is, indicating that the demand changes more, and the more repeated work can be saved.

[0074] Among them, each field is like a building block, which is divided into three types, including time type, limited type of enumeration type and numerical type for calculation. The time type is aimed at the timestamp contained in the business data uploaded by the user, and a field of the time type is generated according to the timestamp, specifically generating two fields of seconds and minutes, which are used to calc...

Embodiment 2

[0109] like Figure 7 As shown, this embodiment provides a real-time monitoring system for massive data, including a field pool module 1, a raw data acquisition module 2, a group calculation configuration module 3, a preview module 4, a group real-time calculation module 5 and a group calculation display module 6 .

[0110] The field pool module 1 is used to configure a field pool of an access service, and the field pool includes several fields divided according to the service.

[0111] In this embodiment, the configuration of the field pool is divided by the user according to the business. The more fields that can be reported by the business, the more suitable the scene is, indicating that the demand changes more, and the more repeated work can be saved.

[0112] Among them, each field is like a building block, which is divided into three types, including time type, limited type of enumeration type and numerical type for calculation. The time type is aimed at the timestamp ...

Embodiment 3

[0123] Figure 10 It is a schematic structural diagram of an electronic device provided by Embodiment 3 of the present invention. The electronic device includes a memory, a processor, and a computer program stored in the memory and operable on the processor. The processor implements the real-time monitoring method for massive data in Embodiment 1 when executing the program. image 3 The electronic device 30 shown is only an example, and should not limit the functions and scope of use of the embodiments of the present invention.

[0124] like Figure 10 As shown, electronic device 30 may take the form of a general-purpose computing device, which may be a server device, for example. Components of the electronic device 30 may include, but are not limited to: at least one processor 31 , at least one memory 32 , and a bus 33 connecting different system components (including the memory 32 and the processor 31 ).

[0125] The bus 33 includes a data bus, an address bus, and a contr...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a mass data real-time monitoring method, system and device and a storage medium, and the method comprises the following steps: configuring a field pool of an access service, the field pool comprising a plurality of fields divided according to the service; combining a plurality of fields selected by a user from the field pool according to service requirements to generate a corresponding configuration; in response to a starting instruction of a user, performing grouping calculation on the original data by using a real-time calculation framework according to the combination of the fields in each configuration to obtain a corresponding calculation result, and storing the calculation result in a database; and outputting the calculation result according to the requirements of the user. Aiming at a scene of real-time calculation of high-concurrency and large-flow mass data, repeated development is not needed for new requirements of users, new packet calculation services can be quickly generated only by simply dragging fields to increase configuration, the change of service requirements is quickly adapted, and the waste of human resources is avoided.

Description

technical field [0001] The invention relates to the technical field of big data processing, in particular to a method, system, device and storage medium for real-time monitoring of massive data. Background technique [0002] With the rapid development of big data, there are more and more demands for real-time calculation and monitoring of massive data with high concurrency and high flow rate. The call volume calculated by the client and the corresponding version is first buried in the system, and then the data is reported to a message queue, and then the topological program is written using the real-time computing framework, uploaded to the real-time computing framework cluster to start real-time computing and storage, and then Develop an interface display. The problem with the existing implementation method is that when the business side puts forward a new monitoring requirement, for example, to see the call volume of computer room group calculations per second, it is nece...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/24G06F16/248
Inventor 郝彦军李娟岳凯峰冯春海赵云霄
Owner BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap