Mass data real-time screening analysis method and system, and storage medium

A technology of massive data and analysis methods, applied in the field of data processing, can solve problems such as ineffective caching of intermediate calculation results, increase server operating costs, and large computing pressure on servers, save data search time, improve computing speed, and expand The effect of the amount of cached data

Active Publication Date: 2018-10-16
杭州美登科技股份有限公司
View PDF5 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The disadvantage of this method is that after the analyst adjusts the filter conditions, the entire task needs to be recalculated, and the intermediate calculation results cannot be effectively cached and utilized, which also affects the calculation speed to a certain extent.
[0006] To sum up, the calculation process of the above two schemes brings great calculation pressure to the server, which also indirectly increases the operating cost of the server

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Mass data real-time screening analysis method and system, and storage medium
  • Mass data real-time screening analysis method and system, and storage medium
  • Mass data real-time screening analysis method and system, and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0038] Taking a real-time screening analysis process for massive data as an example, the technical solution of the present invention is described in detail below, as figure 1 As shown, the analysis process includes the following steps:

[0039] S01: Build a basic condition database, compress the basic condition database and store it in the cloud space, and locally cache the data list corresponding to each data in the basic condition database;

[0040] Summarize all the basic conditions and the existing basic condition sets to obtain the basic condition database, and record the basic condition database as a compressed file in BitSet format, and store it in the general cloud space. This operation is generally performed offline. It should be noted that: the BitSet format is a compressed storage set format, which can only store index information of data sets that meet certain conditions rather than the data itself, and can quickly complete operations such as intersection, union, a...

Embodiment 2

[0059] Below in conjunction with the massive data real-time screening and analysis system of the present invention, the system is applied to realize the above-mentioned massive data real-time screening and analysis method, and its workflow is:

[0060] First, build the basic condition database according to the basic conditions and the existing basic condition sets, record the basic condition database as a compressed BitSet format file through the basic condition compression and cache module, and store it in the general cloud space / server, and record each item in the basic condition database as a compressed BitSet format file. The data list corresponding to the data is cached locally;

[0061] The data extraction module extracts the available data in the basic condition database and the local cache data according to the filtering conditions;

[0062] The data screening module filters the available data on the operation side, and obtains the intermediate operation results and sc...

Embodiment 3

[0065] A storage medium storing a computer program, when the computer program is executed by a processor, implements the steps of the above analysis method.

[0066] The above three embodiments respectively introduce the method, system and storage medium for real-time screening of massive data in detail. Those skilled in the art should understand that the embodiments of the present invention can be provided as methods, apparatuses or computer program products. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a mass data real-time screening analysis method and system, and a storage medium. The method comprises the steps of constructing a basic condition database, and carrying out compression storage; according to screening conditions, in combination with the basic condition database, determining available data in local cache data; performing logic operation at an operating end in combination with the available data, and storing an intermediate operation result; and displaying a final logic operation result. According to the method and the system, by transferring a data screening process from a server end to a browser end, a cache of a browser is reasonably utilized, so that the problem of calculation time is obviously solved, meanwhile, the calculation amount of a serveris reduced, the calculation speed is increased, and the operation and maintenance cost of the server is reduced.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a method, system and storage medium for real-time screening and analysis of massive data. Background technique [0002] With the increasing development of social informatization, big data and data screening have gradually become public knowledge, but how to improve the speed of screening and analysis of massive data is still a difficult problem for those skilled in the art. [0003] There are mainly two existing solutions for screening and analyzing massive data: [0004] One solution is to use a database to process raw data, such as figure 1 , taking the real-time screening of a large number of users as an example, first analyze all the screening conditions, select the condition with the least number of users that meet the conditions, obtain the user list through the database index, and finally judge whether the remaining conditions are satisfied one by one. This filte...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F17/30
Inventor邓鋆
Owner杭州美登科技股份有限公司