Massive data real-time screening and analysis method, system and storage medium

A technology of massive data and analysis methods, applied in the field of data processing, can solve problems such as ineffective caching of intermediate calculation results, increase server operating costs, and large server computing pressure, so as to reduce computing pressure, improve computing speed, and expand cached data. amount of effect

Active Publication Date: 2021-07-02
杭州美登科技股份有限公司
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The disadvantage of this method is that after the analyst adjusts the filter conditions, the entire task needs to be recalculated, and the intermediate calculation results cannot be effectively cached and utilized, which also affects the calculation speed to a certain extent.
[0006] To sum up, the calculation process of the above two schemes brings great calculation pressure to the server, which also indirectly increases the operating cost of the server

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Massive data real-time screening and analysis method, system and storage medium
  • Massive data real-time screening and analysis method, system and storage medium
  • Massive data real-time screening and analysis method, system and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0038] Taking a real-time screening analysis process for massive data as an example, the technical solution of the present invention is described in detail below, as figure 1 As shown, the analysis process includes the following steps:

[0039] S01: Build a basic condition database, compress the basic condition database and store it in the cloud space, and locally cache the data list corresponding to each data in the basic condition database;

[0040] Summarize all the basic conditions and the existing basic condition sets to obtain the basic condition database, and record the basic condition database as a compressed file in BitSet format, and store it in the general cloud space. This operation is generally performed offline. It should be noted that: the BitSet format is a compressed storage set format, which can only store index information of data sets that meet certain conditions rather than the data itself, and can quickly complete operations such as intersection, union, a...

Embodiment 2

[0059] Below in conjunction with the massive data real-time screening and analysis system described in the present invention, this system is applied to realize the above-mentioned massive data real-time screening and analysis method, and its workflow is:

[0060] First, build the basic condition database according to the basic condition and the existing basic condition set, record the basic condition database as a compressed BitSet format file through the basic condition compression cache module, and store it in the general cloud space / server, and record each item in the basic condition database The data list corresponding to the data is cached locally;

[0061] The data extraction module extracts the available data in the basic condition database and local cache data according to the filter conditions;

[0062] The data screening module screens the available data at the operation end to obtain intermediate calculation results and screening results; specifically, the data scre...

Embodiment 3

[0065] A storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above analysis method are realized.

[0066] The above-mentioned three embodiments respectively introduce the massive data real-time screening method, system and storage medium in detail. Those skilled in the art should understand that the embodiments of the present invention can be provided as methods, devices or computer program products. Accordingly, the present invention can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a method, system and storage medium for real-time screening and analysis of massive data. The method includes: constructing a basic condition database and compressing and storing it; combining the screening conditions with the basic condition database to determine the available data in the local cache data; Combine the available data to perform logic operations on the operation side, and store the intermediate operation results; display the final logic operation results. The present invention transfers the data screening process from the server side to the browser side, reasonably utilizes the built-in cache of the browser, and obviously solves the calculation time. operation and maintenance costs.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a method, system and storage medium for real-time screening and analysis of massive data. Background technique [0002] With the increasing development of social informatization, big data and data screening have gradually become public knowledge, but how to improve the speed of screening and analysis of massive data is still a difficult problem for those skilled in the art. [0003] There are mainly two existing solutions for screening and analyzing massive data: [0004] One solution is to use a database to process raw data, such as figure 1 , taking the real-time screening of a large number of users as an example, first analyze all the screening conditions, select the condition with the least number of users that meet the conditions, obtain the user list through the database index, and finally judge whether the remaining conditions are satisfied one by one. This filte...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & AuthorityPatents(China)
IPC IPC(8): G06F16/22G06F16/2458G06F16/2455
Inventor邓鋆
Owner杭州美登科技股份有限公司