Big data statistics method and system, computer device and storage medium

A statistical method and big data technology, applied in the field of big data, can solve the problems of data statistics not coming out, large database resource occupation, etc., to achieve the effect of improving the statistical speed and reducing the pressure of statistics

Inactive Publication Date: 2019-05-14
SHENZHEN MAPGOO TECH
View PDF8 Cites 51 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] The embodiment of the present application provides a big data statistics method, system, computer equipment and storage medium, which are used to solve the problem in th

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Big data statistics method and system, computer device and storage medium
  • Big data statistics method and system, computer device and storage medium
  • Big data statistics method and system, computer device and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0037] The embodiment of this application provides a big data statistics method, system, computer equipment and storage medium. In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation of the present invention will be described in detail below in conjunction with the accompanying drawings instruction of. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, the present invention can be implemented in many other ways different from those described here, and those skilled in the art can make similar improvements without departing from the connotation of the present invention, so the present invention is not limited by the specific implementations disclosed below.

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly unders...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention discloses a big data statistics method and system, a computer device and a storage medium. The method provided by the embodiment of the invention comprises the following steps: reading binlog of a Mysql database, and sequentially putting log records into a message queue; The message queue is consumed through an ETL service, log records in the message queue are extracted, cleaned, converted and loaded to obtain corresponding business data, and the business data are loaded to a corresponding data warehouse; carrying out real-time analysis, aggregation, query and offline calculation on the business data through a Spark distributed query engine to obtain a corresponding statistical result; According to the technical scheme, the data is imported into the warehouse in an incremental mode, the data are cleaned and stored after being cleaned, the statistical data are calculated in advance through offline calculation, the statistical data are directly taken out when the service system is used, the statistical speed is increased, and the statistical pressure of the database is reduced.

Description

technical field [0001] The present application relates to the field of big data technology, and in particular to a big data statistical method, system, computer equipment and storage medium. Background technique [0002] At present, with the explosive growth of the Internet, the amount of data is increasing rapidly, and the loss of the database is increasing. Some statistics and other operations are likely to cause database table locking and freezing. The master-slave deployment of the database will occupy a large amount of server IO. Especially for master-slave replication with relatively large database files, the system is easily stuck. At the beginning, there were not many data services and the pressure was not great, so a database server could meet the demand. With the rapid development of business and more and more data, it is necessary to deploy services such as DB and Web Server separately. However, as the number of users continues to grow and some report statistics...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/25G06F16/2458G06F16/28G06F16/215G06F9/54
Inventor 周志文朱宇翔李朝武王剑
Owner SHENZHEN MAPGOO TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap