Urban area people flow real-time monitoring and calculating method
A real-time computing and real-time monitoring technology, applied in the computer field, can solve problems such as inaccurate business service results, incomplete and continuous delay of data, etc., and achieve the effect of highly flexible streaming time window, high throughput, and high promotion value
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Method used
Image
Examples
Embodiment
[0114] figure 1 Based on hadoop 2.6.0+spark 1.6.0+Flume 1.2.0+Spark Streaming+Kafka environment, the production and consumption process is constructed.
[0115] 1) On the production side, through the data push of Spark streaming real-time stream, the window is opened under the condition of five minutes as the minimum time granularity; five minutes of real-time data can be received, the data is cleaned, and the time is set according to the minimum time granularity Grouping at intervals and judging the quality of each group of data at the same time, will meet the threshold condition of 80% data transfer to the consumer for consumption;
[0116] 2) Buffer pool module: For the data that does not meet 80% of the threshold condition, store it in the partition table partitioned by time granularity in hive. Merge the data in the partition table when the next real-time stream is connected, and jointly judge whether the data quality meets the threshold. If the threshold condition is s...
PUM
Abstract
Description
Claims
Application Information
- R&D Engineer
- R&D Manager
- IP Professional
- Industry Leading Data Capabilities
- Powerful AI technology
- Patent DNA Extraction
Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.
© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com