Spark-based big data weblog acquisition, analysis and early warning method and system
A collection analysis and early warning system technology, applied in the computer field, can solve problems such as large-scale real-time data processing, etc., to achieve the effect of improving operational efficiency, high throughput capacity, and reducing work intensity
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Method used
Image
Examples
Example Embodiment
[0131] Example 1:
[0132] 1. The original log file buried point
[0133] Collect server CPU, memory, network card traffic, disk read and write speed and other log information, and collect points by writing LinuxShell scripts. Application operation information, including search module, resource module, order module survival and response time information, etc., are collected by writing Java programs and stored in Tomca application logs. For external interface access information, through embedding in the application program, using a combination of real-time throwing and separate storage of log files.
[0134] 2. Log collection and storage
[0135] Each server deploys and runs Flume Agent, collects and transmits log files, saves them in the distributed file system HFDS, and pushes real-time data to the Kafka messaging cluster. Then Spark Streaming consumes the messages in Kafka and performs calculations. Spark mainly processes offline data in HDSF and performs business logic processin...
PUM
Abstract
Description
Claims
Application Information
- R&D Engineer
- R&D Manager
- IP Professional
- Industry Leading Data Capabilities
- Powerful AI technology
- Patent DNA Extraction
Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.
© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap