A method and device for processing Internet of things log based on hadoop platform

Abstract

The invention provides an Internet of things log processing method and device based on a Hadoop platform, and aims at improving the analyzability of logs output by a system and meeting the periodic output requirement of report forms of service. The method comprises the steps that the log format of Internet of things logs is configured, and the Internet of things logs are collected according to the configured log format; classification and aggregation are conducted on the collected Internet of things logs according to categories identified by category identification character strings, and storage paths of the Internet of things logs subjected to classification and aggregation are recorded to a log memo sheet; the log memo sheet is periodically updated, or the Internet of things logs, needing to form a data report form, in the Internet of things logs subjected to classification and aggregation are loaded to a date partition of a hive sheet. According to the Internet of things log processing method and device based on the Hadoop platform, the analyzability of the output logs is significantly enhanced, and the service requirement for periodically outputting the analysis results can be met.

Description

Technical field [0001] The invention belongs to the field of data storage, and in particular relates to a method and device for processing Internet of Things logs based on a Hadoop platform. Background technique [0002] With the rapid development of the Internet of Things, more and more devices are connected to the Internet of Things, making the data generated by users, applications, and devices show a trend of rapid growth. The emergence of massive data and the complexity and change of data structures give the Internet of Things server End management and data analysis and processing have brought huge challenges. On the other hand, these data, such as the user's operating habits, equipment (refrigerators, washing machines, air conditioners, etc.) of the water use records, electricity consumption records, etc., often require a long storage period to mine user behavior data , Make smart recommendations. With the increase in the number of users and device connections, the demand ...

AI Technical Summary

Problems solved by technology

However, the logs output by the existing Hadoop platform-based Internet of Things log processing solutions not only have a large amount of information, but also contain various types of log information. The analyzability is poor. If the business needs to periodically output the analysis results, the above-mentioned processing scheme of the IoT log cannot meet this demand.

Images