Active data processing method, device and storage medium of Internet of Things devices

Abstract

The present invention discloses a method, device, and storage medium for processing active data of an Internet of Things device. The method comprises the following steps: obtaining device-reported messages and pushing the device-reported messages to a distributed message queue; the device-reported messages include message types; constructing a data stream of the device-reported messages in the message queue based on a stream computing service; cleaning the data stream according to the message type to obtain a data stream containing only device active data; processing the device active data in a cache according to the data stream; processing the device active data in HBase according to the device active data in the cache; batch processing the device active data in HBase based on a batch processing service to obtain active basic statistical data and store them in a database. The present application meets the high concurrency requirements generated by massive device data through a stream processing service, introduces a cache as a buffer, and can reduce data interaction with HBase, thereby improving data processing performance.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a method, device, and storage medium for processing active data of an Internet of Things device. Background Art

[0002] With the development of IoT technology, the number of devices within it is increasing rapidly. Device activity data is a crucial indicator and operational data for IoT platforms. Due to the sheer number of devices within the IoT, their diverse form factors, and their widely varying behaviors, traditional database collection and recording methods for daily and monthly device activity data are no longer sufficient to meet the high concurrency demands of this massive device population. Summary of the Invention

[0003] The present application aims to solve at least one of the technical problems in the related art to a certain extent. To this end, the present application proposes a method, apparatus and storage medium for processing active data of an Internet of Things device.

[0004] In the first aspect, an embodiment of the present application provides a method for processing active data of an Internet of Things device, comprising: obtaining device-reported messages, and pushing the device-reported messages to a distributed message queue; wherein the device-reported messages include a message type; based on a stream computing service, constructing a data stream of the device-reported messages in the message queue; according to the message type, cleaning the data stream to obtain the data stream containing only device active data; based on the data stream after cleaning, adding, modifying or not processing the device active data in the cache; based on the device active data in the cache, adding, modifying or not processing the device active data in Hbase; based on a batch processing service, batch processing the device active data in the Hbase to obtain active basic statistical data, and storing the active basic statistical data in a database.

[0005] Optionally, the device activity data includes the device ID, the device online and offline dates, and the device online and offline times. The device activity data in the cache is added, modified, or not processed based on the data stream after the cleaning is completed, including: querying the cache based on the device ID and the online and offline dates in the data stream; when it is found that the device activity data of the same device on the same date does not exist in the data stream and the cache, adding the device activity data to the cache based on the device ID, the online and offline dates, and the online and offline times in the data stream; when it is found that the device activity data of the same device on the same date exists in the data stream and the cache, modifying or not processing the device activity data in the cache based on the online and offline times.

[0006] Optionally, when it is queried that the device activity data of the same device on the same date exists in the data stream and the cache, the device activity data in the cache is modified or not processed according to the online and offline time, including: when the device activity data of the same device on the same date exists in the data stream and the cache, the first online and offline time in the data stream is compared with the second online and offline time in the cache; when the first online and offline time is later than the second online and offline time, the second online and offline time in the cache is replaced with the first online and offline time; when the first online and offline time is earlier than or equal to the second online and offline time, no processing is performed.

[0007] Optionally, the device active data in Hbase is added, modified or not processed according to the device active data in the cache, including: setting a scheduled task according to business needs; the scheduled task includes an operation interval; every time the operation interval passes, the device active data in Hbase is added, modified or not processed according to the device active data in the cache.

[0008] Optionally, the adding, modifying or not processing the device activity data in Hbase according to the device activity data in the cache includes: querying the Hbase according to the device ID in the cache; when the device activity data of the same device exists in the cache and the Hbase, comparing the first online and offline date in the cache with the second online and offline date in the Hbase; when the first online and offline date is different from the second online and offline date, adding the device activity data to the Hbase according to the device ID, the online and offline date and the online and offline time in the cache; when the first online and offline date is the same as the second online and offline date, and the first online and offline time in the cache is earlier than the second online and offline time in the Hbase, replacing the device activity data in the Hbase with the device activity data in the cache; when the first online and offline date is the same as the second online and offline date, and the first online and offline time in the cache is later than or equal to the second online and offline time in the Hbase, not processing.

[0009] Optionally, based on the batch processing service, the device activity data in the Hbase is batch processed to obtain active basic statistical data, and the active basic statistical data is stored in the database, including: based on the batch processing service, batch acquiring the device activity data in the Hbase; batch parsing the device activity data to obtain user fields, product fields and device fields in the device activity data; assembling the parsed device activity data into an online and offline data sequence, the online and offline data sequence consisting of multiple unit data; recursively starting from the last unit data in the online and offline data sequence, judging the activity status of the device within the specified time period according to the unit data; performing aggregation calculation on the activity status according to the user field, the product field, the device field and the specified time period, obtaining the active basic statistical data, and storing the active basic statistical data in the database.

[0010] Optionally, the unit data includes device operation and operation time; the recursion starts from the last unit data in the online and offline data sequence and judges the activity of the device within the specified time period according to the unit data, including: when the operation time is within the specified time period, judging that the device is active within the specified time period; when the operation time is later than the end time of the specified time period, taking the previous unit data for re-judgment; when the operation time is earlier than or equal to the start time of the specified time period and the device operation is online, judging that the device is active within the specified time period; when the operation time is earlier than or equal to the start time of the specified time period and the device operation is offline, judging that the device is inactive within the specified time period.

[0011] In the second aspect, an embodiment of the present application provides an active data processing system for IoT devices, including: a first module for obtaining device-reported messages and pushing the device-reported messages to a distributed message queue; wherein the device-reported messages include message types; a second module for constructing a data stream of the device-reported messages in the message queue based on a stream computing service; a third module for cleaning the data stream according to the message type to obtain the data stream containing only device active data; a fourth module for adding, modifying or not processing the device active data in the cache based on the data stream after cleaning; a fifth module for adding, modifying or not processing the device active data in Hbase based on the device active data in the cache; a sixth module for batch processing the device active data in Hbase based on a batch processing service to obtain active basic statistical data, and store the active basic statistical data in a database.

[0012] In a third aspect, an embodiment of the present application provides a device comprising: at least one processor; at least one memory for storing at least one program; when the at least one program is executed by the at least one processor, the at least one processor implements the active data processing method of the Internet of Things device as described in the first aspect.

[0013] In a fourth aspect, an embodiment of the present application provides a computer storage medium storing a program executable by a processor, which, when executed by the processor, is used to implement the active data processing method of the Internet of Things device as described in the first aspect.

[0014] The beneficial effects of the embodiments of the present application are as follows: first, the device report message is obtained, and the device report message is pushed to a distributed message queue; wherein, the device report message includes a message type; based on the stream computing service, a data stream of the device report message is constructed in the message queue; according to the message type, the data stream is cleaned to obtain a data stream containing only device active data; according to the data stream after cleaning, the device active data in the cache is added, modified or not processed; according to the device active data in the cache, the device active data in Hbase is added, modified or not processed; based on the batch processing service, the device active data in Hbase is batch processed to obtain active basic statistical data, and the active basic statistical data is stored in the database. The present application processes the device active data through the stream processing service, which can further meet the high concurrency requirements generated by massive device data. In addition, the present application introduces a cache as a buffer for device active data, which can reduce data interaction with Hbase and reduce a large amount of disk I / O processing, thereby further improving data processing performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings are used to provide a further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0016] Figure 1 A flowchart of the steps of the method for processing active data of an IoT device provided in an embodiment of the present application;

[0017] Figure 2 This is a flowchart for implementing the data acquisition optimization algorithm proposed in the embodiment of this application;

[0018] Figure 3 A flowchart of the steps for synchronizing device active data from cache to HBase provided in an embodiment of the present application;

[0019] Figure 4 A schematic diagram of an active data processing system for an Internet of Things device provided in an embodiment of the present application;

[0020] Figure 5 A schematic diagram of the device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0022] It should be noted that although the system diagrams illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the system or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0023] The embodiments of the present application are further described below with reference to the accompanying drawings.

[0024] refer to Figure 1 , Figure 1 This is a flowchart of a method for processing active data of an IoT device provided in an embodiment of the present application, which includes but is not limited to steps S100-S150:

[0025] S100: Obtain a device-reported message and push the device-reported message to a distributed message queue; wherein the device-reported message includes a message type;

[0026] Specifically, the IoT platform connects to a large number of diverse IoT devices through various protocols, such as MQTT (Message Queuing Telemetry Transport) and LWM2M (Lightweight Machine-To-Machine). In daily IoT operations, devices need to report their information to the IoT platform. This information is called device reports. Because device reports are often large in volume, the IoT platform performs a simple analysis of these reports and pushes them to a distributed message queue for temporary storage.

[0027] It is understandable that the device-reported information may reflect various situations of IoT devices during operation, so the device-reported information may include device failure data, device energy consumption data, or device activity data. The message type of the device-reported information can be used to distinguish various types of device data.

[0028] S110: Based on the stream computing service, construct a data stream of device-reported messages in the message queue;

[0029] Specifically, based on the stream computing service, a data source stream object is constructed, and the data source that needs to be consumed in the distributed message queue is specified, so as to build a data stream of device-reported messages in the message queue, and the consumption data in the message queue (i.e., device-reported messages) is monitored in real time through the stream computing engine.

[0030] S120. Clean the data stream according to the message type to obtain a data stream containing only active data of the device;

[0031] Specifically, as mentioned above, the message type of the device's reported information can be used to distinguish device-reported messages reflecting different device conditions. Since the embodiments of this application propose a method for processing active data of IoT devices, in this step, the data stream is cleaned according to the message type of the device's reported message. Specifically, non-device active data in the data stream, such as device failure data and device energy consumption data, is cleaned and filtered out to obtain a data stream containing only device active data.

[0032] It can be understood that since the device is in an active state, it generally means that the device is online; similarly, the device is in an inactive state, it generally means that the device is offline. Therefore, the device activity data is data that can reflect the online and offline status of the device.

[0033] More specifically, device activity data includes but is not limited to device ID, device online and offline dates, and device online and offline times. It can be understood that online and offline dates are divided into online dates and offline dates, and online and offline times are divided into online time and offline time. The time corresponding to the device online or offline operation is the device online time or offline time.

[0034] S130, adding, modifying, or not processing the device active data in the cache according to the cleaned data stream;

[0035] Specifically, the embodiment of the present application introduces a cache in the process of processing active data of IoT devices. The cache is a window aggregation database. The device active data first enters the cache from the message queue, and the data collection optimization algorithm proposed in the embodiment of the present application is run in the cache to further screen and integrate the device active data. The data in the cache is then integrated into Hbase, thereby greatly reducing the amount of data entering Hbase. By reducing the amount of data, disk I / O processing is reduced, thereby achieving the purpose of improving data processing capabilities.

[0036] As mentioned above, device activity data includes, but is not limited to, the device ID, the device's online and offline dates, and the device's online and offline times. The data collection optimization algorithm proposed in this embodiment can integrate the device activity data from the data stream into the device activity data in the cache based on data such as the device ID. The specific process of the data collection optimization algorithm is described below.

[0037] Reference Figure 2 , Figure 2 This is a flowchart of the implementation of the data acquisition optimization algorithm proposed in the embodiment of the present application. The method includes but is not limited to steps S200-S250:

[0038] S200, query the cache based on the device ID and online / offline date in the data stream;

[0039] Specifically, the device ID and online and offline dates in the data stream are used as keys to query the cache.

[0040] It is understandable that when data is moved from the data stream to the cache, or from the cache to the Hbase, some storage format adjustments may be required, resulting in differences in the device activity data in the data stream, cache and Hbase. However, the device activity data in the data stream, cache and Hbase all contain information such as the device ID, the device's online and offline dates, the device's online and offline times, etc., which can reflect the activity status of a certain device. Therefore, in this embodiment of the application, the data reflecting the activity status of the device stored in the data stream, cache and Hbase are collectively referred to as device activity data, and the device activity data located in different places in the data stream, cache and Hbase can be compared to determine the subsequent data processing steps.

[0041] S210, determining whether there is device activity data for the same device on the same date in the data stream and the cache;

[0042] Specifically, step S200 queries the device activity data in the cache based on the two conditions of device ID and online and offline dates. If the device ID in the data stream is queried in the cache, it means that data of the same device exists in the data stream and the cache; if the online and offline dates of the device queried in the cache are the same as the online and offline information in the data stream, it means that device activity data of the same device on the same date exists in the data stream and the cache.

[0043] Therefore, this step is to judge the result of the query implemented in step S200, that is, to judge whether there is device activity data of the same device on the same date in the data stream and cache. If not, jump to step S220; if so, jump to step S230.

[0044] S220: Add device activity data to the cache based on the device ID, online / offline date, and online / offline time in the data stream;

[0045] Specifically, according to step S210, if it is determined that there is no device activity data of the same device on the same date in the data stream and the cache, then a new device activity data is added to the cache with the device ID, online and offline date and online and offline time.

[0046] S230, determining whether the first online and offline time in the data stream is later than the second online and offline time in the cache;

[0047] Specifically, if it is determined in step S210 that there is device activity data of the same device on the same date in the data stream and the cache, further judgment is performed in this step to compare the first online and offline time in the data stream with the second online and offline time in the cache.

[0048] As mentioned in the above content, online and offline time is divided into online time and offline time. It can be understood that when comparing the first online and offline time and the second online and offline time, the online time in the first online and offline time should be compared with the online time in the second online and offline time, and the offline time in the first online and offline time should be compared with the offline time in the second online and offline time.

[0049] According to the above comparison method, it is determined whether the first online and offline time in the data stream is later than the second online and offline time in the cache. If not, jump to step S240; if so, jump to step S250.

[0050] S240: Do not process the active device data in the cache;

[0051] Specifically, if it is determined in step S230 that the first online and offline time in the data stream is earlier than or equal to the second online and offline time in the cache, the device activity data in the cache is not processed.

[0052] It can be understood that the first online and offline time is earlier than or equal to the second online and offline time specifically means that the online time in the first online and offline time is earlier than or equal to the online time in the second online and offline time, and the offline time in the first online and offline time is earlier than or equal to the offline time in the second online and offline time.

[0053] S250: Replace the second online and offline time in the cache with the first online and offline time;

[0054] Specifically, according to step S230 , if it is determined that the first online and offline time in the data stream is later than the second online and offline time in the cache, the second online and offline time corresponding to the device ID in the cache is replaced with the first online and offline time.

[0055] It can be understood that the first online and offline time is later than the second online and offline time, which specifically means that the online time in the first online and offline time is later than the online time in the second online and offline time and / or the offline time in the first online and offline time is earlier than or equal to the offline time in the second online and offline time. Replacing the second online and offline time with the first online and offline time specifically means: replacing the online time in the second online and offline time with the later online time in the first online and offline time and / or replacing the offline time in the second online and offline time with the later offline time in the first online and offline time.

[0056] Through steps S200-S250, the embodiment of the present application provides a data acquisition optimization algorithm. Running the data acquisition optimization algorithm on the data stream can greatly streamline the device activity data. Taking the reporting frequency of smart community elevator equipment as an example, the traditional device daily activity storage records are as many as 86,400 per day. The device activity optimization acquisition algorithm is used to calculate daily activity, only one original record needs to be stored for daily activity and monthly activity calculations. Taking the reporting frequency of smart power meter equipment as an example, the traditional device daily activity storage records are as many as 96 per day. The device activity optimization acquisition algorithm is used to calculate daily activity, only one original record needs to be stored for daily activity and monthly activity calculations. Therefore, the data acquisition optimization algorithm in the embodiment of the present application helps to reduce the amount of data storage in the cache, effectively avoiding the storage and computing pressure brought to the system by high-frequency online and offline devices.

[0057] Step S130 has been explained above, and step S140 will be explained below.

[0058] S140, adding, modifying, or not processing the device active data in Hbase according to the device active data in the cache;

[0059] Specifically, various IoT services have different timeline requirements. Scheduled tasks can be set based on these needs. These scheduled tasks include an operation interval. After each operation interval, the device activity data in HBase is added, modified, or not processed based on the cached device activity data, thereby achieving the goal of regularly synchronizing the cached device activity data with HBase.

[0060] During data synchronization, the cached data can be further streamlined, further reducing the number of data interactions with HBase, thereby improving data processing performance. The following describes the synchronization process of active device data from the cache to HBase.

[0061] Reference Figure 3 , Figure 3 This is a flowchart of the steps of synchronizing device active data from cache to HBase provided in an embodiment of the present application. The method includes but is not limited to steps S300-S370:

[0062] S300: Query Hbase based on the device ID in the cache;

[0063] Specifically, the device ID in the cache is used as the key to query Hbase.

[0064] S310, determining whether there is active device data for the same device in the cache and Hbase;

[0065] Specifically, if the device ID in HBase is the same as the device ID in the cache, it means that the cache and HBase both have active device data for the same device. Therefore, in this step, it is determined whether the cache and HBase both have active device data for the same device. If not, the process jumps to step S320; if so, the process jumps to step S330.

[0066] S320. Add device activity data to Hbase based on device ID, online / offline date, and online / offline time.

[0067] Specifically, if it is determined in step S310 that device activity data for the same device does not exist in the cache or HBase, then the device activity data corresponding to the device ID is added to HBase based on the device ID, online and offline date, and online and offline time. It is understood that the device activity data corresponding to the device ID will be recorded under this newly created device ID.

[0068] S330: Determine whether the first online / offline date in the cache is the same as the second online / offline date in Hbase;

[0069] Specifically, according to step S310 , if it is determined that there is device activity data of the same device in the cache and Hbase, the online and offline dates corresponding to the device ID are further determined.

[0070] It is understood that the online date in the first online / offline date is compared with the online date in the second online / offline date, and the offline date in the first online / offline date is compared with the offline date in the second online / offline date. In this step, it is determined whether the first online / offline date in the cache and the second online / offline date in HBase are the same. If not, the process jumps to step S340; if so, the process jumps to step S350.

[0071] S340, add device active data in Hbase;

[0072] Specifically, according to step S330, if the first online and offline date is different from the second online and offline date, that is, the online date in the first online and offline date is different from the online date in the second online and offline date, and the offline date in the first online and offline date is different from the offline date in the second online and offline date, then the active number of the device in the cache is inserted under the device ID in Hbase.

[0073] S350: Determine whether the first online and offline time in the cache is earlier than the second online and offline time in Hbase;

[0074] Specifically, according to step S330, it is determined that the first online and offline dates are the same as the second online and offline dates, that is, the online date in the first online and offline dates is different from the online date in the second online and offline dates and / or the offline date in the first online and offline dates is different from the offline date in the second online and offline dates. Then, it is further determined whether the first online and offline time in the cache is earlier than the second online and offline time in Hbase. If not, jump to step S360; if yes, jump to step S370.

[0075] It can be understood that the online time in the first online and offline time is compared with the online time in the second online and offline time, and the offline time in the first online and offline time is compared with the offline time in the second online and offline time.

[0076] S360, discard the current data stream;

[0077] Specifically, according to step S350, if the first online and offline time is later than or equal to the second online and offline time in Hbase, that is, the online time in the first online and offline time is later than or equal to the online time in the second online and offline time, and the offline time in the first online and offline time is later than or equal to the offline time in the second online and offline time, then Hbase will not be processed and the current data stream in the cache will be discarded.

[0078] S370: Replace the active data of the device in Hbase with the active data of the device in the cache;

[0079] Specifically, according to step S350, if the first online and offline time is earlier than the second online and offline time in Hbase, that is, the online time in the first online and offline time is earlier than the online time in the second online and offline time and / or the offline time in the first online and offline time is earlier than the offline time in the second online and offline time, then the device active data in Hbase is replaced with the device active data in the cache, specifically, the device active data in Hbase is deleted, and a new device active data is added according to the device active data in the cache.

[0080] Through steps S300-S370, the embodiment of the present application provides a synchronization process of active data of a device from cache to Hbase. Step S140 has been explained, and step S150 will be explained below.

[0081] S150: Based on the batch processing service, batch process the device activity data in Hbase to obtain basic activity statistics, and store the basic activity statistics in the database;

[0082] Specifically, the present embodiment proposes establishing batch processing tasks based on a batch processing engine to batch process device activity data in HBase, ultimately obtaining basic activity statistics and storing them in a database. Subsequently, when the IoT platform needs to perform daily or monthly device activity statistics at the user, product, or platform level, it can directly call upon the basic activity statistics in the database for statistical analysis.

[0083] In an embodiment of the present application, a method for processing device activity data in Hbase according to a batch processing engine includes: first, based on a batch processing service, batch obtaining device activity data in Hbase; then, by extending a batch processing JobTask, batch parsing the device activity data, obtaining the user field, product field, and device field in the device activity data, and assembling the parsed device activity data into an online and offline data sequence consisting of multiple unit data. Then, the online and offline data sequence is logically processed according to the active determination optimization algorithm proposed in the embodiment of the present application, that is, starting from the last unit data in the online and offline data sequence and recursively forward, the activity of the device within a specified time period is judged according to the unit data. After completing the processing of the online and offline data sequence, the user field, product field, and device field are obtained according to the parsing, and the activity is aggregated and calculated within the specified time period to obtain active basic statistical data of different levels. For example, the active basic statistical data of multiple IoT devices corresponding to the same user within a specified time period can be calculated, and the active basic statistical data of all the same products (such as all cameras, face access control, etc. IoT devices in a smart community) within a specified time period can also be calculated. It is also possible to calculate the active basic statistical data of all devices in the IoT platform. Finally, the calculated active basic statistics are stored in the database.

[0084] The following describes the implementation process of the activity determination optimization algorithm proposed in the embodiment of the present application. First, determine the last unit data in the online and offline data sequence. The unit data includes device operation and operation time. Device operation refers to the time when the device is online or offline. Operation time refers to the time when the device is online or offline. Taking the current unit data as the standard, when the operation time is within the specified time period, the device is judged to be active within the specified time period; when the operation time is later than the end time of the specified time period, the previous unit data in the online and offline data sequence is taken and the activity determination is performed again; when the operation time is earlier than or equal to the start time of the specified time period, and the device operation is online, the device is judged to be active within the specified time period; when the operation time is earlier than or equal to the start time of the specified time period, and the device operation is offline, the device is judged to be inactive within the specified time period.

[0085] Based on the above content, the activity determination optimization algorithm is run on the online and offline data series to finally determine the basic activity statistics.

[0086] Through steps S100-S150, the embodiment of the present application provides an active data processing method for an IoT device, the method comprising: first obtaining a device report message and pushing the device report message to a distributed message queue; wherein the device report message includes a message type; based on a stream computing service, constructing a data stream of the device report message in the message queue; according to the message type, cleaning the data stream to obtain a data stream containing only device active data; based on the cleaned data stream, adding, modifying or not processing the device active data in the cache according to the data acquisition optimization algorithm; adding, modifying or not processing the device active data in Hbase according to the device active data in the cache; based on the batch processing service, batch processing the device active data in Hbase according to the active determination optimization algorithm, obtaining active basic statistical data, and storing the active basic statistical data in the database. The present application processes the device active data through the stream processing service, which can further meet the high concurrency requirements generated by massive device data. In addition, the present application introduces a cache as a buffer for device active data, which can reduce data interaction with Hbase and reduce a large amount of disk I / O processing, thereby further improving data processing performance.

[0087] Below, in combination with the above multiple embodiments and actual situations, the active data processing method of the Internet of Things device provided by the embodiment of the present application is described. The specific steps are as follows:

[0088] 1) Prepare 21 virtual machines with 16GB of memory and 8-core CPUs. Deploy the distributed real-time stream computing engine Flink on 9 servers, with a Flink cluster deployed on every 3 servers (Flink clusters 1 to 3). Deploy the Redis cache service on 3 servers; deploy the distributed message queue Pulsar on 3 servers; deploy the HBase database on 3 servers; and deploy the MySQL database on 3 servers.

[0089] 2) Create a push queue for device-reported messages on Pulsar, which is used by the IoT platform to receive messages reported by massive devices and push them to the specified queue.

[0090] 3) Deploy stream computing services for data collection and storage on the 1st and 2nd Flink clusters, and deploy batch processing services for active computing on massive devices on the 3rd Flink cluster.

[0091] 4) Set the number of job tasks for the stream computing service in Flink clusters 1-2 to 30, supporting a maximum of 30 data streams for data processing services simultaneously. Set the stream computing service for data collection and storage to listen to the topic queue for device-reported messages in the Pulsar cluster.

[0092] 5) After receiving messages reported by devices, the stream computing service first cleans the messages, filtering out messages other than those related to device online and offline. It then parses the cleaned data to identify information such as the user, product, device, operation time, and operation ID, generating structured data.

[0093] 6) Perform data collection optimization algorithm processing on the structured data. The specific processing method refers to the above steps S200-S250:

[0094] 7) Set the number of job tasks for the batch computing service in Flink cluster 3 to 15, supporting a maximum of 15 data streams for batch data processing at the same time.

[0095] 8) The IoT platform scheduling service triggers the batch processing service at 00:05 every night to calculate the daily and monthly active status of the massive number of devices on the platform.

[0096] 9) The batch processing service loads massive amounts of device online and offline data from HBase at one time and parses the data to form a structured list.

[0097] 10) Apply an activity determination optimization algorithm to the device online and offline data list. For specific processing methods, refer to step S150 above. If daily device activity is to be calculated, the designated period is a specific day; if monthly device activity is to be calculated, the designated period is a specific month. Active devices are marked as 1, and inactive devices are marked as 0.

[0098] 11) Aggregate and accumulate the determined device data from the dimensions of user, product, and time range to generate basic activity statistics.

[0099] 12) After the 15 job tasks are completed, the massive amount of active basic statistical data is stored and inserted into the MySQL database for subsequent business-level data statistical processing.

[0100] 13) After obtaining a large amount of basic data on device activity, the IoT platform can use this data to generate daily and monthly statistics on user devices, product-level devices, or platform-wide devices.

[0101] In addition, refer to Figure 4 , Figure 4This is a schematic diagram of an active data processing system for IoT devices provided in an embodiment of the present application. The system 400 includes a first module 410, a second module 420, a third module 430, a fourth module 440, a fifth module 450, and a sixth module 460. The first module is used to obtain device-reported messages and push the device-reported messages to a distributed message queue; wherein the device-reported messages include message types; the second module is used to construct a data stream of device-reported messages in the message queue based on a stream computing service; the third module is used to clean the data stream based on the message type to obtain a data stream containing only device active data; the fourth module is used to add, modify, or not process the device active data in the cache based on the cleaned data stream; the fifth module is used to add, modify, or not process the device active data in HBase based on the device active data in the cache; and the sixth module is used to batch process the device active data in HBase based on a batch processing service to obtain basic active statistical data and store the basic active statistical data in a database.

[0102] refer to Figure 5 , Figure 5 Schematic diagram of a device provided in an embodiment of the present application, wherein the device 500 includes at least one processor 510 and at least one memory 520 for storing at least one program; Figure 5 A processor and a memory are taken as an example.

[0103] The processor and memory can be connected via a bus or other means. Figure 5 The bus connection is taken as an example.

[0104] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely located relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0105] Another embodiment of the present application further provides a device, which can be used to execute the control method in any of the above embodiments, for example, to execute the above described Figure 1 The method steps in .

[0106] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0107] An embodiment of the present application further discloses a computer storage medium storing a program executable by a processor, wherein the program executable by the processor is used to implement the active data processing method of the Internet of Things device proposed in the present application when executed by the processor.

[0108] Those skilled in the art will appreciate that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those skilled in the art, the term computer storage media is included in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data) and is volatile and non-volatile, removable, and non-removable. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage, or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0109] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above implementation mode. Technical personnel familiar with the field can also make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A method for processing active data of an Internet of Things device, characterized in that: include: Obtaining a device-reported message and pushing the device-reported message to a distributed message queue; wherein the device-reported message includes a message type; Based on the stream computing service, construct a data stream of the message reported by the device in the message queue; Cleaning the data stream according to the message type to obtain the data stream containing only active data of the device; Add, modify or not process the device active data in the cache according to the cleaned data stream; Add, modify or not process the device active data in Hbase according to the device active data in the cache; Based on a batch processing service, batch processing is performed on the device activity data in the Hbase to obtain basic activity statistics data, and the basic activity statistics data is stored in a database, including: Based on the batch processing service, batch obtain the device active data in the Hbase; Batch parsing the device activity data to obtain user fields, product fields, and device fields in the device activity data; Assembling the parsed device activity data into an online and offline data sequence, wherein the online and offline data sequence is composed of multiple unit data; the unit data includes device operations and operation time; Starting from the last unit data in the online and offline data sequence and recursively forward, judging the activity of the device within the specified time period based on the unit data, including: when the operation time is within the specified time period, judging that the device is active within the specified time period; when the operation time is later than the end time of the specified time period, re-judging by taking the previous unit data; when the operation time is earlier than or equal to the start time of the specified time period and the device operation is online, judging that the device is active within the specified time period; when the operation time is earlier than or equal to the start time of the specified time period and the device operation is offline, judging that the device is inactive within the specified time period; The activity situation is aggregated and calculated based on the user field, the product field, the device field and the specified time period to obtain the activity basic statistical data, and the activity basic statistical data is stored in the database.

2. The method for processing active data of an IoT device according to claim 1, wherein: The device activity data includes the device ID, the device online and offline date, and the device online and offline time. The adding, modifying, or not processing the device activity data in the cache according to the data stream after the cleaning is completed includes: querying the cache according to the device ID and the online and offline dates in the data stream; When it is found that the device activity data of the same device on the same date does not exist in the data stream and the cache, the device activity data is added to the cache according to the device ID, the online and offline dates, and the online and offline times in the data stream; When it is found that the device activity data of the same device on the same date exists in the data stream and the cache, the device activity data in the cache is modified or not processed according to the online and offline time.

3. The method for processing active data of an IoT device according to claim 2, characterized in that: When the query finds that the device activity data of the same device on the same date exists in the data stream and the cache, modifying or not processing the device activity data in the cache according to the online and offline time includes: When the data stream and the cache contain the device activity data of the same device on the same date, comparing the first online and offline time in the data stream with the second online and offline time in the cache; When the first online and offline time is later than the second online and offline time, replacing the second online and offline time in the cache with the first online and offline time; When the first online and offline time is earlier than or equal to the second online and offline time, no processing is performed.

4. The method for processing active data of an IoT device according to claim 2, wherein: The adding, modifying, or not processing the device active data in Hbase according to the device active data in the cache includes: Set up scheduled tasks according to business needs; the scheduled tasks include the operation interval length; Every time the operation interval passes, the device active data in the Hbase is added, modified, or not processed according to the device active data in the cache.

5. The method for processing active data of an Internet of Things device according to any one of claims 2 to 4, characterized in that: The adding, modifying, or not processing the device active data in Hbase according to the device active data in the cache includes: Query the Hbase according to the device ID in the cache; When the cache and the Hbase both contain the device activity data of the same device, the first online and offline date in the cache is compared with the second online and offline date in the Hbase; When the first online / offline date is different from the second online / offline date, adding the device activity data to the Hbase according to the device ID, the online / offline date, and the online / offline time in the cache; When the first online / offline date is the same as the second online / offline date, and the first online / offline time in the cache is earlier than the second online / offline time in the Hbase, the device activity data in the Hbase is replaced with the device activity data in the cache; When the first online / offline date is the same as the second online / offline date, and the first online / offline time in the cache is later than or equal to the second online / offline time in the Hbase, no processing is performed.

6. An active data processing system for an Internet of Things device, characterized in that: include: The first module is used to obtain a device-reported message and push the device-reported message to a distributed message queue; wherein the device-reported message includes a message type; The second module is configured to construct a data stream of the device-reported message in the message queue based on a stream computing service; A third module is configured to clean the data stream according to the message type to obtain the data stream containing only active data of the device; A fourth module is configured to add, modify, or not process the device active data in the cache according to the cleaned data stream; A fifth module is configured to add, modify, or not process the device active data in Hbase according to the device active data in the cache; A sixth module is configured to perform batch processing on the device activity data in the Hbase based on a batch processing service, obtain basic activity statistics data, and store the basic activity statistics data in a database; The sixth module is specifically used for: Based on the batch processing service, batch obtain the device active data in the Hbase; Batch parsing the device activity data to obtain user fields, product fields, and device fields in the device activity data; Assembling the parsed device activity data into an online and offline data sequence, wherein the online and offline data sequence is composed of multiple unit data; the unit data includes device operations and operation time; Starting from the last unit data in the online and offline data sequence and recursively forward, judging the activity of the device within the specified time period based on the unit data, including: when the operation time is within the specified time period, judging that the device is active within the specified time period; when the operation time is later than the end time of the specified time period, re-judging by taking the previous unit data; when the operation time is earlier than or equal to the start time of the specified time period and the device operation is online, judging that the device is active within the specified time period; when the operation time is earlier than or equal to the start time of the specified time period and the device operation is offline, judging that the device is inactive within the specified time period; The activity situation is aggregated and calculated based on the user field, the product field, the device field and the specified time period to obtain the activity basic statistical data, and the activity basic statistical data is stored in the database.

7. A device, characterized in that include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the active data processing method for the Internet of Things device according to any one of claims 1 to 5.

8. A computer storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to implement the active data processing method of the Internet of Things device as described in any one of claims 1 to 5 when executed by the processor.

Images