A heartbeat data processing method and device, a cluster, and a medium

Abstract

The application discloses a heartbeat data processing method, comprising: receiving a heartbeat data sequence reported by an Internet of Things (IoT) device, wherein the heartbeat data sequence comprises a plurality of first heartbeat data; merging the first heartbeat data satisfying a preset condition in the heartbeat data sequence by using a Flink state function to obtain second heartbeat data; and storing the second heartbeat data, so as to facilitate data analysis based on the second heartbeat data. The method can effectively reduce the size of data, reduce the storage order of data, and further reduce data storage pressure and load by fully utilizing the Flink state function to merge the first heartbeat data satisfying the preset condition in the heartbeat data sequence, so that the first heartbeat data satisfying the first preset condition can share a data header.

Description

Technical Field

[0001] This invention relates to the field of Internet of Things (IoT) technology, and in particular to a heartbeat data processing method, apparatus, cluster, and medium. Background Technology

[0002] With the development of computer technology and the increasing demands of users, connecting various devices to the Internet to form the Internet of Things (IoT) has become one of the mainstream trends. IoT, or "Internet of Everything," is an extension and expansion of the Internet. It is a huge network formed by combining various information sensing devices with the Internet, enabling interconnection and interoperability between people, machines, and things anytime, anywhere.

[0003] To achieve interconnectivity, IoT devices (hereinafter referred to as IoT devices), such as lights, refrigerators, and air conditioners, typically need to periodically send heartbeat data to the IoT engine. The IoT engine stores this heartbeat data and uses it to confirm that the IoT device is still online.

[0004] Currently, IoT engines typically store heartbeat data in time-series databases or in the Hadoop Distributed File System (HDFS). With the rapid increase in data volume, storage pressure continues to rise.

[0005] Assuming each IoT device reports a heartbeat every minute, the daily heartbeat data volume per device would be 1440 records. If the IoT network includes 10 million devices, the daily heartbeat data volume would reach 14.4 billion records, occupying approximately 4 terabytes (TB) of storage space. If the heartbeat data is stored for 30 days, 120TB of storage space would be required. Furthermore, to ensure data security, distributed systems such as time-series databases and HDFS typically employ a triple-backup storage method, thus requiring a total of 360TB of storage space. Therefore, the extraction, analysis, and use of heartbeat data place a very high load on the system.

[0006] The industry urgently needs a method for processing heartbeat data to achieve efficient storage of heartbeat data and reduce storage pressure and load. Summary of the Invention

[0007] The purpose of this application is to provide a heartbeat data processing method, apparatus, computer cluster, computer-readable storage medium, and computer program product that can process heartbeat data to achieve efficient storage of heartbeat data, thereby reducing storage pressure and load.

[0008] Firstly, this application provides a method for processing heartbeat data, including:

[0009] Receive a heartbeat data sequence reported by an Internet of Things (IoT) device, wherein the heartbeat data sequence includes multiple first heartbeat data;

[0010] The first heartbeat data that meets the preset conditions in the heartbeat data sequence is merged using Flink state functions to obtain the second heartbeat data;

[0011] The second heartbeat data is stored to facilitate data analysis based on the second heartbeat data.

[0012] In some possible implementations, the step of merging the first heartbeat data that meets preset conditions in the heartbeat data sequence using Flink state functions to obtain the second heartbeat data includes:

[0013] The first heartbeat data reported in N consecutive cycles in the heartbeat data sequence is merged using Flink state functions to obtain the second heartbeat data.

[0014] In some possible implementations, the method further includes:

[0015] The characters used to represent the state in the merged heartbeat data are modified to binary numbers, and the second heartbeat data includes the binary numbers.

[0016] In some possible implementations, the method further includes:

[0017] The second heartbeat data is analyzed using a Bloom filter or a bitmap algorithm.

[0018] In some possible implementations, the step of merging the first heartbeat data that meets preset conditions in the heartbeat data sequence using Flink state functions to obtain the second heartbeat data includes:

[0019] The state function is used to group multiple first heartbeat data in the heartbeat data sequence according to the device identifier to obtain at least one group of first heartbeat data, and each group of first heartbeat data includes the same device identifier.

[0020] The first heartbeat data that meet the preset conditions from the at least one set of first heartbeat data are merged to obtain the second heartbeat data.

[0021] In some possible implementations, the method further includes:

[0022] The second heartbeat data is assigned to the corresponding operator subtask according to the device identifier, and each operator subtask is used to process the second heartbeat data of one or more IoT devices;

[0023] When the number of operator subtasks changes, the distribution of the second heartbeat data on the operator subtasks is updated.

[0024] In some possible implementations, updating the distribution of the second heartbeat data on the operator subtask includes:

[0025] Based on the offset of the message queue, the second heartbeat data is re-consumed to update the distribution of the second heartbeat data on the operator subtask.

[0026] Secondly, this application provides a heartbeat data processing apparatus, the apparatus comprising:

[0027] The communication module is used to receive heartbeat data sequences reported by IoT devices, wherein the heartbeat data sequences include multiple first heartbeat data;

[0028] The merging module is used to merge the first heartbeat data that meets the preset conditions in the heartbeat data sequence using Flink state functions to obtain the second heartbeat data.

[0029] A storage module is used to store the second heartbeat data so as to facilitate data analysis based on the second heartbeat data.

[0030] In some possible implementations, the merging module is specifically used for:

[0031] The first heartbeat data reported in N consecutive cycles in the heartbeat data sequence is merged using Flink state functions to obtain the second heartbeat data, where N is greater than 1.

[0032] In some possible implementations, the merging module is also used for:

[0033] The characters used to represent the state in the merged heartbeat data are modified to binary numbers, and the second heartbeat data includes the binary numbers.

[0034] In some possible implementations, the device further includes:

[0035] The analysis module is used to perform data analysis on the second heartbeat data using a Bloom filter or a bitmap algorithm.

[0036] In some possible implementations, the merging module is specifically used for:

[0037] The state function is used to group multiple first heartbeat data in the heartbeat data sequence according to the device identifier to obtain at least one group of first heartbeat data, and each group of first heartbeat data includes the same device identifier.

[0038] The first heartbeat data that meet the preset conditions from the at least one set of first heartbeat data are merged to obtain the second heartbeat data.

[0039] In some possible implementations, the device further includes:

[0040] The allocation module is used to allocate the second heartbeat data to the corresponding operator subtasks according to the device identifier. Each operator subtask is used to process the second heartbeat data of one or more IoT devices.

[0041] An update module is used to update the distribution of the second heartbeat data on the operator subtasks when the number of operator subtasks changes.

[0042] In some possible implementations, the update module is specifically used for:

[0043] Based on the offset of the message queue, the second heartbeat data is re-consumed to update the distribution of the second heartbeat data on the operator subtask.

[0044] Thirdly, this application provides a computer cluster, the computer cluster including at least one computer, the at least one computer including at least one processor and at least one memory;

[0045] The at least one processor is configured to execute instructions stored in the at least one memory to cause the computer cluster to perform the heartbeat data processing method as described in any one of the first aspects.

[0046] Fourthly, this application provides a computer-readable storage medium for storing a computer program for performing the heartbeat data processing method described in any one of the first aspects.

[0047] Fifthly, this application provides a computer program product comprising instructions. The computer cluster executes the instructions to perform the heartbeat data processing method described in any one of the first aspects.

[0048] Compared with the prior art, the advantages of the above-mentioned technical solution of this application are as follows:

[0049] This application provides a heartbeat data processing method. The method receives a heartbeat data sequence reported by an Internet of Things (IoT) device. The heartbeat data sequence includes multiple first heartbeat data. Then, the first heartbeat data in the heartbeat data sequence that meet preset conditions are merged using a Flink state function to obtain second heartbeat data. The second heartbeat data is then stored to facilitate data analysis based on the second heartbeat data.

[0050] This method is based on Flink, a unified computing framework combining batch processing and stream processing. It fully utilizes Flink's state functions to merge the first heartbeat data in the heartbeat data sequence that meets preset conditions. This allows the first heartbeat data meeting the first preset condition to share a data header, effectively reducing the data size and storage requirements, thereby reducing data storage pressure and load. Furthermore, when data analysis or querying is needed, data can be efficiently extracted for analysis or querying, improving the efficiency of subsequent analysis and querying, and meeting the latency requirements of business needs for heartbeat data. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 A system architecture diagram of a heartbeat data processing method provided in this application embodiment;

[0053] Figure 2 A flowchart illustrating a heartbeat data processing method provided in this application embodiment;

[0054] Figure 3 This is a schematic flowchart of a heartbeat data processing method provided in an embodiment of this application.

[0055] Figure 4 A schematic diagram illustrating a stateful computation process for an operator subtask provided in an embodiment of this application;

[0056] Figure 5 A schematic diagram illustrating a stateful computation process for an operator subtask provided in an embodiment of this application;

[0057] Figure 6 A schematic diagram illustrating the process of updating the status of an operator subtask, provided in an embodiment of this application;

[0058] Figure 7 This is a schematic diagram of the structure of a heartbeat data processing device provided in an embodiment of this application;

[0059] Figure 8 This is a schematic diagram of a computer cluster provided in an embodiment of this application. Detailed Implementation

[0060] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0061] The terms "first" and "second" used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include one or more of that feature.

[0062] The technical terms used in this application will be introduced below.

[0063] The Internet of Things (IoT), or "Internet of Everything," is a network that extends and expands upon the internet. It's a vast network that combines various information sensing devices with the internet, enabling interconnection and communication between people, machines, and things anytime, anywhere. When information sensing devices are connected to the internet to form an IoT network, they can also be called IoT devices.

[0064] A heartbeat mechanism is a mechanism for verifying the validity of devices. Specifically in the IoT field, a heartbeat mechanism can involve IoT devices (such as lights, refrigerators, and air conditioners) periodically sending heartbeat data (also called heartbeat packets) to the IoT engine. The IoT engine stores this heartbeat data and uses it to confirm the validity of the IoT device, that is, to confirm whether the IoT device is still online.

[0065] Flink is a unified computing framework that combines batch and stream processing. Its core is a stream data processing engine that provides data distribution and concurrent computation. Similar to Storm, Flink is an event-driven real-time streaming system. Flink is suitable for low-latency data processing scenarios, such as internet finance, clickstream log processing, and public opinion monitoring.

[0066] The core of Flink is the stateful function. A stateful function is a function that stores state. See also... Figure 1 The diagram shown illustrates the execution flow of Flink. Figure 1As shown, each state function f(a,b) in Flink's execution flow has a histogram (cylinder), indicating that each state function in Flink has its own independent storage, which is used to store the state. Unlike other computing engines such as Spark Streaming and Storm, Flink-based applications do not rely on external storage such as Redis or MySQL when performing stateful computations; instead, they directly utilize Flink's built-in state storage for processing.

[0067] Flink state functions are similar to Akka actors. A state function is a small piece of code that runs on each instance of the cluster. State functions do not need to be manually invoked by the user; instead, they are triggered by messages. Flink state functions have the following characteristics:

[0068] 1. In a state

[0069] Each Flink state function has its own fault-tolerant state store, and provides a way to access the Flink state function as if it were a local variable.

[0070] 2. Virtual

[0071] Flink state functions do not retain computational resources. In other words, inactive or unexecuted Flink state functions typically do not consume resources such as the central processing unit (CPU) and memory.

[0072] Currently, IoT engines typically store heartbeat data in time-series databases or in the Hadoop Distributed File System (Hadoop Distributed File System). Taking a Hive data warehouse storing heartbeat data in the Hadoop Distributed File System as an example, assuming a single IoT device reports heartbeat data once per minute, the daily heartbeat data volume per IoT device would be 1440 records. If the IoT includes 10 million IoT devices, the daily uploaded heartbeat data volume would reach 14.4 billion records, occupying approximately 4TB of storage space. If the heartbeat data is stored for 30 days, it would require 120TB of storage space. Furthermore, to ensure data security, Hive typically uses a 3-backup storage method, so a total of 360TB of storage space is required. Thus, a large amount of storage space is needed, resulting in significant storage pressure.

[0073] Moreover, each downstream data processing link requires middleware (such as Flume or Logstash) to perform a full subscription to the Kafka queue. Assuming that the current data throughput of the business scenario is 16 million messages / minute, the message frequency that needs to be subscribed to each time under this architecture is: 3 * 16 million messages / minute = 48 million messages / minute, resulting in a 3-fold increase in Kafka load.

[0074] In view of this, embodiments of this application provide a heartbeat data processing method. This method can be executed by an IoT engine. Specifically, the IoT engine receives a heartbeat data sequence reported by an IoT device. The heartbeat data sequence includes multiple first heartbeat data points. Then, the IoT engine uses a Flink state function to merge the first heartbeat data points in the heartbeat data sequence that meet preset conditions to obtain second heartbeat data. The IoT engine then stores the second heartbeat data to facilitate data analysis based on the second heartbeat data.

[0075] In this method, the IoT engine leverages Flink, a unified computing framework combining batch processing and stream processing. It fully utilizes Flink state functions to merge the first heartbeat data in the heartbeat data sequence that meets preset conditions. This allows the first heartbeat data meeting the first preset condition to share a common data header, effectively reducing the data size and storage requirements, thereby alleviating data storage pressure and load. Furthermore, when data analysis or querying is needed, data can be efficiently extracted for analysis or querying, improving the efficiency of subsequent analysis and querying and meeting the business's latency requirements for heartbeat data.

[0076] To facilitate understanding, the application scenarios of the embodiments of this application will be described below with reference to the accompanying drawings.

[0077] See Figure 1 The diagram shows the system architecture of the heartbeat data processing method. This system 100 includes an IoT engine 10 and multiple IoT devices 20. The IoT devices 20 are internet-connected information sensing devices, such as televisions, air conditioners, and refrigerators.

[0078] In some embodiments, the IoT engine 10 may be software, which may be deployed in a computer cluster. The computer cluster executes the program code of the software to perform the heartbeat data processing method of the embodiments of this application. In other embodiments, the IoT engine 10 may be hardware, which, when running, executes the heartbeat data processing method of the embodiments of this application.

[0079] Specifically, the IoT engine 10 establishes communication connections with multiple IoT devices 20. The IoT engine 10 receives heartbeat data sequences reported by the IoT devices 20. These heartbeat data sequences include multiple first heartbeat data points. The IoT engine 10 then uses Flink state functions to merge the first heartbeat data points in the heartbeat data sequence that meet preset conditions to obtain second heartbeat data. The IoT engine 10 then stores the second heartbeat data to facilitate data analysis based on the second heartbeat data.

[0080] The IoT Engine 10 has been optimized for high-frequency access and inefficient storage of heartbeat data. By using Flink state functions, the accessed heartbeat data is pre-transformed, which greatly reduces the amount of data stored and improves the efficiency of subsequent analysis and query.

[0081] Next, from the perspective of IoT engine 10, the heartbeat data processing method of this application embodiment will be described in detail.

[0082] See Figure 2 The flowchart shown illustrates a heartbeat data processing method, which includes:

[0083] S202: IoT engine 10 receives heartbeat data sequences reported by IoT device 20.

[0084] The heartbeat data sequence includes multiple first heartbeat data points. These multiple first heartbeat data points can originate from the same IoT device 20 or from different IoT devices 20. For example, when only one IoT device 20 is online in the IoT, or when different IoT devices 20 are online at different times, the multiple first heartbeat data points in the heartbeat data sequence can originate from the same IoT device 20. When multiple IoT devices 20 are online simultaneously in the IoT, the multiple first heartbeat data points in the heartbeat data sequence can originate from different IoT devices 20.

[0085] In this process, IoT devices 20 typically report their first heartbeat data periodically. For example, some IoT devices 20 may report their data every minute. Based on this, the IoT engine 10 can receive the first heartbeat data reported by each IoT device 20 according to the aforementioned period, thereby obtaining a heartbeat data sequence.

[0086] S204: The IoT engine 10 uses Flink state functions to merge the first heartbeat data that meets the preset conditions in the heartbeat data sequence to obtain the second heartbeat data.

[0087] The IoT engine 10 employs the Flink framework, which allows it to process heartbeat data sequences using Flink state functions. See also... Figure 3The diagram illustrates the heartbeat data processing flow. After IoT device 20 collects the first heartbeat data, it sends the first heartbeat data to the Kafka message system for caching via middleware such as Flume. The Flink framework in IoT engine 10 consumes the aforementioned first heartbeat data by subscribing to the Kafka message queue. Specifically, IoT engine 10 can utilize Flink state functions to merge the first heartbeat data reported in N consecutive periods from the heartbeat data sequence to obtain the second heartbeat data.

[0088] The first heartbeat data that meets the preset conditions can be the first heartbeat data reported in N consecutive cycles, where N is greater than 1. The value of N can be set according to experience, for example, it can be 5. Based on this, in some examples, the IoT engine 10 can use Flink state functions to merge the first heartbeat data reported every 5 minutes to obtain the second heartbeat data.

[0089] It should be noted that the Flink state function is responsible for maintaining a unique identifier for each IoT device 20, which can be called a device ID. Based on this, the IoT engine 10 can use the Flink state function to group multiple first heartbeat data in the heartbeat data sequence according to the device ID, obtaining at least one group of first heartbeat data, each group of first heartbeat data including the same device ID. Then, the IoT engine 10 merges the first heartbeat data that meet preset conditions from the at least one group of first heartbeat data to obtain the second heartbeat data.

[0090] Furthermore, the IoT engine 10 can also modify the characters used to represent the status in the merged heartbeat data to binary numbers. Accordingly, the second heartbeat data includes the binary numbers. For example, the IoT engine 10 can maintain heartbeat records in memory. During the reporting period, when the IoT device 20 reports first heartbeat data, that is, when the IoT device 20 is online, the IoT engine 10 can assign the value of the status field to 1. When the IoT device 20 does not report first heartbeat data, that is, when the IoT device 20 is offline, the IoT engine 10 can assign the value of the status field to 0.

[0091] The IoT engine 10 can configure strategies via triggers. Specifically, within a set timeout period, newly arrived heartbeat data and device identifiers without heartbeats can be sent to the ClickHouse Sink module. The ClickHouse Sink module performs parallel processing on the above data and saves it to the ClickHouse heartbeat record table as a wide table.

[0092] The IoT Engine 10 converts the state from JavaScript Object Notation (JSON) format to a numerical form of 0s and 1s. This saves storage space and facilitates subsequent analysis models to use more efficient algorithms for data analysis, thus improving analysis efficiency. Furthermore, using 0s and 1s to represent the state also facilitates data retrieval, improving query efficiency.

[0093] S206: The IoT engine 10 stores the second heartbeat data to facilitate data analysis based on the second heartbeat data.

[0094] Specifically, the IoT engine 10 stores the second heartbeat data in memory. When performing stateful computations, the second heartbeat data can be retrieved directly from memory, thus improving computational efficiency.

[0095] The IoT engine 10 stores the second heartbeat data in memory in the form of a wide table. A wide table, literally, is a database table with many fields; it typically refers to a database table that links together metrics, dimensions, and attributes related to a business theme. In this embodiment, the wide table can be a database table that stores the status of multiple periods together.

[0096] In some possible implementations, the IoT engine 10 can also analyze the stored second heartbeat data. When the state in the second heartbeat data is represented by binary numbers, the IoT engine 10 can perform data analysis on the second heartbeat data using a Bloom filter or a bitmap algorithm. This reduces the number of data transformation steps. The Bloom filter has a constant complexity of O(k) in terms of storage space and insertion / query time, which significantly improves the analysis speed. Through these more efficient analysis algorithms, the IoT engine 10 improves analysis efficiency.

[0097] Based on the above description, this application provides a heartbeat data processing method. In this method, the IoT engine 10, based on the Flink framework, fully utilizes Flink state functions to merge the first heartbeat data in the heartbeat data sequence that meets preset conditions. This allows the first heartbeat data meeting the first preset conditions to share a data header, effectively reducing the data size and storage requirements, thereby reducing data storage pressure and load. Furthermore, when data analysis or querying is needed, the IoT engine 10 can efficiently extract data for analysis or querying, improving the efficiency of subsequent analysis and querying, and meeting the latency requirements of business operations for heartbeat data.

[0098] In some application scenarios, data applications can generate at least one operator subtask (or simply subtask). For example... Figure 4As shown, a subtask can receive an input stream, retrieve the state from the state storage unique to each state function, perform stateful computation based on the state, obtain the output result, and update the state.

[0099] In particular, the generation of heartbeat data typically needs to be real-time, with minimal latency. Flink IoT Engine 10's high throughput, high efficiency, and exact-once consumption semantics ensure that data is not lost or duplicated, and is calculated exactly once, thus guaranteeing data consistency, reliability, and real-time performance. Especially when state data is very large or when data applications experience failures requiring recovery, Flink's checkpointing mechanism ensures that state calculations are error-free, thereby improving reliability.

[0100] In this embodiment, the IoT engine 10 can distribute the second heartbeat data to corresponding operator subtasks according to device identifiers. Each operator subtask is used to process the second heartbeat data of one or more IoT devices 20. When the number of operator subtasks changes, the IoT engine 10 updates the distribution of the second heartbeat data on the operator subtasks.

[0101] See Figure 5 The diagram illustrates stateful computation of operator subtasks. In this embodiment, the input stream is grouped by device ID as the key, forming a KeyedStream. The Keyed State is the state on the KeyedStream. All data with device ID 1 in the input stream can share a single state, and so on, each key corresponds to its own state. This state is typically recorded in the second heartbeat data. Figure 5 In the example, an operator subtask can process the second heartbeat data corresponding to one or more keys (or IoT devices 20). For example, operator subtask 1 processes the second heartbeat data corresponding to key1 and key2, and operator subtask 2 processes the second heartbeat data corresponding to key3.

[0102] In some possible implementations, horizontal scaling of the state is also possible. Horizontal scaling refers to modifying the parallelism of Flink-based data applications. Specifically, when the number of parallel instances or operator subtasks changes, some operator subtasks can be stopped or started. See also Figure 6 The diagram illustrates the update status of operator subtasks. A set of state data from a previous operator subtask (recorded in the second heartbeat data) can be updated to a new operator subtask, thus facilitating scaling up or down. Specifically, the aforementioned state data can be updated to the new operator subtask using a balancing method.

[0103] Specifically, the IoT engine 10 can re-consume the second heartbeat data based on the offset of the message queue, such as the offset of the Kafka message queue, to update the distribution of the second heartbeat data on the operator subtask.

[0104] Assuming IoT device 20 reports heartbeat data every minute, each IoT device 20 would report 1440 heartbeat data points per day. If there are 10 million IoT devices 20, the daily heartbeat data volume would reach 14.4 billion, occupying approximately 4TB of storage space. If stored for 30 days, this would require 120TB. Using 3-backup storage, this would require 120 * 3 = 360TB. After processing with Flink state functions, the occupied storage space can be reduced to 1 / 10 of the original, thus significantly saving storage space and meeting business needs.

[0105] Based on the heartbeat data processing method provided in the embodiments of this application, this application also provides a heartbeat data processing device. This heartbeat data processing device may be the IoT engine 10 in the above embodiments, and this device is used to implement the function of heartbeat data processing.

[0106] See Figure 7 The schematic diagram shown illustrates the structure of a heartbeat data processing device 700, which includes:

[0107] Communication module 702 is used to receive heartbeat data sequences reported by IoT devices, wherein the heartbeat data sequences include multiple first heartbeat data;

[0108] The merging module 704 is used to merge the first heartbeat data that meets the preset conditions in the heartbeat data sequence using Flink state functions to obtain the second heartbeat data.

[0109] Storage module 706 is used to store the second heartbeat data so as to facilitate data analysis based on the second heartbeat data.

[0110] In some possible implementations, the merging module 704 is specifically used for:

[0111] The first heartbeat data reported in N consecutive cycles in the heartbeat data sequence is merged using Flink state functions to obtain the second heartbeat data.

[0112] In some possible implementations, the merging module 704 is further configured to:

[0113] The characters used to represent the state in the merged heartbeat data are modified to binary numbers, and the second heartbeat data includes the binary numbers.

[0114] In some possible implementations, the device 700 further includes:

[0115] The analysis module is used to perform data analysis on the second heartbeat data using a Bloom filter or a bitmap algorithm.

[0116] In some possible implementations, the merging module 704 is specifically used for:

[0117] The state function is used to group multiple first heartbeat data in the heartbeat data sequence according to the device identifier to obtain at least one group of first heartbeat data, and each group of first heartbeat data includes the same device identifier.

[0118] The first heartbeat data that meet the preset conditions from the at least one set of first heartbeat data are merged to obtain the second heartbeat data.

[0119] In some possible implementations, the device 700 further includes:

[0120] The allocation module is used to allocate the second heartbeat data to the corresponding operator subtasks according to the device identifier. Each operator subtask is used to process the second heartbeat data of one or more IoT devices.

[0121] An update module is used to update the distribution of the second heartbeat data on the operator subtasks when the number of operator subtasks changes.

[0122] In some possible implementations, the update module is specifically used for:

[0123] Based on the offset of the message queue, the second heartbeat data is re-consumed to update the distribution of the second heartbeat data on the operator subtask.

[0124] The heartbeat data processing apparatus 700 according to the embodiments of this application can correspondingly execute the methods described in the embodiments of this application, and the above and other operations and / or functions of each module / unit of the heartbeat data processing apparatus 700 are respectively for implementing Figure 2 For the sake of brevity, the corresponding processes of each method in the illustrated embodiments will not be described in detail here.

[0125] This application also provides a computer cluster. This computer cluster is used to implement, for example... Figure 7 The function of the heartbeat data processing device 700 shown. See also Figure 8 The schematic diagram of the computer cluster shown shows that the computer cluster 80 includes at least one computer 800, and the at least one computer 800 includes at least one processor 801 and at least one memory 802.

[0126] In this configuration, at least one processor 801 and at least one memory 802 can be connected via a bus. The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be categorized as an address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus is represented by only one line, but this does not mean that there is only one bus or one type of bus.

[0127] The processor 801 can be any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).

[0128] Memory 802 may include volatile memory, such as random access memory (RAM). Memory 802 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0129] The at least one memory 802 may store instructions that are computer-readable, and at least one processor 801 is used to execute the instructions stored in the at least one memory 802 to cause the computer cluster to perform the heartbeat data processing method described in any of the above embodiments.

[0130] Specifically, in achieving Figure 7 In the case of the heartbeat data processing device 700 shown in the embodiment, and Figure 7 The heartbeat data processing device 700 described herein, including modules such as communication module 702 and merging module 704, performs the following functions when implemented in software: Figure 2 The software or program code required for the functions of each module can be stored in at least one memory 1004 in the computing device cluster 10. At least one processor 1002 executes the program code stored in the memory 1004 to cause the computing device cluster 10 to perform the aforementioned video analysis method.

[0131] This application also provides a computer-readable storage medium for storing a computer program for executing the heartbeat data processing method described in any of the above embodiments.

[0132] This application also provides a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on the computer cluster 80, all or part of the processes or functions described in this application are generated. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computing device, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer program product may be a software installation package, which can be downloaded and executed on the computer cluster 80 when any of the aforementioned heartbeat data processing methods are required.

[0133] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The units and modules described as separate components may or may not be physically separate. Furthermore, some or all of the units and modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0134] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0135] The above description is only a specific embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for processing heartbeat data, characterized in that, Applications in IoT engines include: Receive a heartbeat data sequence reported by an Internet of Things (IoT) device, wherein the heartbeat data sequence includes multiple first heartbeat data; The first heartbeat data that meets the preset conditions in the heartbeat data sequence is merged using Flink state functions to obtain the second heartbeat data; The second heartbeat data is stored to facilitate data analysis based on the second heartbeat data; The process of merging the first heartbeat data that meets preset conditions in the heartbeat data sequence using Flink state functions to obtain the second heartbeat data includes: The state function is used to group multiple first heartbeat data in the heartbeat data sequence according to the device identifier to obtain at least one group of first heartbeat data, and each group of first heartbeat data includes the same device identifier. The first heartbeat data reported in N consecutive cycles from the at least one set of first heartbeat data is merged to obtain the second heartbeat data, where N is greater than 1; The method further includes: The second heartbeat data is assigned to the corresponding operator subtask according to the device identifier. The operator subtask is used to process the second heartbeat data of one or more IoT devices. When the number of operator subtasks changes, the second heartbeat data is re-consumed according to the message queue offset to update the distribution of the second heartbeat data on the operator subtasks.

2. The method according to claim 1, characterized in that, The method further includes: The characters used to represent the state in the merged heartbeat data are modified to binary numbers, and the second heartbeat data includes the binary numbers.

3. The method according to claim 2, characterized in that, The method further includes: The second heartbeat data is analyzed using a Bloom filter or a bitmap algorithm.

4. A heartbeat data processing device, characterized in that, The device is deployed within the Internet of Things (IoT) engine, and the device includes: The communication module is used to receive heartbeat data sequences reported by IoT devices, wherein the heartbeat data sequences include multiple first heartbeat data; The merging module is used to merge the first heartbeat data that meets the preset conditions in the heartbeat data sequence using Flink state functions to obtain the second heartbeat data. A storage module is used to store the second heartbeat data so as to facilitate data analysis based on the second heartbeat data; The merging module is specifically used for: The state function is used to group multiple first heartbeat data in the heartbeat data sequence according to the device identifier to obtain at least one group of first heartbeat data, and each group of first heartbeat data includes the same device identifier. The first heartbeat data reported in N consecutive cycles from the at least one set of first heartbeat data is merged to obtain the second heartbeat data. The device further includes: The allocation module is used to allocate the second heartbeat data to the corresponding operator subtask according to the device identifier. The operator subtask is used to process the second heartbeat data of one or more IoT devices. An update module is used to re-consume the second heartbeat data according to the offset of the message queue when the number of operator subtasks changes, so as to update the distribution of the second heartbeat data on the operator subtasks.

5. A computer cluster, characterized in that, The computer cluster includes at least one computer, and the at least one computer includes at least one processor and at least one memory; The at least one processor is configured to execute instructions stored in the at least one memory to cause the computer cluster to perform the heartbeat data processing method as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for performing the heartbeat data processing method according to any one of claims 1 to 3.

Images