An Internet of Things data query method, device and medium

Abstract

The application discloses an Internet of Things data query method, device and medium, the method comprising: extracting Internet of Things data of an Internet of Things device; clustering the Internet of Things data to divide the Internet of Things data into multiple data classification groups; matching each data classification group with multiple data shards set in advance, and storing each data classification group in a data shard of the same data type; one data shard stores one data type; generating a binding relationship between each data classification group and the data shard of the same data type; receiving a query request for the Internet of Things data, analyzing the query request to obtain Internet of Things data to be queried; determining a data classification group to which the Internet of Things data to be queried belongs according to the data type of the Internet of Things data to be queried; obtaining a data shard bound to the data classification group according to the binding relationship, and extracting the Internet of Things data to be queried from the bound data shard. The method improves the Internet of Things data query efficiency.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a method, device and medium for Internet of Things (IoT) data query. Background Technology

[0002] With the gradual popularization of the Industrial Internet ecosystem, more and more sensors need to connect to the network to transmit data and communicate with each other, which is commonly referred to as the Internet of Things (IoT). Due to production needs, sensors need to collect and transmit data on a minute or even second basis, generating a large amount of data in a short period of time. How to store this data has become a problem.

[0003] Currently, while structured or unstructured storage is commonly used, single-machine failures can occur. Distributed storage requires data to be stored in a dispersed manner, and the result of this dispersed processing will directly affect the balance of data distribution. For example, excessively widespread data distribution will lead to low query efficiency and slow querying of summary processing (e.g., data reports). Excessively concentrated data distribution will cause data skew, resulting in concentrated query pressure and potentially triggering single-point failures that render the cluster unavailable. In summary, IoT data query efficiency and performance are low. Summary of the Invention

[0004] This application provides an IoT data query method, device, and medium to address the issues of low efficiency and performance in IoT data querying.

[0005] The embodiments of this application adopt the following technical solutions:

[0006] On one hand, this application provides an IoT data query method, which includes: extracting IoT data from IoT devices within a preset period; dividing the IoT data into multiple data classification groups by performing clustering processing on the IoT data; each data classification group including multiple IoT data of the same data type; storing each data classification group in a data shard of the same data type by matching each data classification group with multiple pre-defined data shards; storing one data shard for one data type; generating a binding relationship between each data classification group and a data shard of the same data type; receiving a query request for IoT data, parsing the query request to obtain the IoT data to be queried; determining the data classification group to which the IoT data to be queried belongs based on the data type of the IoT data to be queried; obtaining the data shard bound to the data classification group based on the binding relationship, and extracting the IoT data to be queried from the bound data shard.

[0007] In one example, the step of extracting IoT data uploaded by IoT devices within a preset period specifically includes: acquiring raw IoT data uploaded by IoT devices within a preset period; cleaning the raw IoT data to obtain IoT data of the IoT devices; storing the IoT data in a queue to generate a message queue for the IoT devices; and extracting the IoT data from the message queue in batches according to a preset number of clustered data in a single clustering process.

[0008] In one example, the step of clustering the IoT data to divide it into multiple data classification groups specifically includes: acquiring sample IoT data and various data types of the sample IoT data; randomly generating a centroid for the sample IoT data; calculating the distance between each sample IoT data and the centroid using the Manhattan algorithm; dividing the sample IoT data into clusters based on the distance and the various data types, and outputting multiple data classification groups of the sample IoT data to generate a clustering processing model; inputting the IoT data into the clustering processing model, and outputting multiple data classification groups of the IoT data.

[0009] In one example, the step of matching each data category group with multiple pre-defined data shards and storing each data category group in a data shard of the same data type specifically includes: matching each data category group with multiple pre-defined data shards to generate multiple matching combinations; in each matching combination, the data category group and the data shard have the same data type; in each matching combination, storing the data category group in a data shard; determining whether there are any mismatched data category groups; if so, creating a new data shard for the mismatched data category group; and storing the mismatched data category group in the new data shard.

[0010] In one example, the method further includes: determining the absolute value of a first difference between the average read / write clock frequency of the data shard and the operating clock frequency of the current disk; determining whether the absolute value of the first difference is higher than a preset difference threshold; if so, initiating a data shard migration instruction, marking the data shard as being migrated, and pausing all operations in the data shard except for the IoT data read; if the average read / write clock frequency of the data shard is lower than the operating clock frequency of the current disk, obtaining the target disk from a pre-set spare disk combination; the operating clock frequency of the target disk is lower than the operating clock frequency of the current disk where the data shard is located; performing a copy operation on the data shard to the target disk; when the data shard copy is completed, restoring the data shard on the target disk to a normal access state, and deleting the data shard from the current disk.

[0011] In one example, after pausing all operations in the data shard except for the IoT data read, the method further includes: starting from the current time point, obtaining read / write operation instructions for the data shard; generating log data for the data shard by recording the read / write operation instructions; and deleting the data shard from the current disk when the data shard copying is complete, specifically including: merging the log data into the target disk when the data shard copying is complete, so as to perform read / write operations on the data shard on the target disk according to the log data; setting the data shard to an inoperable state on the current disk; and deleting the data shard from the current disk.

[0012] In one example, the method further includes: if the average read / write clock frequency of the data shard is higher than or equal to the operating clock frequency of the current disk, then obtaining a hot disk combination from the disk spare combination; in the hot disk combination, the operating clock frequency of each disk is higher than the operating clock frequency of the current disk; when the number of disks in the hot disk combination is a single disk, then determining the disk in the hot disk combination as the target hot disk for the data shard to be migrated.

[0013] In one example, the method further includes: if the number of disks in the hot disk combination is multiple, calculating a second difference between the average read / write clock frequency of the data shard and the operating clock frequency of each disk; sorting each disk in ascending order of the second difference; if the number of disks is odd, determining the disk corresponding to the middle number as the target hot disk for the data shard to be migrated; if the number of disks is even, randomly selecting one disk from the two middle disks and determining the randomly selected disk as the target hot disk for the data shard to be migrated.

[0014] On the other hand, embodiments of this application provide an IoT data query device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: extract IoT data from IoT devices within a preset period; divide the IoT data into multiple data classification groups by performing clustering processing on the IoT data; each data classification group includes multiple IoT data items of the same data type; store each data classification group in a data shard of the same data type by matching each data classification group with multiple pre-defined data shards; each data shard stores one data type; generate a binding relationship between each data classification group and a data shard of the same data type; receive a query request for IoT data, parse the query request to obtain the IoT data to be queried; determine the data classification group to which the IoT data to be queried belongs based on the data type of the IoT data to be queried; obtain the data shard bound to the data classification group based on the binding relationship, and extract the IoT data to be queried from the bound data shard.

[0015] On the other hand, embodiments of this application provide a non-volatile computer storage medium for querying IoT data, storing computer-executable instructions. The computer-executable instructions are characterized by: extracting IoT data from IoT devices within a preset period; dividing the IoT data into multiple data classification groups by performing clustering processing on the IoT data; each data classification group including multiple IoT data items of the same data type; storing each data classification group in a data shard of the same data type by matching each data classification group with multiple pre-defined data shards; storing one data shard for each data type; generating a binding relationship between each data classification group and a data shard of the same data type; receiving a query request for IoT data, parsing the query request to obtain the IoT data to be queried; determining the data classification group to which the IoT data to be queried belongs based on the data type of the IoT data to be queried; obtaining the data shard bound to the data classification group based on the binding relationship, and extracting the IoT data to be queried from the bound data shard.

[0016] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects:

[0017] It can automatically classify and aggregate IoT data by clustering, placing data of the same data type into data shards of the same data type. During the query process, the query request is guided to the corresponding data shard to extract IoT data, which can improve the response speed of the aggregation processing, improve system performance, and improve the efficiency of IoT data query. Attached Figure Description

[0018] To more clearly illustrate the technical solution of this application, some embodiments of this application will be described in detail below with reference to the accompanying drawings, in which:

[0019] Figure 1 A flowchart illustrating an IoT data query method provided in an embodiment of this application;

[0020] Figure 2 This is a schematic diagram of the framework of an Internet of Things (IoT) data query system provided in an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of the structure of an Internet of Things (IoT) data query device provided in an embodiment of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] Some embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0024] Figure 1 This is a flowchart illustrating an IoT data query method provided in an embodiment of this application. This method can be applied to different business domains, such as internet finance, e-commerce, instant messaging, gaming, and government services. The process can be executed by computing devices in the corresponding domain (e.g., risk control servers or smart mobile terminals for payment services). Certain input parameters or intermediate results in the process can be manually adjusted to help improve accuracy.

[0025] The analysis method involved in the embodiments of this application can be implemented by a terminal device or a server, and this application does not impose any special limitations on it. For ease of understanding and description, the following embodiments are all described in detail using a server as an example.

[0026] It should be noted that the server can be a single device or a system composed of multiple devices, i.e., a distributed server. This application does not make any specific limitations on this.

[0027] Figure 1 The process includes the following steps:

[0028] S101: Extract IoT data from IoT devices within a preset period.

[0029] In some embodiments of this application, a situation is considered where a large amount of IoT data suddenly appears. In this case, since the amount of data for clustering is limited, the clustering process will be untimely or the clustering results will be inaccurate when there is a large amount of data. Therefore, a message queue mode is adopted, which is first-in-first-out, to avoid the data backlog in the clustering module when a large amount of data suddenly floods in. A fixed amount of IoT data is taken out from the message queue each time for clustering processing.

[0030] Specifically, within a preset period, raw IoT data uploaded by IoT devices is acquired.

[0031] Then, the raw IoT data is cleaned to obtain IoT data from IoT devices.

[0032] Data cleaning enables us to check data consistency, handle invalid and missing values, and obtain valid IoT data.

[0033] Then, the IoT data is stored in a queue to generate a message queue for IoT devices.

[0034] Finally, IoT data is extracted from the message queue in batches according to the preset number of clustered data in a single clustering process.

[0035] In summary, by introducing a front-end message queue to enqueue IoT data, system traffic is stabilized, and slow response times caused by sudden large data spikes are avoided.

[0036] S102: By performing clustering processing on the IoT data, the IoT data is divided into multiple data classification groups.

[0037] It should be noted that the K-means algorithm is used to cluster IoT data and continuously learn from the data to ensure that similar data should fall into the same data segment.

[0038] Each data category group includes multiple IoT data points of the same data type. For example, the data type can be temperature data, humidity data, pressure data, etc. Data category group A includes multiple temperature data points, which may come from the same IoT device or from multiple IoT devices. Data category group B includes multiple humidity data points.

[0039] In some embodiments of this application, it is necessary to pre-train the clustering model. The training process is as follows:

[0040] First, acquire sample IoT data and various data types of sample IoT data.

[0041] Then, a centroid is randomly generated from the sample IoT data.

[0042] The distance between each sample of IoT data and the centroid is calculated using the Manhattan algorithm;

[0043] Based on distance and various data types, the sample IoT data is divided into clusters, and multiple data classification groups of the sample IoT data are output to generate a clustering processing model.

[0044] It should be noted that a cluster represents a data classification group, and a data classification group represents a data type.

[0045] In addition, during the training process, IoT data that are close to each other are divided into clusters. The criteria for being close to each other can be set according to actual needs. For example, when there are two types of data, data that are less than 5cm apart are set as one cluster, and data that are greater than 5cm apart and less than 10cm apart are set as another cluster.

[0046] Based on this, IoT data is input into a clustering model, which outputs multiple data classification groups of IoT data.

[0047] In summary, by introducing machine learning and using the K-means algorithm to cluster data, similar data are grouped into the same data shard, ensuring that similar data are in the same shard during queries, thus improving the response speed of summary processing.

[0048] It's important to note that if discrete values ​​are obtained by hashing data IDs, and then certain discrete intervals are categorized and stored, this strategy heavily relies on the degree of data diversity. For example, if a factory centrally procures a batch of temperature sensors, due to the bulk purchase, the unique identifiers (IMEI, MAC, or ID) of these sensors often appear as consecutive sequences (001, 002, 003, etc.), resulting in very low data distinguishability. Even after hash calculation, the data cannot be dispersed, causing data skew. The query and storage pressure is concentrated on a few fragments, leading to slow system operation.

[0049] S103: By matching each data category group with multiple pre-defined data shards, each data category group is stored in a data shard of the same data type.

[0050] In this system, each data shard stores one data type. Each data type has one data shard.

[0051] In some embodiments of this application, there are cases where the data fragment data types are fewer than the IoT data types. For example, the data types of IoT devices may increase, but the management user does not increase the data fragment data types in a timely manner.

[0052] Based on this, each data category group is matched with multiple pre-defined data segments to generate multiple matching combinations. In each matching combination, the data category group and the data segment have the same data type.

[0053] Then, within each matching combination, the data classification group is stored to the data shard.

[0054] Next, we continue to determine whether there are any mismatched data category groups.

[0055] If so, create a new data shard for the mismatched data category group and store the mismatched data category group in the new data shard.

[0056] It should be noted that new data shards are generated by creating corresponding replica structures.

[0057] S104: Generate the binding relationship between each data category group and the data shard of the same data type.

[0058] This involves managing users to pre-configure data shards for multiple data types. For example, a data category group for the temperature data type is bound to a data shard for the temperature data type.

[0059] S105: Receive a query request for IoT data, parse the query request, and obtain the IoT data to be queried.

[0060] S106: Determine the data classification group to which the IoT data to be queried belongs based on the data type of the IoT data to be queried.

[0061] S107: Based on the binding relationship, obtain the data fragments bound to the data classification group, and extract the IoT data to be queried from the bound data fragments.

[0062] In some embodiments of this application, different data types have different query frequencies, while the operating clock frequency of each disk is fixed and varies; disks with higher operating clock frequencies are more expensive. If the read / write clock frequency of the data shard is too high compared to the disk's operating clock frequency, query speed will be slow; if the read / write clock frequency of the data shard is too low compared to the disk's operating clock frequency, disk resources will be wasted. Therefore, the read / write clock frequency of the data shard is monitored periodically, and the data shards are migrated to suitable disks in a timely manner. Each read / write clock frequency represents the number of times a data shard is read and written within 1 second.

[0063] Based on this, the absolute value of the first difference between the average read / write clock frequency of the data shard and the current disk operating clock frequency is determined.

[0064] Specifically, the read and write clock frequencies of the data shards during the current migration cycle are obtained, and the average value of the multiple read and write clock frequencies is calculated to obtain the average read and write clock frequency of the data shards during the current migration cycle.

[0065] Then, it is determined whether the absolute value of the first difference is higher than a preset difference threshold. If not, the data fragmentation migration instruction is not initiated.

[0066] If so, initiate the data shard migration command, mark the data shard as being migrated, and pause all operations in the data shard except for IoT data reading.

[0067] Then, the target disk for the data shard to be migrated is obtained from a pre-defined pool of backup disks. The target disk's clock frequency is lower than that of the current disk containing the data shard.

[0068] It should be noted that if there are multiple target disks, one target disk will be selected randomly.

[0069] Then, the data shards are copied to the target disk.

[0070] Finally, once the data shard copying is complete, restore the data shards on the target disk to a normal access state and delete the data shards from the current disk.

[0071] It should be noted that pausing all operations in data sharding except for reading IoT data is to prevent the loss of newly added IoT data during the migration process.

[0072] Based on this, the user's read and write operation commands for the data shards are recorded. After the migration is complete, the IoT data of the data shards is processed according to the read and write operation commands. That is, it is equivalent to delaying the processing of the user's read and write operation commands.

[0073] Specifically, after pausing all operations in the data sharding process except for IoT data reads, read and write operation instructions for the data shards are retrieved starting from the current time point. Then, log data for the data shards is generated by recording these read and write operation instructions.

[0074] Based on this, when data shard replication is complete, log data is merged into the target disk so that read and write operations on the data shards can be performed on the target disk according to the log data. The data shards are then set to an inoperable state on the current disk and deleted from the current disk to free up space.

[0075] In some embodiments of this application, in order to transfer high-frequency data to a high-performance memory, if the average read / write clock frequency of the data shard is higher than or equal to the current disk's operating clock frequency, a hot disk combination is obtained from the disk spare combination.

[0076] In the hot disk combination, the operating clock frequency of each disk is higher than that of the current disk.

[0077] When the number of disks in a hot disk group is a single disk, the disk in the hot disk group is identified as the target hot disk for the data sharding migration.

[0078] In addition, if there are multiple disks in the hot disk combination, a second difference is calculated between the average read / write clock frequency of the data shards and the operating clock frequency of each disk.

[0079] Sort each disk in ascending order of the second difference.

[0080] If the number of disks is odd, the disk corresponding to the middle number will be identified as the target hot disk for the data shard migration.

[0081] If the number of disks is even, then a disk is randomly selected from the two middle disks, and the randomly selected disk is determined as the target hot disk for the data shard migration.

[0082] It should be noted that for the specific process of migrating data shards to the target hot disk, please refer to the above-mentioned technical solution for migrating data shards to the target disk, which will not be described again here.

[0083] In summary, by introducing an IoT data hot migration mechanism, hot and cold zones can be automatically balanced based on IoT data hotspots without downtime. High-frequency access data can be automatically transferred to high-performance disks without interrupting business operations, thereby improving query speed and saving storage costs.

[0084] It should be noted that, although the embodiments in this application are based on... Figure 1Steps S101 to S107 will be described sequentially, but this does not mean that steps S101 to S107 must be performed in a strict order. The reason this embodiment follows this order is... Figure 1 The order in which steps S101 to S107 are described is provided to facilitate understanding of the technical solutions of the embodiments of this application by those skilled in the art. In other words, in the embodiments of this application, the order of steps S101 to S107 can be appropriately adjusted according to actual needs.

[0085] pass Figure 1 This method enables the automatic categorization and aggregation of IoT data through clustering. Data of the same data type is placed into data shards of the same data type, allowing similar IoT data to be stored in the same data shards. During the query process, the query request is guided to the corresponding data shard to extract IoT data, which can improve the response speed of the aggregation processing, improve system performance, and improve the efficiency of IoT data query.

[0086] Figure 2 This is a schematic diagram of the framework of an Internet of Things (IoT) data query system provided in an embodiment of this application.

[0087] exist Figure 2 The query system includes a message queue module, a data clustering module, a data sharding module, a data migration module, and a core query module.

[0088] To avoid single points of failure, all modules adopt a multi-active mode.

[0089] The message queue module stores the collected IoT sensor data in a message queue and smooths out storage requests to ensure that data does not accumulate.

[0090] The data clustering module uses the K-means clustering algorithm to continuously learn from the data, ensuring that similar IoT data should fall into the same data shard, and then summarizes the results to the core queryer.

[0091] The data sharding module records the actual storage location of data shards, is responsible for actual data access, and reports the range of IoT data stored in the shards to the core queryer. Furthermore, it maintains the underlying storage and access of the data, avoiding single points of failure through a distributed multi-replica mechanism.

[0092] The data migration module automatically balances hot and cold zones based on data hotspots, predicts query requests, and transfers high-frequency data to high-performance storage.

[0093] The core query module is the main entry point for system queries. It maintains all data sharding information internally and directs query requests to the corresponding data sharding module.

[0094] Based on the same idea, some embodiments of this application also provide devices and non-volatile computer storage media corresponding to the above methods.

[0095] Figure 3 A schematic diagram of the structure of an Internet of Things (IoT) data query device provided in this application embodiment includes:

[0096] At least one processor; and,

[0097] A memory communicatively connected to the at least one processor; wherein,

[0098] The memory stores instructions that can be executed by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0099] Extract IoT data from IoT devices within a preset period;

[0100] By performing clustering processing on the IoT data, the IoT data is divided into multiple data classification groups; each data classification group includes multiple IoT data items of the same data type.

[0101] Each data category group is matched with multiple pre-defined data shards, and each data category group is stored in a data shard of the same data type; each data shard stores one data type.

[0102] Generate the binding relationship between each data category group and the data shard of the same data type;

[0103] Receive a query request for IoT data, parse the query request, and obtain the IoT data to be queried;

[0104] Based on the data type of the IoT data to be queried, determine the data classification group to which the IoT data to be queried belongs;

[0105] Based on the binding relationship, the data fragments bound to the data classification group are obtained, and the IoT data to be queried is extracted from the bound data fragments.

[0106] Some embodiments of this application provide a non-volatile computer storage medium for IoT data querying, storing computer-executable instructions, wherein the computer-executable instructions are configured as follows:

[0107] Extract IoT data from IoT devices within a preset period;

[0108] By performing clustering processing on the IoT data, the IoT data is divided into multiple data classification groups; each data classification group includes multiple IoT data items of the same data type.

[0109] Each data category group is matched with multiple pre-defined data shards, and each data category group is stored in a data shard of the same data type; each data shard stores one data type.

[0110] Generate the binding relationship between each data category group and the data shard of the same data type;

[0111] Receive a query request for IoT data, parse the query request, and obtain the IoT data to be queried;

[0112] Based on the data type of the IoT data to be queried, determine the data classification group to which the IoT data to be queried belongs;

[0113] Based on the binding relationship, the data fragments bound to the data classification group are obtained, and the IoT data to be queried is extracted from the bound data fragments.

[0114] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0115] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0116] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0117] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0118] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0119] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0120] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0121] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0122] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0123] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0124] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the technical principles of this application should fall within the protection scope of this application.

Claims

1. A method for querying Internet of Things (IoT) data, characterized in that, The method includes: Extract IoT data from IoT devices within a preset period; By performing clustering processing on the IoT data, the IoT data is divided into multiple data classification groups; each data classification group includes multiple IoT data items of the same data type. Each data category group is matched with multiple pre-defined data shards, and each data category group is stored in a data shard of the same data type; each data shard stores one data type. Generate the binding relationship between each data category group and the data shard of the same data type; Receive a query request for IoT data, parse the query request, and obtain the IoT data to be queried; Based on the data type of the IoT data to be queried, determine the data classification group to which the IoT data to be queried belongs; Based on the binding relationship, the data fragments bound to the data classification group are obtained, and the IoT data to be queried is extracted from the bound data fragments; Periodically monitor the read / write clock frequency of data shards and migrate and store the data shards to an appropriate disk, including: Determine the absolute value of the first difference between the average read / write clock frequency of the data fragment and the current disk operating clock frequency; Determine whether the absolute value of the first difference is higher than a preset difference threshold; If not, the data shard migration command will not be initiated; If so, then initiate a data shard migration command, mark the data shard as being migrated, and pause all operations in the data shard except for the IoT data read; If the average read / write clock frequency of the data shard is lower than the operating clock frequency of the current disk, then the target disk for the data shard to be migrated is obtained from a pre-set backup disk combination; the operating clock frequency of the target disk is lower than the operating clock frequency of the current disk where the data shard is located. The method further includes: If the average read / write clock frequency of the data shard is higher than or equal to the operating clock frequency of the current disk, then a hot disk combination is obtained from the spare disk combination; in the hot disk combination, the operating clock frequency of each disk is higher than the operating clock frequency of the current disk. When the number of disks in the hot disk combination is a single disk, the disk in the hot disk combination is identified as the target hot disk for the data shard to be migrated. The method further includes: If there are multiple disks in the hot disk combination, then calculate the second difference between the average read / write clock frequency of the data shard and the operating clock frequency of each disk; Sort each disk in ascending order of the second difference; If the number of disks is odd, the disk corresponding to the middle number will be identified as the target hot disk for the data shard to be migrated. If the number of disks is even, then a disk is randomly selected from the two middle disks, and the randomly selected disk is determined as the target hot disk for the data shard to be migrated; The data fragments are copied to the target disk. When the data shard copying is complete, the data shards on the target disk are restored to normal access status, and the data shards are deleted from the current disk.

2. The method according to claim 1, characterized in that, The step of extracting IoT data uploaded by IoT devices within a preset period specifically includes: Within a preset period, acquire raw IoT data uploaded by IoT devices; The raw IoT data is cleaned to obtain the IoT data of the IoT device; The IoT data is stored in a queue to generate a message queue for the IoT device. The IoT data is extracted from the message queue in batches according to the preset number of clustered data in a single clustering process.

3. The method according to claim 1, characterized in that, The process of clustering the IoT data to divide it into multiple data classification groups specifically includes: Acquire sample IoT data and various data types of the sample IoT data; A centroid is randomly generated for the sample IoT data; The distance between each sample of IoT data and the centroid is calculated using the Manhattan algorithm; Based on the distance and the various data types, the sample IoT data is divided into clusters, and multiple data classification groups of the sample IoT data are output to generate a clustering processing model. The IoT data is input into the clustering model, which outputs multiple data classification groups of the IoT data.

4. The method according to claim 1, characterized in that, The step of matching each data category group with multiple pre-defined data shards and storing each data category group in a data shard of the same data type specifically includes: Each data category group is matched with multiple pre-defined data segments to generate multiple matching combinations; in each matching combination, the data categories and data segments have the same data type. Within each matching combination, the data category groups are stored in the data shards; Determine if there are mismatched data category groups; If so, create a new data shard for the mismatched data category group; Store mismatched data categories into new data shards.

5. The method according to claim 1, characterized in that, After pausing all operations in the data segment except for the IoT data read, the method further includes: Starting from the current time point, obtain read and write operation instructions for the data fragment; The log data of the data fragments is generated by recording the read and write operation instructions; The step of deleting the data fragment from the current disk after the data fragment copying is completed specifically includes: When the data shard copying is complete, the log data is merged into the target disk so that the data shards can be read and written on the target disk according to the log data. Set the data fragments to an inoperable state on the current disk; The data fragments are deleted from the current disk.

6. An Internet of Things (IoT) data query device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to: perform an Internet of Things data query method as described in any one of claims 1-5.

7. A non-volatile computer storage medium for Internet of Things (IoT) data query, storing computer-executable instructions, characterized in that, The computer-executable instructions are configured to execute the Internet of Things data query method according to any one of claims 1-5.

Images