A method for dynamically expanding the storage capacity of monitoring data of IoT devices

Abstract

The present application discloses a method for dynamically expanding the storage capacity of monitoring data of Internet of Things devices, comprising the following steps: S1, allocating device storage slot space for a data storage server, and reasonably initializing and dividing the device storage slots according to the computing resources and storage resources configured by the database server, and reserving resources for subsequent expansion of the storage slots of newly added devices; S2, initial access of the Internet of Things device; S3, obtaining the storage node with the lightest current load through algorithm comparison and judgment; the technical key points are: supporting the storage and efficient access of massive monitoring data, and being designed with good scalability and elasticity; having high adaptability to various monitoring and collection devices, and greatly reducing the subsequent operation and maintenance costs of the Internet of Things platform; the dynamic online expansion mechanism is flexible, and can support vertical expansion of device storage slots and horizontal expansion of storage server nodes to ensure that the application does not stop, and data transplantation and migration are also relatively convenient.

Description

Technical Field

[0001] The present invention belongs to the field of Internet of Things big data, and specifically is a method for dynamically expanding the storage capacity of Internet of Things equipment monitoring data. Background Art

[0002] At present, the information technology of society is developing rapidly. New technologies such as Internet+ and big data have been widely used in various industries. In the application fields of smart cities, industrial Internet, smart environmental protection, smart manufacturing, smart medical care, smart transportation and other industries, more and more IoT smart terminals are being developed and applied. The terminals are deployed in different application environments and rely on sensors with different purposes such as light, sound, heat, and gas to simulate vision, hearing, touch, and smell to collect all-round industry monitoring data, and apply them to industry segmentation application fields to provide more innovative application scenarios;

[0003] IoT sensing data is collected automatically by smart terminal devices and uploaded to the IoT cloud platform without human intervention. The collected sensing monitoring data has the following main features:

[0004] 1. Automated and continuous data collection,

[0005] IoT sensing terminal devices used in many industry monitoring fields have the characteristics of high-frequency and uninterrupted continuous collection of indicator data. For example, environmental monitoring equipment collects a set of monitoring indicators every 30 seconds and uploads them to the cloud;

[0006] 2. The amount of data collected by the equipment is growing rapidly.

[0007] The high frequency of data collection by IoT devices leads to a rapid growth in data volume. For example, if a single device collects a set of 6 monitoring indicators and uploads them every 30 seconds, the amount of data collected by a single device can reach 17,280 items per day.

[0008] 3. Data has time series,

[0009] The data collected automatically by IoT sensing is not manually reported and has a time-series feature, that is, each collected indicator data has a collection time point, and the order of data collection is clear;

[0010] Sorting out the problems existing in the existing conventional technical solutions in the field of IoT big data applications;

[0011] 1. Using traditional relational databases to store massive monitoring data will result in obvious performance bottlenecks as time goes by, and it is impossible to expand capacity online and the subsequent migration costs are huge;

[0012] 2. The use of big data technology solutions will result in excessive investment in server resources in the early stage, and non-large projects will lead to high investment and maintenance costs, making it difficult to rationalize cost control. Summary of the invention

[0013] 1. Technical issues to be resolved

[0014] In view of the shortcomings of the prior art, the present invention provides a method for dynamically expanding the storage capacity of monitoring data of an IoT device, which solves the problem of dynamically expanding the storage capacity of massive monitoring data and maintaining high access performance.

[0015] (II) Technical solution

[0016] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0017] A method for dynamically expanding the storage capacity of monitoring data of an IoT device comprises the following steps:

[0018] S1. Allocate device storage slot space for the data storage server, initialize and allocate device storage slots reasonably according to the computing resources and storage resources configured on the database server, and reserve resources for subsequent expansion of new device storage slots;

[0019] S2. When an IoT device is connected for the first time, if there are multiple database storage nodes, each database storage node needs to record the latest number of used and available device storage slots after allocating device storage slots for the connected device, and uniformly summarize the available storage idle state data for the new device to bind and allocate device storage slots to obtain the latest device storage slot idle rate of the storage node;

[0020] S3. Obtain the storage node with the lightest current load through algorithm comparison and determination, and allocate device storage slots for the device from this node. After the device is bound to the device storage slot, the storage database and storage physical table of the device are determined.

[0021] Furthermore, in the S1, the process of dividing the device storage slots corresponds to the process of automatically creating the physical database table in physical operation, which is implemented by writing an automated script execution program, and the table is named by slot+sequential number.

[0022] Furthermore, in S1, each device storage slot is used to store data of a single device in a logical sense, and the device storage slot in a physical sense corresponds to a physical table of a database, which has a standardized library name and table name;

[0023] When an IoT device is bound to a device storage slot, device-related data storage and access can be addressed to the corresponding library and table.

[0024] Furthermore, in said S2 and S3, the above steps are a device binding access storage process, and the algorithm based on the whole process is a device routing-binding device storage slot algorithm;

[0025] When obtaining the device storage slot idle rate, the calculation formula of the device storage slot idle rate of a single server is: device storage slot idle rate = number of device storage slots in use / total number of device storage slots.

[0026] Furthermore, in the S3, the summary storage status data and detailed allocation data of each server storage node will be updated and changed after each allocation of a bound device storage slot, and will be uniformly collected and centrally stored for use in quickly calling an algorithm to calculate the device storage slot idle rate of each storage node when accessing the device, thereby determining to allocate the device storage slot space on the storage node with the lightest current load to the device.

[0027] Furthermore, after S3, S4 is also included, which is a device data query process in the application;

[0028] There are several device data queries in IoT applications. When querying single or multiple device data, it is convenient to obtain the target library and table information corresponding to the device storage slot bound to the device, and realize the connection query execution of different databases through the database middleware tool, and return the query results to the application.

[0029] Furthermore, in the S4, the database middleware is used to connect multiple underlying distributed data sources, support multi-threaded parallel query and aggregate query results of multiple devices, and support dynamic update of configuration files to achieve dynamic expansion of data sources.

[0030] (III) Beneficial effects

[0031] First, it supports the storage and efficient access of massive monitoring data, and is designed with good scalability and extensibility. Second, it has high adaptability to various monitoring and collection equipment, greatly reducing the subsequent operation and maintenance costs of the IoT platform. Third, the dynamic online expansion mechanism is flexible and can support vertical expansion of device storage slots and horizontal expansion of storage server nodes to ensure that the application does not stop. In addition, data transplantation and migration are relatively convenient. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is the design diagram of the equipment storage slot in the present invention;

[0033] Figure 2 It is the overall operation diagram of IoT monitoring data storage access in the present invention;

[0034] Figure 3 It is a flow chart of device routing algorithm-binding device slot access in the present invention;

[0035] Figure 4 This is the device routing algorithm-multi-device data access flow chart in the present invention. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In addition, for the convenience of description below, the referenced "upper", "lower", "left", and "right" are the same as the upper, lower, left, and right directions of the drawings themselves. The "first", "second", etc. below are distinguished for the description and have no other special meanings.

[0037] In view of the problems existing in the prior art, the present invention provides a method for dynamically expanding the storage of monitoring data of IoT devices, which can effectively solve the problems in the background technology under the condition of controlling reasonable cost investment; the method of the present invention adopts an innovative way combining algorithms with information technology points to flexibly solve the problems encountered, and is designed to reduce the coupling of device monitoring data storage and monitoring data access application under the premise that the server input resource cost is reasonable and controllable, realize dynamic online expansion of massive monitoring data storage, and provide high-performance query access support capabilities, including the following steps:

[0038] S1. Allocate device storage slot space for the data storage server, initialize and allocate device storage slots reasonably according to the computing resources and storage resources configured on the database server, and reserve resources for subsequent expansion of new device storage slots;

[0039] First, please refer to the attached Figure 1 , divide the device storage slot space for the data storage server, according to the computing resources and storage resources configured by the database server, and evaluate the annual data growth of the equipment used in the project, reasonably initialize and divide the device storage slot, and reserve resources for subsequent expansion of new device storage slots;

[0040] In S1, the process of dividing device storage slots corresponds to the process of automatically creating physical database tables in physical operations. The process is implemented by writing automated script execution programs. The table is named using slot + sequential numbers. In S1, each device storage slot is used to store the data of a single device in a logical concept. The device storage slot in the physical concept corresponds to the physical table of the database, which has a standardized library name and table name. When the IoT device is bound to the device storage slot, the storage and access of device-related data can be addressed to the corresponding library and table.

[0041] Secondly, from the perspective of the complete IoT monitoring data storage access process, please refer to the attached Figure 2 ,From the perspective of IoT functions, there are device binding access storage ,process and device data query process in the application;

[0042] Please refer to the attached Figure 3 It can be obtained that: S2, when the IoT device is connected for the first time, when there are multiple database storage nodes, each database storage node needs to record the latest number of used and available device storage slots after allocating device storage slots for the access device, and uniformly summarize the available storage idle state data for the storage node to obtain the latest device storage slot idle rate when the new device is bound and allocated. In S2 and S3, the above steps are the device binding access storage process, and the algorithm based on the whole process is the device routing-binding device storage slot algorithm. When obtaining the device storage slot idle rate, the calculation formula of the idle rate of the device storage slot of a single server is: device storage slot idle rate = number of device storage slots in use / total number of device storage slots;

[0043] S3. Obtain the storage node with the lightest current load through algorithm comparison and determination, and divide the device storage slot for the device from this node. After the device is bound to the device storage slot, the storage database and storage physical table of the device are determined, providing a basis for subsequent device data access addressing;

[0044] In S3, the summary storage status data and detailed allocation data of each server storage node will be updated and changed after each allocation of a bound device storage slot, and will be centrally stored. When the device is connected, the algorithm is quickly called to calculate the device storage slot idle rate of each storage node, so as to determine the device storage slot space on the storage node with the lightest current load for the device.

[0045] Please refer to the attached Figure 4 It can be obtained that: S4, this step is the device data query process in the application; there are several device data queries in the IoT application. When querying single or multiple device data, it is convenient to obtain the target library and table information corresponding to the device storage slot bound to the device, and realize the connection query execution of different databases through the database middleware tool, and return the query results to the application; in S4, the database middleware is used to connect to multiple underlying distributed data sources, support multi-threaded parallel query and aggregate multiple device query results, and support dynamic update of configuration files to realize dynamic expansion of data sources.

[0046] Combining the above algorithms and application technical points, the invention of the method for dynamic expansion of IoT device monitoring data storage can not only enhance the computing power and storage resource configuration of storage nodes, and vertically increase the number of connected IoT devices by dynamically expanding the storage slots of new devices; it can also enhance the number of underlying storage node servers, support horizontal dynamic linear expansion to add storage servers and configure available device storage slots on the new storage nodes to improve the overall throughput of the system, and has good scalability, practicality and rationality for IoT time-series monitoring data applications.

[0047] The applicant analyzes the defects of the current conventional technical solutions:

[0048] For example, using traditional relational databases to store data presents difficulties in the early stage of table structure design. Based on the time series characteristics of IoT monitoring data,

[0049] On the one hand, the annual increase in the amount of data stored in a single device can reach millions. Storing multiple devices in one table will result in too much data in the single table, and performance bottlenecks will be quickly exposed. Storing some device data in one table according to business rules will lead to differentiation of different application characteristics, such as: devices in different factory areas are grouped together, and devices with different purposes are grouped together. Business rules may vary in different applications. Changes in rules make it difficult to evenly distribute device storage space, and differentiated adaptation requires increased R&D costs.

[0050] On the other hand, the storage design mechanism is related to the application access efficiency and storage expansion and maintenance convenience after a period of data growth. Under the traditional design mechanism, it is inconvenient to expand and migrate data. It is necessary to read fixed device data from the database table and move the storage to the expansion library to complete the migration. The workload is huge and the poor access experience of the application system during the migration period brings hidden dangers; the uneven data volume of physical tables stored in multiple devices will also lead to deviations in access efficiency after time. The access performance of the IoT table with a small space is relatively better, while the access performance of the IoT table with a large space is seriously reduced, which brings experience problems.

[0051] For example, if we fully adopt the big data technology solution, we can technically realize offline data processing through offline data warehouse and real-time analysis and processing of data by real-time stream computing engine, so as to meet the needs of delayed analysis and statistics and real-time analysis and statistics. However, the complete big data technology solution is more suitable for application in larger projects. Big data technology relies more on multiple servers with average computing power to form a cluster, and solves the problem of large-scale computing and storage expansion by multi-server cluster parallel computing. To achieve the effect, a certain amount of server resources must be invested in the early stage of the project, which has relatively high cost and long technology research and development cycle. It is not suitable for some projects with limited volume and project funds.

[0052] The present invention comprehensively analyzes the defects of the above conventional technical solutions and designs a method for dynamically expanding the storage capacity of IoT device monitoring data, which adopts an innovative way of combining algorithms with information technology points to flexibly solve the problems encountered; the method for dynamically expanding the storage capacity of IoT device monitoring data consists of three key technical parts: device data storage slot, distributed database connection middleware, and IoT device storage access routing algorithm, specifically:

[0053] 1. Use device data storage slot design

[0054] The device data storage slot is a logical concept in the present invention. A device storage slot actually corresponds to a physical storage table of a relational database, emphasizing that a single device has a unique device storage slot, and the monitoring data collected by a device is only stored in a physical table of the database. In the field of IoT sensing monitoring data storage, the advantages of a single device storing a single table are: on the one hand, it avoids the access performance problem caused by the long-term rapid growth of data volume stored in a table by multiple devices, and avoids the need to scan and filter a large amount of data from other device tables from the table when checking individual specified devices in many device data query scenarios;

[0055] If the independent storage of data of a single device is calculated based on 5 years, the data volume of a single table can reach tens of millions. The data indicators of a single device are all marked with the upload timestamp. The timestamp is used to create an index. The data volume of tens of millions can also be queried quickly in real time in a short time range query window (such as 5 minutes, 10 minutes, etc.); the calculation of periodic statistics can also ensure that the statistical results are obtained by delayed processing and summary calculation; the device storage slot has an identifiable storage slot name. After the storage slot is allocated to the device, the device will be bound to the storage slot space. Subsequently, combined with the intelligent device routing algorithm, data can be automatically read and written from the fixed storage slot;

[0056] In addition to solving the above-mentioned storage access performance issues, the storage slot design also greatly improves the convenience of operations when storage expansion and migration occur. By evaluating the storage performance and capacity of the database server, the device data storage slot space can be automatically created in the early stage, that is, the program automation can be used to create the appropriate number of device storage tables based on computing power and capacity. When improving the server hardware performance, the same expansion program script can still be used to automatically expand the device storage slots. When there are special reasons for the need to migrate device data, because a single device data is stored in a unique storage slot, the device data storage slot can be smoothly migrated and dumped to other databases by using the device-based automated migration method.

[0057] The design of device storage slots avoids the difficulty of table design and reduces the coupling between device data query access applications and device data storage;

[0058] Second, the use of distributed database connection middleware design

[0059] As the number of IoT devices online increases and time goes by, the amount of stored monitoring data continues to grow, and a single data storage resource server will gradually reach its upper limit. When a certain amount of data is carried, the database server can be expanded online, and device data storage slot space can be added to the newly expanded database server. The purpose of the distributed database connection middleware design is to support linear expansion and connect multiple database servers;

[0060] Database middleware is a middleware application that supports multiple database connections. The middleware program creates a configurable number of database connections for each connected data source and maintains database connections for device data storage and device data access. The database connection generally runs single or multiple device-related SQL statements stored in the corresponding library.

[0061] The database middleware provides a database connection configuration file in YAML format. When expanding the database connection, it supports dynamic configuration modification of the database connection configuration file. After the modification, the program configuration content is updated online to take effect, completing the dynamic expansion of the underlying distributed database to increase computing power and storage;

[0062] 3. Design of storage access routing algorithm using IoT devices

[0063] When an IoT device is first connected to the IoT cloud platform, the system will allocate specific device storage slots for the connected IoT device to store monitoring data based on the idle rate of the existing device storage slots on the backend distributed storage nodes. The device storage slots on the storage server have busy and idle states. The storage slots that have been bound to the device are busy, and the storage slots that have only allocated storage space but have not yet bound to the device are idle.

[0064] The summary storage status data and detailed allocation data of each server storage node will be updated and changed after each allocation of a bound device storage slot, and will be centrally stored. When the device is connected, the algorithm will be quickly called to calculate the idle rate of each storage node, so as to determine the storage slot space on the storage node with the lightest current load for the device, so as to achieve uniform distribution of device data storage.

[0065] The number of storage slots that can be allocated on the storage server node needs to be reasonably evaluated. For example, if the server computing and storage resources are consistent, a consistent number of storage slots can always be pre-allocated for each server. If the computing and storage resources of different servers are inconsistent, it is recommended to pre-allocate a reasonable number of storage slots according to the actual situation after evaluation. It is recommended not to pre-allocate too many storage slots. After a certain amount of use, the storage slot expansion function can be used to expand online through program functions when the server computing and storage resources are redundant. The advantage of this design is that when dealing with projects of different sizes, resource requirements can be reasonably evaluated, and server storage resources can be gradually increased flexibly and reasonably.

[0066] The advantages of the method for dynamic expansion of IoT device monitoring data storage composed of the above three key technologies compared with other conventional technical solutions are summarized as follows, with reference to the contents mentioned in the beneficial effects.

[0067] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the scope of protection of the present invention.

Claims

1. A method for dynamically expanding the storage capacity of monitoring data of an IoT device, characterized in that: The steps include: S1. Allocate device storage slot space for the data storage server, initialize and allocate device storage slots reasonably according to the computing resources and storage resources configured on the database server, and reserve resources for subsequent expansion of new device storage slots; S2. When an IoT device is connected for the first time, if there are multiple database storage nodes, each database storage node needs to record the latest number of used and available device storage slots after allocating device storage slots for the connected device, and uniformly summarize the available storage idle state data for the new device to bind and allocate device storage slots to obtain the latest device storage slot idle rate of the storage node; S3. Obtain the storage node with the lightest current load through algorithm comparison and determination, and divide the device storage slot for the device from this node. After the device is bound to the device storage slot, the storage database and storage physical table of the device are determined; In S1, each device storage slot is used to store data of a single device in a logical sense, and the device storage slot in a physical sense corresponds to a physical table of a database, which has a standardized library name and table name; When an IoT device is bound to a device storage slot, device-related data storage and access can be addressed to the corresponding library and table.

2. A method for dynamically expanding the storage capacity of IoT device monitoring data as claimed in claim 1, characterized in that: In S1, the process of dividing the device storage slots corresponds to the process of automatically creating the physical database table in physical operation, which is implemented by writing an automated script execution program, and the table is named by slot+sequential number.

3. A method for dynamically expanding the storage capacity of monitoring data of an IoT device according to claim 1, characterized in that: In S2 and S3, the above steps are the device binding access storage process, and the algorithm based on the whole process is the device routing-binding device storage slot algorithm; When obtaining the device storage slot idle rate, the calculation formula of the device storage slot idle rate of a single server is: device storage slot idle rate = number of device storage slots in use / total number of device storage slots.

4. A method for dynamically expanding the storage capacity of monitoring data of an IoT device according to claim 1, characterized in that: In the S3, the summary storage status data and detailed allocation data of each server storage node will be updated and changed after each allocation of a bound device storage slot, and will be uniformly collected and centrally stored for fast calling of an algorithm to calculate the device storage slot idle rate of each storage node when accessing the device, thereby determining to allocate the device storage slot space on the storage node with the lightest current load to the device.

5. A method for dynamically expanding the storage capacity of monitoring data of an IoT device according to claim 1, characterized in that: After S3, S4 is also included, which is the device data query process in the application; There are several device data queries in IoT applications. When querying single or multiple device data, it is convenient to obtain the target library and table information corresponding to the device storage slot bound to the device, and realize the connection query execution of different databases through the database middleware tool, and return the query results to the application.

6. A method for dynamically expanding the storage capacity of monitoring data of an Internet of Things device according to claim 5, characterized in that: In the S4, the database middleware is used to connect multiple underlying distributed data sources, support multi-threaded parallel query and aggregate query results of multiple devices, and support dynamic update of configuration files to achieve dynamic expansion of data sources.

Images