A storage method, system, terminal and storage medium for Internet of Things log data

Abstract

The present invention discloses a storage method, system, terminal and storage medium for Internet of Things log data, the method comprising: obtaining a target configuration file and a target single table, and performing table processing on the target single table according to the target configuration file to obtain multiple configuration tables, wherein the configuration tables include a monthly table, a sharding table corresponding to the monthly table and a rolling table corresponding to the sharding table; obtaining current Internet of Things log data, and storing the current Internet of Things log data in the corresponding multiple monthly tables; when the storage data of the target monthly table in the multiple monthly tables exceeds a first preset threshold, the storage data of the target monthly table is stored in the corresponding multiple sharding tables; when the storage data of the target sharding table in the multiple sharding tables exceeds a second preset threshold, the storage data of the target sharding table is stored in the corresponding multiple rolling tables. The present invention can effectively alleviate the storage pressure of a single table, and at the same time, it also effectively improves the efficiency of log query.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a storage method, system, terminal and computer-readable storage medium for Internet of Things log data. Background Art

[0002] With the rapid development of Internet of Things (IoT) technology, the number of devices and the amount of data generated are growing exponentially. Efficient log data storage solutions are essential for real-time monitoring, fault diagnosis and data analysis.

[0003] In the prior art, the collected log data is generally stored directly in a time series database. However, as the equipment is used and the date increases, a large amount of log data will be stored in the time series database. The table name of the time series database is created according to the device ID. When some devices are used frequently, the data storage volume of the table will increase exponentially, thereby affecting the query efficiency of the log.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide a storage method, system, terminal and computer-readable storage medium for IoT log data, aiming to solve the problem in the prior art that with the use of equipment and the increase of date, the amount of log data stored in a single table in a time series database continues to increase, resulting in high storage pressure on the single table and reduced log query efficiency.

[0006] To achieve the above object, the present invention provides a method for storing IoT log data, and the method for storing IoT log data comprises the following steps:

[0007] Obtain a target configuration file and a target single table, and perform table sharding on the target single table according to the target configuration file to obtain multiple configuration sharding tables, wherein the configuration sharding tables include a monthly table, a sharding table corresponding to the monthly table, and a rolling table corresponding to the sharding table;

[0008] Acquire current IoT log data, and store the current IoT log data in the corresponding multiple monthly tables;

[0009] When the storage data of the target monthly table among the multiple monthly tables exceeds a first preset threshold, the storage data of the target monthly table is stored in the corresponding multiple shard tables;

[0010] When the storage data of the target shard table among the plurality of shard tables exceeds a second preset threshold, the storage data of the target shard table is stored in the corresponding plurality of rolling tables.

[0011] Optionally, the method for storing IoT log data, wherein the step of obtaining a target configuration file and a target single table, and performing table splitting processing on the target single table according to the target configuration file to obtain multiple configuration table splits, specifically includes:

[0012] Obtain a target configuration file, wherein the target configuration file includes a monthly table configuration, a shard table configuration, and a rolling table configuration;

[0013] Acquire a target single table, and perform first sub-table processing on the target single table according to the monthly table configuration to obtain multiple monthly tables;

[0014] Performing second sharding processing on each of the monthly tables according to the sharding table configuration to obtain a plurality of sharding tables corresponding to each of the monthly tables;

[0015] A third table splitting process is performed on each of the shard tables according to the rolling table configuration to obtain a plurality of rolling tables corresponding to each of the shard tables.

[0016] Optionally, the method for storing IoT log data, wherein the step of obtaining current IoT log data and storing the current IoT log data in the corresponding plurality of monthly tables, specifically includes:

[0017] Collect current IoT log data in real time and obtain the data generation time of the current IoT log data;

[0018] Determine the multiple monthly tables corresponding to the current Internet of Things log data according to the data generation time, and store the current Internet of Things log data in the corresponding multiple monthly tables.

[0019] Optionally, the storage method of the Internet of Things log data, wherein when the storage data of the target monthly table among the multiple monthly tables exceeds a first preset threshold, the storage data of the target monthly table is stored in the corresponding multiple shard tables, specifically comprising:

[0020] Performing storage threshold detection on the storage data in the plurality of monthly tables, and when the storage data of a target monthly table among the plurality of monthly tables exceeds a first preset threshold, obtaining a plurality of device IDs corresponding to the storage data in the target monthly table;

[0021] A hash value calculation method is used to determine multiple shard tables corresponding to the multiple device IDs, to obtain multiple shard tables corresponding to the storage data in the target monthly table, and to store the storage data in the target monthly table into the corresponding multiple shard tables.

[0022] Optionally, the storage method of the Internet of Things log data, wherein the use of a hash value calculation method to determine a plurality of shard tables corresponding to a plurality of the device IDs, specifically includes:

[0023] Determine the number of shard tables corresponding to the plurality of shard tables, and use the hash value calculation method to calculate a plurality of hash values ​​corresponding to each of the plurality of device IDs;

[0024] An absolute value is taken for each of the hash values, and the absolute value is divided by the number of the shard tables to determine a plurality of shard tables corresponding to the plurality of device IDs.

[0025] Optionally, the storage method of the Internet of Things log data, wherein when the storage data of the target shard table among the multiple shard tables exceeds the second preset threshold, the storage data of the target shard table is stored in the corresponding multiple rolling tables, specifically comprising:

[0026] Setting a preset detection period, and performing storage threshold detection on the storage data in the plurality of shard tables according to the preset detection period;

[0027] When the storage data of the target shard table among the plurality of shard tables exceeds a second preset threshold, a plurality of rolling tables corresponding to the target shard table are determined, and the storage data of the target shard table are stored in the corresponding plurality of rolling tables.

[0028] Optionally, the storage method of the Internet of Things log data, wherein the storing data of the target shard table into the corresponding plurality of rolling tables, specifically comprises:

[0029] Storing the storage data of the target shard table in a first scrolling table among the scrolling tables;

[0030] When the stored data in the first rolling table exceeds a third preset threshold, the stored data exceeding the third preset threshold is stored in a second rolling table among the plurality of rolling tables until all the stored data in the target shard table is stored.

[0031] In addition, to achieve the above-mentioned purpose, the present invention also provides a storage system for IoT log data, wherein the storage system for IoT log data comprises:

[0032] A sub-table processing module is used to obtain a target configuration file and a target single table, and perform sub-table processing on the target single table according to the target configuration file to obtain multiple configuration sub-tables, wherein the configuration sub-tables include a monthly table, a shard table corresponding to the monthly table, and a rolling table corresponding to the shard table;

[0033] A monthly table storage module, used for acquiring current IoT log data, and storing the current IoT log data into corresponding multiple monthly tables;

[0034] A shard table storage module, configured to store the storage data of the target monthly table in the corresponding plurality of shard tables when the storage data of the target monthly table among the plurality of monthly tables exceeds a first preset threshold;

[0035] The rolling table storage module is used to store the storage data of the target shard table into the corresponding plurality of rolling tables when the storage data of the target shard table among the plurality of shard tables exceeds a second preset threshold.

[0036] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a storage program for Internet of Things log data stored on the memory and run on the processor, and when the storage program for Internet of Things log data is executed by the processor, the steps of the storage method for Internet of Things log data as described above are implemented.

[0037] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a storage program for Internet of Things log data, and when the storage program for Internet of Things log data is executed by a processor, the steps of the storage method for Internet of Things log data as described above are implemented.

[0038] In the present invention, a target configuration file and a target single table are obtained, and the target single table is sub-table processed according to the target configuration file to obtain multiple configuration sub-tables, wherein the configuration sub-tables include a monthly table, a shard table corresponding to the monthly table, and a rolling table corresponding to the shard table; current Internet of Things log data is obtained, and the current Internet of Things log data is stored in the corresponding multiple monthly tables; when the storage data of the target monthly table in the multiple monthly tables exceeds a first preset threshold, the storage data of the target monthly table is stored in the corresponding multiple shard tables; when the storage data of the target shard table in the multiple shard tables exceeds a second preset threshold, the storage data of the target shard table is stored in the corresponding multiple rolling tables. The present invention performs table sharding on the target single table according to the target configuration file, thereby obtaining multiple monthly tables. The log data can be stored on a monthly basis, effectively alleviating the storage pressure of the single table. Furthermore, the present invention also provides a sharding table corresponding to the monthly table and a rolling table corresponding to the sharding table, both of which can be dynamically expanded according to the continuous storage of log data, further alleviating the storage pressure of the monthly table, thereby solving the problem that data storage in a single table leads to a large storage volume of the single table, slow query and storage response, and service crashes. At the same time, it also effectively improves the query efficiency of log data. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart of a preferred embodiment of the storage method of the Internet of Things log data of the present invention;

[0040] Figure 2It is a structural diagram of a preferred embodiment of the storage system of the Internet of Things log data of the present invention;

[0041] Figure 3 It is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0043] With the rapid development of Internet of Things (IoT) technology, the number of devices and the amount of data generated are growing exponentially. An efficient log data storage solution is essential for real-time monitoring, fault diagnosis and data analysis. The present invention provides an IoT log data storage solution based on cloud computing and distributed database.

[0044] In the prior art, log data is generally stored directly in the time series database of Elasticsearch (search server). As the device is used and the date increases, a large amount of log data will be stored in the Elasticsearch time series database. The database table name is created according to the device ID. When some devices are used frequently, the data storage volume of the table will increase exponentially, affecting the query efficiency of the log data and increasing the probability of service crashes.

[0045] According to customer requirements, the subsequent time series database can be switched to ClickHouse, Influxdb, TDengine, Cassandra and other time series databases, but the common problems still exist. Therefore, it is necessary to design a universal storage solution to optimize storage and solve the problem of platform crash caused by excessive storage of data in a single table, resulting in high warehousing and query delays.

[0046] The storage method of the Internet of Things log data described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the storage method of the Internet of Things log data includes the following steps:

[0047] Step S10, obtaining a target configuration file and a target single table, and performing table segmentation processing on the target single table according to the target configuration file to obtain multiple configuration sub-tables, wherein the configuration sub-tables include a monthly table, a shard table corresponding to the monthly table, and a rolling table corresponding to the shard table.

[0048] Among them, the target configuration file refers to the amount of stored log data. At the beginning, the total storage capacity of the log data will be obtained, and then the table will be divided according to the total storage capacity to construct multiple monthly tables, multiple shard tables corresponding to the multiple monthly tables, and multiple rolling tables corresponding to the multiple shard tables.

[0049] Specifically, a target configuration file is obtained, wherein the target configuration file includes a monthly table configuration, a shard table configuration, and a rolling table configuration; a target single table is obtained, and a first shard table processing is performed on the target single table according to the monthly table configuration to obtain multiple monthly tables; a second shard table processing is performed on each of the monthly tables according to the shard table configuration to obtain multiple shard tables corresponding to each of the monthly tables; a third shard table processing is performed on each of the shard tables according to the rolling table configuration to obtain multiple rolling tables corresponding to each of the shard tables.

[0050] The present invention first optimizes the storage aspect, divides the original single table into monthly tables, and stores the data of the previous table monthly according to the time when the data was generated (stores the data in the corresponding monthly table according to the month when the data was generated), thereby reducing the data storage capacity of the single table.

[0051] As shown in Table 1, the tables before and after optimization are:

[0052] Table 1: Table before and after optimization

[0053]

[0054] Furthermore, in order to prevent the device from generating too much log data in the current month, the present invention is configured to perform sharding on the basis of monthly tables, generate shard tables according to the sharding configuration, further distribute the data, and reduce the data storage of a single table. For example, a single device log is configured with 3 shards (as shown in Table 2, which refers to a table for two months, September and October, and according to the configuration of 3 shards, corresponding 3 shards are generated for each month):

[0055] Table 2: Table before and after sharding

[0056]

[0057] The sharding configuration process is as follows: This sharding configuration is configured in a configuration file.

[0058] For example: storage:

[0059] partition:

[0060] num: 3# The default partition is 1;

[0061] max_size: 10240 #Unit: MB, default 10240 MB = 10G;

[0062] max_docs: 2000000000 # quantity, default is 200000000;

[0063] refreshRate: 600 #Refresh rate, default is 10 minutes and 600 seconds.

[0064] As shown in Table 3, in order to prevent the data from being too large after the table is sharded, the present invention can configure a rolling strategy, which can be configured in the configuration file to set the data storage quantity and / or storage size of a single table (here refers to the sharded table) for rolling. For example, if the current table (here refers to the sharded table) exceeds 1 million rows of logs or the single table storage exceeds 20G, it will be rolled and a rolling table will be generated to prevent a large amount of data from being stored in a single table (here refers to the sharded table).

[0065] Table 3: Table before and after scrolling

[0066]

[0067] It should be noted that in the present invention, the collected data is not first stored in a table and then split into multiple monthly tables by month, but is split according to the target configuration file at the beginning, that is, the table splitting operation in the present invention has been created in advance.

[0068] Step S20: Acquire current IoT log data, and store the current IoT log data in the corresponding multiple monthly tables.

[0069] In the IoT platform, device calls and rule calculations will generate a large amount of log data, including sensor data, status information, and event records. The cloud platform collects log data through the log collection module, converts it into formatted data, and then stores it in a distributed time series database.

[0070] Among them, IoT log data includes sensor data, status information (the current status of the device and online or offline records) and event records (event logs generated by the calculation of the rule or scenario linkage module). The difference between event records and sensor data and status information is that sensor data and status information refer to the operation and status record logs of a single device, while event records refer to the event records of multiple devices. For example, a scenario linkage is configured in the cloud, and the temperature is collected every 5 minutes. When the temperature exceeds 80 degrees, the alarm device is turned on and an alarm is issued to the user, and an event record log of the joint operation of multiple devices is obtained.

[0071] Sensor data: The device object model sent from the cloud has four functions: sending attributes, sending functions, reporting attributes, and reporting events, including: 1. Sending attributes: For example, the temperature data reported by the temperature sensor is sent to the panel through the cloud sending attributes and set as the temperature data of the panel. 2. Sending functions: For example, let the light device execute the function of turning on the switch. 3. Reporting attributes: For example, the temperature sensor reports temperature data to the cloud. 4. Reporting events: For example, a high temperature event is defined, and when it exceeds 80 degrees, the high temperature event is reported.

[0072] Currently, IoT log data needs to be converted before being stored in the monthly table. For example, a log text such as log.info("deiviceID" = "123", "product" = "456", "messageType" = "upmsg", "timestamp" = 1727338073034, "wendu" = 36.7, unit = "degrees Celsius", ip = "127.0.0.1", deviceType = "123") is extracted according to business needs, and the target record data is converted into formatted results for storage.

[0073] Specifically, the current Internet of Things log data is collected in real time, and the data generation time of the current Internet of Things log data is obtained; the multiple monthly tables corresponding to the current Internet of Things log data are determined according to the data generation time, and the current Internet of Things log data is stored in the corresponding multiple monthly tables.

[0074] After the monthly table is created, the subsequent log data storage will be stored in the form of monthly sub-tables (i.e., the monthly table in the present invention). For example, the current collected IoT log data corresponds to the time of generation in September, then each piece of data in the current collected IoT log data will be stored in the September table, that is, the present invention stores the log data according to its generation time.

[0075] Step S30: When the storage data of the target monthly table among the multiple monthly tables exceeds a first preset threshold, the storage data of the target monthly table is stored in the corresponding multiple shard tables.

[0076] When the amount of data in a certain month is too large, multiple shard tables corresponding to that monthly table will be determined, and the data of that monthly table will be stored in the corresponding shard tables to reduce the storage pressure of the monthly table.

[0077] Specifically, a storage threshold detection is performed on the storage data in the multiple monthly tables. When the storage data of the target monthly table among the multiple monthly tables exceeds a first preset threshold, the multiple device IDs corresponding to the storage data in the target monthly table are obtained; the number of shard tables corresponding to the multiple shard tables is determined, and the hash value calculation method is used to calculate the multiple hash values ​​corresponding to the multiple device IDs; the absolute value of each hash value is taken and divided by the number of shard tables to determine the multiple shard tables corresponding to the multiple device IDs, and the multiple shard tables corresponding to the storage data in the target monthly table are obtained, and the storage data in the target monthly table is stored in the corresponding multiple shard tables.

[0078] For example, the present invention configures 5 shards in the monthly table, and when the program is started for the first time, 5 shard tables corresponding to the monthly table will be generated according to the configuration.

[0079] Furthermore, the storage of log data in the shard table is calculated by an algorithm according to the device ID. For example, if 5 shards are configured, the hash value of the device ID is taken, and then the absolute value is taken and modulo 5 is performed. In this way, it is determined in which shard the log data corresponding to the current device ID needs to be stored, and then the log data corresponding to the device ID is stored in the corresponding shard table.

[0080] Step S40: When the storage data of the target shard table among the plurality of shard tables exceeds a second preset threshold, the storage data of the target shard table is stored in the corresponding plurality of rolling tables.

[0081] Specifically, a preset detection cycle is set, and storage threshold detection is performed on the storage data in the multiple shard tables according to the preset detection cycle; when the storage data of the target shard table in the multiple shard tables exceeds a second preset threshold, the multiple rolling tables corresponding to the target shard table are determined, and the storage data of the target shard table is stored in the first rolling table among the multiple rolling tables; when the storage data in the first rolling table exceeds a third preset threshold, the storage data exceeding the third preset threshold is stored in the second rolling table among the multiple rolling tables until all the storage data of the target shard table is stored.

[0082] When the program starts, it will perform periodic checks according to the refreshRate value. When the data of a table exceeds the configuration, the shard table will be rolled (the shard table will be sharded again).

[0083] The data storage methods of the rolling table and the sharding table in the present invention are different. The sharding table calculates the hash value according to the device ID to determine which sharding table the data is placed in. In the rolling table, the data is stored in sequence. Only when the data storage of the first rolling table is full will the second rolling table be opened. When the data storage of the second rolling table is full, the third rolling table, the fourth rolling table, and so on will be opened until all the data are stored.

[0084] Through the storage optimization of the present invention, the problem that data is stored in a single table, resulting in large storage capacity of the single table, slow query and storage response, and service crash can be effectively solved. The present invention is a universal solution, applicable to a variety of time series databases, and can cope with database replacement changes.

[0085] Furthermore, if Figure 2 As shown, based on the above-mentioned storage method of Internet of Things log data, the present invention also provides a storage system for Internet of Things log data, wherein the storage system for Internet of Things log data includes:

[0086] The sub-table processing module 51 is used to obtain a target configuration file and a target single table, and perform sub-table processing on the target single table according to the target configuration file to obtain multiple configuration sub-tables, wherein the configuration sub-tables include a monthly table, a shard table corresponding to the monthly table, and a rolling table corresponding to the shard table;

[0087] A monthly table storage module 52, used for acquiring current IoT log data and storing the current IoT log data into corresponding multiple monthly tables;

[0088] A shard table storage module 53, configured to store the storage data of the target monthly table in the corresponding plurality of shard tables when the storage data of the target monthly table among the plurality of monthly tables exceeds a first preset threshold;

[0089] The rolling table storage module 54 is used to store the storage data of the target shard table into the corresponding plurality of rolling tables when the storage data of the target shard table among the plurality of shard tables exceeds a second preset threshold.

[0090] Furthermore, if Figure 3 As shown, based on the above-mentioned storage method and system for IoT log data, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Figure 3 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0091] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a storage program 40 for Internet of Things log data is stored on the memory 20, and the storage program 40 for Internet of Things log data can be executed by the processor 10, thereby realizing the storage method for Internet of Things log data in the present application.

[0092] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the storage method of the IoT log data.

[0093] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0094] In one embodiment, when the processor 10 executes the storage program 40 of the IoT log data in the memory 20, the following steps are implemented:

[0095] Obtain a target configuration file and a target single table, and perform table sharding on the target single table according to the target configuration file to obtain multiple configuration sharding tables, wherein the configuration sharding tables include a monthly table, a sharding table corresponding to the monthly table, and a rolling table corresponding to the sharding table;

[0096] Acquire current IoT log data, and store the current IoT log data in the corresponding multiple monthly tables;

[0097] When the storage data of the target monthly table among the multiple monthly tables exceeds a first preset threshold, the storage data of the target monthly table is stored in the corresponding multiple shard tables;

[0098] When the storage data of the target shard table among the plurality of shard tables exceeds a second preset threshold, the storage data of the target shard table is stored in the corresponding plurality of rolling tables.

[0099] The step of obtaining a target configuration file and a target single table, and performing table-splitting processing on the target single table according to the target configuration file to obtain multiple configuration table-splittings specifically includes:

[0100] Obtain a target configuration file, wherein the target configuration file includes a monthly table configuration, a shard table configuration, and a rolling table configuration;

[0101] Acquire a target single table, and perform first sub-table processing on the target single table according to the monthly table configuration to obtain multiple monthly tables;

[0102] Performing second sharding processing on each of the monthly tables according to the sharding table configuration to obtain a plurality of sharding tables corresponding to each of the monthly tables;

[0103] A third table splitting process is performed on each of the shard tables according to the rolling table configuration to obtain a plurality of rolling tables corresponding to each of the shard tables.

[0104] The obtaining of current IoT log data and storing the current IoT log data in the corresponding multiple monthly tables specifically includes:

[0105] Collect current IoT log data in real time and obtain the data generation time of the current IoT log data;

[0106] Determine the multiple monthly tables corresponding to the current Internet of Things log data according to the data generation time, and store the current Internet of Things log data in the corresponding multiple monthly tables.

[0107] When the storage data of the target monthly table in the multiple monthly tables exceeds a first preset threshold, the storage data of the target monthly table is stored in the corresponding multiple shard tables, specifically including:

[0108] Performing storage threshold detection on the storage data in the plurality of monthly tables, and when the storage data of a target monthly table among the plurality of monthly tables exceeds a first preset threshold, obtaining a plurality of device IDs corresponding to the storage data in the target monthly table;

[0109] A hash value calculation method is used to determine multiple shard tables corresponding to the multiple device IDs, to obtain multiple shard tables corresponding to the storage data in the target monthly table, and to store the storage data in the target monthly table into the corresponding multiple shard tables.

[0110] The method of using a hash value calculation method to determine a plurality of shard tables corresponding to a plurality of device IDs specifically includes:

[0111] Determine the number of shard tables corresponding to the plurality of shard tables, and use the hash value calculation method to calculate a plurality of hash values ​​corresponding to each of the plurality of device IDs;

[0112] An absolute value is taken for each of the hash values, and the absolute value is divided by the number of the shard tables to determine a plurality of shard tables corresponding to the plurality of device IDs.

[0113] When the storage data of the target shard table in the plurality of shard tables exceeds a second preset threshold, the storage data of the target shard table is stored in the corresponding plurality of rolling tables, specifically including:

[0114] Setting a preset detection period, and performing storage threshold detection on the storage data in the plurality of shard tables according to the preset detection period;

[0115] When the storage data of the target shard table among the plurality of shard tables exceeds a second preset threshold, a plurality of rolling tables corresponding to the target shard table are determined, and the storage data of the target shard table are stored in the corresponding plurality of rolling tables.

[0116] The step of storing the storage data of the target shard table into the corresponding plurality of rolling tables specifically includes:

[0117] Storing the storage data of the target shard table in a first scrolling table among the scrolling tables;

[0118] When the stored data in the first rolling table exceeds a third preset threshold, the stored data exceeding the third preset threshold is stored in a second rolling table among the plurality of rolling tables until all the stored data in the target shard table is stored.

[0119] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a storage program for Internet of Things log data, and when the storage program for Internet of Things log data is executed by a processor, the steps of the storage method for Internet of Things log data as described above are implemented.

[0120] In summary, the present invention provides a storage method, system, terminal and computer-readable storage medium for Internet of Things log data, the method comprising: obtaining a target configuration file and a target single table, and performing table segmentation processing on the target single table according to the target configuration file to obtain multiple configuration sub-tables, wherein the configuration sub-tables include a monthly table, a sharding table corresponding to the monthly table and a rolling table corresponding to the sharding table; obtaining current Internet of Things log data, and storing the current Internet of Things log data in the corresponding multiple monthly tables; when the storage data of a target monthly table among the multiple monthly tables exceeds a first preset threshold, storing the storage data of the target monthly table in the corresponding multiple sharding tables; when the storage data of a target sharding table among the multiple sharding tables exceeds a second preset threshold, storing the storage data of the target sharding table in the corresponding multiple rolling tables. The present invention performs table sharding on the target single table according to the target configuration file, thereby obtaining multiple monthly tables. The log data can be stored on a monthly basis, effectively alleviating the storage pressure of the single table. Furthermore, the present invention also provides a sharding table corresponding to the monthly table and a rolling table corresponding to the sharding table, both of which can be dynamically expanded according to the continuous storage of log data, further alleviating the storage pressure of the monthly table, thereby solving the problem that data storage in a single table leads to a large storage volume of the single table, slow query and storage response, and service crashes. At the same time, it also effectively improves the query efficiency of log data.

[0121] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or terminal including the element.

[0122] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0123] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for storing IoT log data, characterized in that: The storage method of the Internet of Things log data includes: Obtain a target configuration file and a target single table, and perform table sharding on the target single table according to the target configuration file to obtain multiple configuration sharding tables, wherein the configuration sharding tables include a monthly table, a sharding table corresponding to the monthly table, and a rolling table corresponding to the sharding table; The step of obtaining a target configuration file and a target single table, and performing table-splitting processing on the target single table according to the target configuration file to obtain multiple configuration sub-tables specifically includes: Obtain a target configuration file, wherein the target configuration file includes a monthly table configuration, a shard table configuration, and a rolling table configuration; Acquire a target single table, and perform first sub-table processing on the target single table according to the monthly table configuration to obtain multiple monthly tables; Performing second sharding processing on each of the monthly tables according to the sharding table configuration to obtain a plurality of sharding tables corresponding to each of the monthly tables; Performing a third table splitting process on each of the shard tables according to the rolling table configuration to obtain a plurality of rolling tables corresponding to each of the shard tables; The target configuration file refers to the amount of stored log data, obtaining the total storage capacity of log data, performing table sharding according to the total storage capacity, and constructing multiple monthly tables, multiple shard tables corresponding to the multiple monthly tables, and multiple rolling tables corresponding to the multiple shard tables; Acquire current IoT log data, and store the current IoT log data in the corresponding multiple monthly tables; When the storage data of the target monthly table among the multiple monthly tables exceeds a first preset threshold, the storage data of the target monthly table is stored in the corresponding multiple shard tables; When the storage data of the target monthly table in the plurality of monthly tables exceeds a first preset threshold, the storage data of the target monthly table is stored in the corresponding plurality of shard tables, specifically including: Performing storage threshold detection on the storage data in the plurality of monthly tables, and when the storage data of a target monthly table among the plurality of monthly tables exceeds a first preset threshold, obtaining a plurality of device IDs corresponding to the storage data in the target monthly table; Determine multiple shard tables corresponding to multiple device IDs using a hash value calculation method, obtain multiple shard tables corresponding to the data stored in the target monthly table, and store the data stored in the target monthly table in the corresponding multiple shard tables; The method of using a hash value calculation method to determine a plurality of shard tables corresponding to a plurality of device IDs specifically includes: Determine the number of shard tables corresponding to the plurality of shard tables, and use the hash value calculation method to calculate a plurality of hash values ​​corresponding to each of the plurality of device IDs; Taking an absolute value of each of the hash values ​​and dividing the absolute value by the number of the shard tables to determine a plurality of shard tables corresponding to the plurality of device IDs; When the storage data of the target shard table among the plurality of shard tables exceeds a second preset threshold, the storage data of the target shard table is stored in the corresponding plurality of rolling tables; The storing the storage data of the target shard table into the corresponding plurality of rolling tables specifically includes: Storing the storage data of the target shard table in a first scrolling table among the scrolling tables; When the stored data in the first rolling table exceeds a third preset threshold, the stored data exceeding the third preset threshold is stored in a second rolling table among the plurality of rolling tables until all the stored data in the target shard table is stored.

2. The storage method of Internet of Things log data according to claim 1, characterized in that: The obtaining of current IoT log data and storing the current IoT log data in the corresponding plurality of monthly tables specifically includes: Collect current IoT log data in real time and obtain the data generation time of the current IoT log data; Determine the multiple monthly tables corresponding to the current IoT log data according to the data generation time, and store the current IoT log data in the corresponding multiple monthly tables.

3. The storage method of Internet of Things log data according to claim 1, characterized in that: When the storage data of the target shard table in the plurality of shard tables exceeds a second preset threshold, the storage data of the target shard table is stored in the corresponding plurality of rolling tables, specifically including: Setting a preset detection period, and performing storage threshold detection on the storage data in the plurality of shard tables according to the preset detection period; When the storage data of the target shard table among the plurality of shard tables exceeds a second preset threshold, a plurality of rolling tables corresponding to the target shard table are determined, and the storage data of the target shard table are stored in the corresponding plurality of rolling tables.

4. A storage system for IoT log data, characterized in that: The storage system of the Internet of Things log data includes: A sub-table processing module is used to obtain a target configuration file and a target single table, and perform sub-table processing on the target single table according to the target configuration file to obtain multiple configuration sub-tables, wherein the configuration sub-tables include a monthly table, a shard table corresponding to the monthly table, and a rolling table corresponding to the shard table; The step of obtaining a target configuration file and a target single table, and performing table-splitting processing on the target single table according to the target configuration file to obtain multiple configuration sub-tables specifically includes: Obtain a target configuration file, wherein the target configuration file includes a monthly table configuration, a shard table configuration, and a rolling table configuration; Acquire a target single table, and perform first sub-table processing on the target single table according to the monthly table configuration to obtain multiple monthly tables; Performing second sharding processing on each of the monthly tables according to the sharding table configuration to obtain a plurality of sharding tables corresponding to each of the monthly tables; Performing a third table splitting process on each of the shard tables according to the rolling table configuration to obtain a plurality of rolling tables corresponding to each of the shard tables; The target configuration file refers to the amount of stored log data, obtaining the total storage capacity of log data, performing table sharding according to the total storage capacity, and constructing multiple monthly tables, multiple shard tables corresponding to the multiple monthly tables, and multiple rolling tables corresponding to the multiple shard tables; A monthly table storage module, used for acquiring current IoT log data, and storing the current IoT log data into corresponding multiple monthly tables; A shard table storage module, configured to store the storage data of the target monthly table in the corresponding plurality of shard tables when the storage data of the target monthly table among the plurality of monthly tables exceeds a first preset threshold; When the storage data of the target monthly table in the plurality of monthly tables exceeds a first preset threshold, the storage data of the target monthly table is stored in the corresponding plurality of shard tables, specifically including: Performing storage threshold detection on the storage data in the plurality of monthly tables, and when the storage data of a target monthly table among the plurality of monthly tables exceeds a first preset threshold, obtaining a plurality of device IDs corresponding to the storage data in the target monthly table; Determine multiple shard tables corresponding to multiple device IDs using a hash value calculation method, obtain multiple shard tables corresponding to the data stored in the target monthly table, and store the data stored in the target monthly table in the corresponding multiple shard tables; The method of using a hash value calculation method to determine a plurality of shard tables corresponding to a plurality of device IDs specifically includes: Determine the number of shard tables corresponding to the plurality of shard tables, and use the hash value calculation method to calculate a plurality of hash values ​​corresponding to each of the plurality of device IDs; Taking an absolute value of each of the hash values ​​and dividing the absolute value by the number of the shard tables to determine a plurality of shard tables corresponding to the plurality of device IDs; A rolling table storage module, used for storing the storage data of the target shard table in the plurality of shard tables into the corresponding plurality of rolling tables when the storage data of the target shard table in the plurality of shard tables exceeds a second preset threshold; The storing the storage data of the target shard table into the corresponding plurality of rolling tables specifically includes: Storing the storage data of the target shard table in a first scrolling table among the scrolling tables; When the stored data in the first rolling table exceeds a third preset threshold, the stored data exceeding the third preset threshold is stored in a second rolling table among the plurality of rolling tables until all the stored data in the target shard table is stored.

5. A terminal, characterized in that: The terminal includes: a memory, a processor, and a storage program for Internet of Things log data stored in the memory and executable on the processor. When the storage program for Internet of Things log data is executed by the processor, the steps of the storage method for Internet of Things log data as described in any one of claims 1 to 3 are implemented.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a storage program for Internet of Things log data, and when the storage program for Internet of Things log data is executed by a processor, the steps of the storage method for Internet of Things log data according to any one of claims 1 to 3 are implemented.

Images