Internet of Things data access processing method and device
Patent Information
- Application Number
- CN202411525976.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-30
- Publication Date
- 2025-02-28
Smart Images
Figure CN119537422A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer application technology, and in particular to a method and device for data access processing in the Internet of Things, and also to an electronic device, a non-transitory computer-readable storage medium, and a computer program product. Background Art
[0002] The Internet of Things refers to a network that connects IoT devices to the Internet through information sensing devices and according to agreed protocols to exchange and communicate information to achieve intelligent identification, positioning, tracking, monitoring and management. IoT data has the characteristics of massive IoT device connections, time series, structured data, more writes and less reads, and no updates. The IoT system architecture can be summarized as data collection, data storage, data query, data analysis, data display and application, among which the IoT data storage and data query capabilities directly affect the operating cost and access efficiency of the IoT system. At present, the common IoT data storage databases mainly include Apache Hive, Apache Hbase, Apache IoTDB, Apache doris, Clickhouse, InfluxDB, CrateDB, MongoDB, Cassandra, and TDengine, which are divided into columnar databases or time series databases. Common storage methods are divided into: (1) large wide table storage method: define a large wide table based on product attributes to save all attribute data in the same time frame; (2) single table multi-point storage method: multiple attributes of the same type are saved in a single table; (3) single table point storage method: each device and each attribute are stored in a separate table; (4) document storage method: each data sent is saved in the form of a document in Json format. However, although the above storage methods can meet most IoT scenarios, there is a lot of data redundancy, increased storage volume, and unreasonable data structure, resulting in low query efficiency and query performance. Therefore, how to design a more efficient, accurate and low-cost IoT data access and processing method has become a technical problem that needs to be solved urgently. Summary of the invention
[0003] The present invention provides an Internet of Things data access processing method, which is used to solve the defects of the Internet of Things data access scheme existing in the prior art, that is, the Internet of Things data access scheme has high limitations, resulting in poor actual query efficiency and query performance.
[0004] The present invention provides an Internet of Things data access processing method, comprising: Get the attribute key used to query IoT data stored in key-value vectorized group storage; Based on the device type of the IoT device, obtain the mapping conversion coefficient and the mapping conversion offset corresponding to the attribute key from the relational storage database; wherein the relational storage database stores static attribute data of the IoT device; Based on the mapping relationship between the attribute key and the preset key-value pair corresponding to the quantized group, the attribute value corresponding to the attribute key is obtained from the column storage database; the column storage database stores the dynamic attribute data of the IoT device during operation; the attribute value is the physical value of the dynamic attribute data; Based on the mapping conversion coefficient and the mapping conversion offset corresponding to the attribute key and the attribute value corresponding to the attribute key, the business logic value of the Internet of Things data corresponding to the attribute value is obtained.
[0005] According to the IoT data access processing method provided by the present invention, before obtaining the attribute key for querying the IoT data stored in the key-value pair vectorized group storage mode, it also includes: Obtain static attribute data of the IoT device, and store the static attribute data of the IoT device in a relational storage database; wherein the static attribute data includes a device type identifier, a mapping conversion coefficient, and a mapping conversion offset; The dynamic attribute data of the IoT device during operation is obtained, and the dynamic attribute data is stored in a column storage database in a key-value pair quantized group storage manner; wherein the dynamic attribute data is the collected dynamically changing attribute data; the key-value pair quantized group includes an attribute key vector and an attribute value vector, the attribute key vector includes the physical values of static attribute data corresponding to multiple attribute keys, and the attribute value vector includes the physical values of dynamic attribute data corresponding to multiple attribute values.
[0006] The method for accessing and processing Internet of Things data provided by the present invention further includes: In a case where the attribute key is at least two attribute keys corresponding to at least two IoT data respectively, based on the device type of the IoT device, obtaining at least two mapping conversion coefficients and at least two mapping conversion offsets corresponding to the at least two attribute keys from a relational storage database; Based on the mapping relationship between the at least two attribute keys and the preset key-value pairs corresponding to the quantized group, obtaining at least two attribute values corresponding to the at least two attribute keys in the column storage database; Based on at least two mapping conversion coefficients and at least two mapping conversion offsets corresponding to the at least two attribute keys and at least two attribute values corresponding to the at least two attribute keys, at least two business logic values of the Internet of Things data corresponding to the at least two attribute values are obtained respectively.
[0007] According to the IoT data access processing method provided by the present invention, storing the dynamic attribute data in a column storage database specifically includes: Binary serialization and compression processing are performed on the dynamic attribute data to obtain compressed dynamic attribute data; The compressed dynamic attribute data is stored in a column storage database.
[0008] According to the IoT data access processing method provided by the present invention, the business logic value of the IoT data corresponding to the attribute value is obtained based on the mapping conversion coefficient and mapping conversion offset corresponding to the attribute key and the attribute value corresponding to the attribute key, specifically including: The mapping conversion coefficient and mapping conversion offset corresponding to the attribute key and the attribute value corresponding to the attribute key are input into a preset business logic value algorithm model for processing to obtain the business logic value of the Internet of Things data corresponding to the attribute value; wherein the business logic value algorithm model is pre-configured.
[0009] According to the IoT data access processing method provided by the present invention, the device type based on the IoT device, obtaining the mapping conversion coefficient and the mapping conversion offset corresponding to the attribute key in the relational storage database, specifically includes: The device type of the Internet of Things device is used as an index to search the relational storage database to obtain a mapping conversion coefficient and a mapping conversion offset corresponding to the index.
[0010] The present invention also provides an Internet of Things data access processing device, comprising: An attribute key acquisition unit, used for acquiring an attribute key for querying IoT data stored in a key-value pair vectorized group storage manner; A conversion parameter acquisition unit, used to acquire a mapping conversion coefficient and a mapping conversion offset corresponding to the attribute key from a relational storage database based on the device type of the Internet of Things device; wherein the relational storage database stores static attribute data of the Internet of Things device; An attribute value search unit is used to obtain an attribute value corresponding to the attribute key from a column storage database based on the mapping relationship between the attribute key and the preset key-value pair corresponding to the quantized group; the column storage database stores dynamic attribute data of the IoT device during operation; the attribute value is a physical value of the dynamic attribute data; The attribute value processing unit is used to obtain the business logic value of the Internet of Things data corresponding to the attribute value based on the mapping conversion coefficient and mapping conversion offset corresponding to the attribute key and the attribute value corresponding to the attribute key.
[0011] According to the Internet of Things data access processing device provided by the present invention, before obtaining the attribute key for querying the Internet of Things data stored in the key-value pair vectorized group storage mode, it also includes: Obtain static attribute data of the IoT device, and store the static attribute data of the IoT device in a relational storage database; wherein the static attribute data includes a device type identifier, a mapping conversion coefficient, and a mapping conversion offset; The dynamic attribute data of the IoT device during operation is obtained, and the dynamic attribute data is stored in a column storage database in a key-value pair quantized group storage manner; wherein the dynamic attribute data is the collected dynamically changing attribute data; the key-value pair quantized group includes an attribute key vector and an attribute value vector, the attribute key vector includes the physical values of static attribute data corresponding to multiple attribute keys, and the attribute value vector includes the physical values of dynamic attribute data corresponding to multiple attribute values.
[0012] The Internet of Things data access processing device provided by the present invention also includes: In a case where the attribute key is at least two attribute keys corresponding to at least two IoT data respectively, based on the device type of the IoT device, obtaining at least two mapping conversion coefficients and at least two mapping conversion offsets corresponding to the at least two attribute keys from a relational storage database; Based on the mapping relationship between the at least two attribute keys and the preset key-value pairs corresponding to the quantized group, obtaining at least two attribute values corresponding to the at least two attribute keys in the column storage database; Based on at least two mapping conversion coefficients and at least two mapping conversion offsets corresponding to the at least two attribute keys and at least two attribute values corresponding to the at least two attribute keys, at least two business logic values of the Internet of Things data corresponding to the at least two attribute values are obtained respectively.
[0013] According to the IoT data access processing device provided by the present invention, storing the dynamic attribute data in a column storage database specifically includes: Binary serialization and compression processing are performed on the dynamic attribute data to obtain compressed dynamic attribute data; The compressed dynamic attribute data is stored in a column storage database.
[0014] According to the IoT data access processing device provided by the present invention, the obtaining of the business logic value of the IoT data corresponding to the attribute value based on the mapping conversion coefficient and mapping conversion offset corresponding to the attribute key and the attribute value corresponding to the attribute key specifically includes: The mapping conversion coefficient and mapping conversion offset corresponding to the attribute key and the attribute value corresponding to the attribute key are input into a preset business logic value algorithm model for processing to obtain the business logic value of the Internet of Things data corresponding to the attribute value; wherein the business logic value algorithm model is pre-configured.
[0015] According to the IoT data access processing device provided by the present invention, the device type based on the IoT device, obtaining the mapping conversion coefficient and the mapping conversion offset corresponding to the attribute key in the relational storage database, specifically includes: The device type of the Internet of Things device is used as an index to search the relational storage database to obtain a mapping conversion coefficient and a mapping conversion offset corresponding to the index.
[0016] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for accessing and processing Internet of Things data as described in any one of the above items is implemented.
[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for accessing and processing Internet of Things data as described in any one of the above items is implemented.
[0018] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the Internet of Things data access processing method as described in any of the above items.
[0019] The Internet of Things data access processing method provided by the present invention obtains the attribute key for querying the Internet of Things data stored in the key-value pair vectorized group storage mode; based on the device type of the Internet of Things device, obtains the mapping conversion coefficient and mapping conversion offset corresponding to the attribute key in the relational storage database; based on the attribute key and the mapping relationship corresponding to the preset key-value pair vectorized group, obtains the attribute value corresponding to the attribute key in the column storage database; the column storage database stores the dynamic attribute data of the Internet of Things device during operation; the attribute value is the physical value of the dynamic attribute data; based on the mapping conversion coefficient and mapping conversion offset corresponding to the attribute key and the attribute value corresponding to the attribute key, obtains the business logic value of the Internet of Things data corresponding to the attribute value. The method of the present invention greatly reduces the storage space, can effectively improve the efficiency and accuracy of reading Internet of Things data, thereby greatly reducing the operating cost. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0021] Figure 1 It is a flow chart of the Internet of Things data access processing method provided by the present invention.
[0022] Figure 2 It is a structural schematic diagram of the Internet of Things data access processing device provided by the present invention.
[0023] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0025] Combine the following Figure 1-Figure 3 The Internet of Things data access processing method and device of the present invention are described, and its embodiments are described in detail.
[0026] The following is a detailed description of an embodiment of the IoT data access processing method according to the present invention. Figure 1 As shown, it is a flow chart of the IoT data access processing method provided by the present invention, and the specific implementation process includes the following steps: Step 101: Obtain an attribute key for querying IoT data stored in a key-value pair vectorized group storage manner.
[0027] In an embodiment of the present invention, the attribute key may be a single attribute key or at least two attribute keys of a combination of multiple attributes. Before obtaining the attribute key for querying the IoT data stored in a key-value pair vectorized group storage method, it is necessary to obtain the static attribute data of the IoT device in advance and store the static attribute data of the IoT device in a relational storage database. Among them, the static attribute data includes device type identification, mapping conversion coefficient and mapping conversion offset, attribute identification, attribute number, attribute description, attribute unit, regular acquisition frequency, and abnormal acquisition frequency. In addition, the dynamic attribute data of the IoT device during operation can also be obtained, and the dynamic attribute data is stored in a column storage database in a key-value pair vectorized group storage method; wherein, the dynamic attribute data is the dynamically changing attribute data collected; the key-value pair vectorized group includes an attribute key vector and an attribute value vector, the attribute key vector includes the physical values of the static attribute data corresponding to multiple attribute keys, and the attribute value vector includes the physical values of the dynamic attribute data corresponding to multiple attribute values. In the process of storing the dynamic attribute data in the column storage database, the dynamic attribute data needs to be binary serialized and compressed to obtain the compressed dynamic attribute data, and the compressed dynamic attribute data is stored in the column storage database.
[0028] Specifically, in the embodiment of the present invention, the static attribute data (i.e., static master data) of the IoT device is modeled. The static attribute data of the same device type is unchanged, and the static attribute data is saved in a relational database. This part separates the static master data from the real-time collected data (i.e., dynamic attribute data), reduces data redundancy, and reduces the amount of data storage. The physical network data includes static attribute data and dynamic attribute data.
[0029] First, classify the IoT devices that need to be collected according to the business type, and organize and model the static attribute data such as attribute key, attribute description, attribute unit, regular collection frequency, abnormal collection frequency, mapping conversion coefficient, and mapping conversion offset that each IoT device needs to collect according to the business collection requirements and rules. Example: IoT devices with device type S1 (i.e., device type number) define the collection of multiple attribute data such as temperature, pressure, rotation speed, and vehicle speed, and set the regular collection frequency and abnormal collection frequency.
[0030] The classification table for building IoT devices is as follows:
[0031] Construct the IoT device table as follows:
[0032] The static master data table for collecting IoT device classification attributes is constructed as follows:
[0033] In an embodiment of the present invention, the classification table of Internet of Things devices, the Internet of Things device table, and the static master data table for classification attribute collection of Internet of Things devices are stored in a relational storage database.
[0034] Step 102: Based on the device type of the IoT device, obtain a mapping conversion coefficient and a mapping conversion offset corresponding to the attribute key from a relational storage database; wherein the relational storage database stores static attribute data of the IoT device.
[0035] Specifically, the device type of the IoT device may be used as an index to search the relational storage database to obtain a mapping conversion coefficient and a mapping conversion offset corresponding to the index.
[0036] The static attribute data includes device type identification (such as device type id), mapping conversion coefficient and mapping conversion offset, attribute identification (such as attribute id), attribute number, attribute description, attribute unit, regular collection frequency, and abnormal collection frequency.
[0037] The present invention uses a column storage database to store the dynamic attribute data collected by frequency when the IoT device is running in the form of a vectorized group of dynamic attribute keys and attribute values. The present invention maps the real-time collected dynamic attribute data of each attribute to an attribute ID and a physical value, and converts the attribute list collected at the same time into a vectorized group of key-value pairs, solving the problem of saving dynamic attributes collected each time. Among them, the mapping relationship of the attribute is the static master data defined in the first part, and the collected attribute value is the physical value x , the mapping conversion coefficient and mapping conversion offset defined by the static master data can be converted into business logic values f (x), the conversion formula is:
[0038] Among them, F is the mapping conversion factor, O is the mapping conversion offset. Finally, the data (i.e. physical value) is converted into x ) compression storage, greatly reducing the amount of data storage, reducing operating costs and improving query efficiency. It should be noted that in the actual data collection process, the physical values collected are usually current signals or voltage signals, such as speed, and the collected physical values need to be converted according to the above conversion formula x Converted to the value of the rotation speed (i.e. one of the business logic values). For example, the physical value of the temperature can be 3191. After conversion by the conversion formula, the actual temperature value can be 31.91℃.
[0039] Among them, the LZ4 compression algorithm is a lossless compression algorithm that provides a high compression ratio and a low decompression / compression speed. The compression ratio of time, date, and number is 1.92. Delta encoding is a differential encoding technology that reduces the data size by storing the difference between adjacent data. Delta encoding is suitable for date, time, and numeric types.
[0040] Common IoT data storage methods are divided into: Large wide table storage method: Define a large wide table based on product attributes to save all attribute data in the same time frame. Support online addition of devices and device types, support multi-attribute joint query analysis, and do not support dynamic attribute storage function. When adding new attributes, you need to change the table structure offline and add corresponding attributes, which will affect the operation of online services; Single table multi-point storage method: Multiple attributes of the same type are saved in a single table. Support online addition of devices, device types and attributes, and support dynamic attribute storage function. However, there is redundancy in the attribute master data, and the attributes are not in the same data row, so multi-attribute joint query analysis is not supported; Single table point storage method: Each device and each attribute are separated into separate tables. It does not support online addition of devices, device types and attributes, and does not support dynamic attribute storage function. When adding devices and attributes, you need to create a new table. Attributes are not in the same data row, so multi-attribute joint query analysis is not supported; Document storage method: Each data sent is saved in the form of a document in Json format. Support online addition of devices, device types and attributes, support dynamic attribute storage function, and support multi-attribute joint query analysis. However, there is a lot of redundancy in the data, and the document data structure analysis query algorithm is inefficient.
[0041]
[0042] The function comparison table of different storage methods is shown above. It can be seen that both the document storage method and the key-value pair vectorized group storage method meet the functional requirements.
[0043] For example, taking the device data of device numbers S100001 and S200002 as an example, each device collects attribute data such as temperature, pressure, rotation speed, ..., vehicle speed, etc. at a regular collection frequency, and lists different storage method tables respectively. The storage date is the main partition, and the data is sorted by upload time and device number. The vectorized group storage method of attribute keys and attribute values of the present invention is as follows: CREATE TABLE test.productRealtimeData
[0044] (`transfer_date` Date DEFAULT toDate(now(),'Asia / Shanghai') commit 'Save date',
[0045] `create_time` DateTime DEFAULT now() commit 'save time',
[0046] `upload_time` Int64 commit 'upload timestamp',
[0047] `product_id` Int32 commit 'device type id',
[0048] `device_id` String commit 'device id',
[0049] `property.id` Array(Int32) commit 'property key vector',
[0050] `property.value` Array(Int32)) commit 'property value vector'
[0051] ENGINE = MergeTree
[0052] PARTITION BY toYYYYMMDD(transfer_date)
[0053] ORDER BY (transfer_date,product_id,device_id,upload_time)
[0054] SETTINGS index_granularity = 8192; The table of the vectorized group storage method of the attribute key and the attribute value of the present invention is as follows:
[0055] The tables in the large wide table storage mode are as follows:
[0056] The table of single table multiple point storage mode is as follows:
[0057] The S100001.temperature1 table is as follows:
[0058] The S100001.press1 table is as follows:
[0059] The S100001.rpm1 table is as follows:
[0060] S100001...The table is as follows:
[0061] The S100001.speed1 table is as follows:
[0062] The S200002.temperature2 table is as follows:
[0063] The S200002.press2 table is as follows:
[0064] The S200002.rpm2 table is as follows:
[0065] S200002...The table is as follows:
[0066] The S200002.speed2 table is as follows:
[0067] The above S100001.temperature1 table to S200002.speed2 table are stored in single table point mode.
[0068] The document storage table is as follows:
[0069] As can be seen from the examples, the storage method of the present invention reduces redundancy, supports dynamic attribute storage, stores data at the same time in the same data row, and supports multi-attribute joint query analysis.
[0070] Constructing 50,000 sample data, when each data has 332 attributes, the storage capacity S4 of the document storage method and the storage capacity S of the vectorized group storage method are compared: S4:4=407,685,600Byte:14,404,389Byte=28.3:1. It can be seen that the storage capacity of the present invention is greatly reduced, and the storage cost is reduced.
[0071] Step 103: Based on the mapping relationship between the attribute key and the preset key-value pair to the quantized group, obtain the attribute value corresponding to the attribute key in the column storage database.
[0072] The column storage database stores dynamic attribute data of the IoT device during operation; the attribute value is a physical value of the dynamic attribute data.
[0073] Step 104: Based on the mapping conversion coefficient and mapping conversion offset corresponding to the attribute key and the attribute value corresponding to the attribute key, obtain the business logic value of the Internet of Things data corresponding to the attribute value.
[0074] Specifically, the mapping conversion coefficient and mapping conversion offset corresponding to the attribute key and the attribute value corresponding to the attribute key can be input into a preset business logic value algorithm model for processing to obtain the business logic value of the Internet of Things data corresponding to the attribute value; wherein the business logic value algorithm model is pre-configured.
[0075] When the attribute key is a single attribute key, a single attribute query is performed. Given the attribute key (ie, the attribute key), the attribute value (ie, the value of the attribute key) is queried according to the mapping relationship. First, obtain the attribute key according to the attribute master data (the attribute key is the attribute key corresponding to the attribute id, for example, the attribute id is 1) and match it to the static master data table of the classified attribute collection of the IoT device to find the corresponding mapping conversion coefficient F (the attribute id is 1 and the corresponding F is -100) and the mapping conversion offset O (the attribute id is 1 and the corresponding O is 0), and then query the index i of the attribute key in the attribute key vector (i.e., property.id) K in the repository (for example, the value corresponding to the third index i of the property.id column in the table of the vectorized group storage method of the attribute key and the attribute value is 3; in addition, as shown in the table, the value corresponding to the first index i is 3191, and the value corresponding to the second index i is 1), and then obtain the value of index i in V in the attribute value vector (i.e., property.value) according to the preset mapping relationship corresponding to the vectorized group of key-value pairs (for example, the value corresponding to the third index i of the property.value column in the table of the vectorized group storage method of the attribute key and the attribute value is 1600; in addition, as shown in the table, the value corresponding to the first index i is 1, and the value corresponding to the second index i is 2), and then query the index i in V in the attribute value vector (i.e., property.value) according to the conversion formula ,Will Substitute x into the conversion formula to calculate the attribute value (i.e., the business logic value of the IoT data corresponding to the attribute value) f ( x )), forming an attribute value set (i.e., a set of business logic values of the Internet of Things data corresponding to the attribute value).
[0076] Among them, K is the attribute key vector, V is the attribute value vector, attribute key= ; F is the mapping conversion coefficient of the attribute key, O is the mapping conversion offset of the attribute key, then the attribute value value=F +O.
[0077] It should be noted that "3191, 1, 1600, ..., 500" corresponding to the property value vector in the property.value column is the physical value; 1, 2, 3, ..., 5 corresponding to the property key vector in the property.id column is the property id.
[0078] Example SQL for storing key-value pairs in vectorized groups: select (`property.value`[arrayFirstIndex(x->x=7, `property.id`)]*-10+0) as value from test.productRealtimeData where transfer_date='2024-08-13'and product_id=2 and deviceId=4.
[0079] In addition, it should be noted that, in the case where the attribute key is at least two attribute keys corresponding to at least two IoT data respectively, based on the device type of the IoT device, at least two mapping conversion coefficients and at least two mapping conversion offsets corresponding to the at least two attribute keys are obtained from the relational storage database. Based on the at least two attribute keys and the mapping relationship corresponding to the preset key-value pair vectorization group, at least two attribute values corresponding to the at least two attribute keys are obtained from the column storage database. Based on the at least two mapping conversion coefficients and at least two mapping conversion offsets corresponding to the at least two attribute keys and the at least two attribute values corresponding to the at least two attribute keys, at least two business logic values of the IoT data corresponding to the at least two attribute values are obtained respectively.
[0080] For example, the following takes the case where the attribute keys are two attribute keys corresponding to two IoT data as an example for explanation. The multi-attribute joint query analysis is as follows. Given the attribute key1 corresponding to the first IoT data and the attribute key2 corresponding to the second IoT data, query all values of attribute key2 when the attribute key1 value is within a certain range S (S=(a,b]). The first IoT data may be temperature, and a in the range S may be 30°C and b may be 36°C. When the temperature range is set, analyze all values of attribute key2 corresponding to the second IoT data. For example, the second IoT data may be rotation speed. First, obtain the mapping conversion coefficient F1, mapping conversion coefficient F2 and mapping conversion offset O1, mapping conversion offset O2 of the attribute key1 and attribute key2 according to the attribute master data, then query the index i of the attribute key1 and attribute key2 in the attribute key vector K in the repository, and then obtain the value of index i in V in the attribute value vector according to the mapping relationship, and then calculate the attribute value set VA1 of the attribute key1 according to the above conversion formula, and obtain the row index set R according to the attribute value set VA1, in which all the value sets VA2 of the attribute key2 are found.
[0081] Example SQL for storing key-value pairs in vectorized groups: select (`property.value`[arrayFirstIndex(x->x=8, `property.id`)]*1+0)as value1, (`property.value`[arrayFirstIndex(x->x=7, `property.id`)]*-10+0)as value2 from test.productRealtimeData where transfer_date='2024-08-13' andproduct_id=2 and deviceId=4 and (value1>a and value1<=b).
[0082] During the implementation, 379,000 IoT data were constructed, each with 332 attributes and a data frequency of 1 Hz. The time consumption of multi-attribute joint query analysis in the document storage mode C4 and the time consumption of multi-attribute joint query analysis in the vectorized group storage mode C4: C = 5.58s: 0.159s = 35.1: 1. It can be seen that the query performance of the present invention has been greatly improved.
[0083] In actual business, the following scenarios have low access efficiency or occupy a large amount of storage space: (1) Dynamic attribute column collection of equipment: The collection and storage of equipment attributes are configurable. During normal operation, only a few necessary indicator data need to be collected and saved. During debugging and abnormal operation, a large amount of related attribute data needs to be added for data analysis and troubleshooting; (2) Variable frequency collection of equipment attributes: The frequency of change of different equipment business attributes is not constant, so the collection frequency is not consistent. In addition, a higher frequency is required during debugging and abnormal operation.
[0084] The present invention mainly includes three parts. The first part is to use a relational storage database to model the attributes of an IoT device; the second part is to use a column storage database to store vectorized group data of dynamic attribute keys and attribute values; and the third part is a data query method. By using a relational storage database in combination with a column storage database, the attribute master data (i.e., static attribute data) and the real-time data collected during operation (i.e., dynamic attribute data) are decoupled and separated, and a key-value vectorized combination method is used to store and read IoT data, thereby solving the problem that it is difficult to store dynamic data columns and variable-frequency storage data in the IoT system. At the same time, the storage space is greatly reduced, the system access efficiency is improved, and the operating cost is reduced. Among them, in the method described in the present invention, the static and unchanged master data is separated and decoupled from the dynamically changing collected data for storage, which reduces data redundancy, reduces storage space, and improves system access efficiency. The present invention performs binary serialization and compression on the stored data to reduce storage space and improve system access efficiency. The present invention stores and reads attribute keys and attribute values in a vectorized group manner, thereby solving the problem of storing dynamic data columns, variable-frequency storage data, and being unable to perform multi-attribute joint query and analysis. In addition, the present invention provides an efficient vectorized query method to improve query efficiency.
[0085] The following is a description of the IoT data access processing device provided by the present invention. The IoT data access processing device described below and the IoT data access processing method described above can be referred to in correspondence with each other. Figure 2 As shown, it is a schematic diagram of the structure of the Internet of Things data access processing device provided by the present invention. The Internet of Things data access processing device of the present invention specifically includes the following parts: The attribute key acquisition unit 201 is used to acquire an attribute key for querying the IoT data stored in a key-value pair vectorized group storage manner.
[0086] The conversion parameter acquisition unit 202 is used to obtain a mapping conversion coefficient and a mapping conversion offset corresponding to the attribute key from a relational storage database based on the device type of the Internet of Things device; wherein the relational storage database stores static attribute data of the Internet of Things device.
[0087] The attribute value search unit 203 is used to obtain the attribute value corresponding to the attribute key from the column storage database based on the mapping relationship between the attribute key and the preset key-value pair corresponding to the quantized group; the column storage database stores the dynamic attribute data of the Internet of Things device during operation; the attribute value is the physical value of the dynamic attribute data.
[0088] The attribute value processing unit 204 is used to obtain the business logic value of the Internet of Things data corresponding to the attribute value based on the mapping conversion coefficient and mapping conversion offset corresponding to the attribute key and the attribute value corresponding to the attribute key.
[0089] According to the Internet of Things data access processing device provided by the present invention, before obtaining the attribute key for querying the Internet of Things data stored in the key-value pair vectorized group storage mode, it also includes: Obtain static attribute data of the IoT device, and store the static attribute data of the IoT device in a relational storage database; wherein the static attribute data includes a device type identifier, a mapping conversion coefficient, and a mapping conversion offset; The dynamic attribute data of the IoT device during operation is obtained, and the dynamic attribute data is stored in a column storage database in a key-value pair quantized group storage manner; wherein the dynamic attribute data is the collected dynamically changing attribute data; the key-value pair quantized group includes an attribute key vector and an attribute value vector, the attribute key vector includes the physical values of static attribute data corresponding to multiple attribute keys, and the attribute value vector includes the physical values of dynamic attribute data corresponding to multiple attribute values.
[0090] The Internet of Things data access processing device provided by the present invention also includes: In a case where the attribute key is at least two attribute keys corresponding to at least two IoT data respectively, based on the device type of the IoT device, obtaining at least two mapping conversion coefficients and at least two mapping conversion offsets corresponding to the at least two attribute keys from a relational storage database; Based on the mapping relationship between the at least two attribute keys and the preset key-value pairs corresponding to the quantized group, obtaining at least two attribute values corresponding to the at least two attribute keys in the column storage database; Based on at least two mapping conversion coefficients and at least two mapping conversion offsets corresponding to the at least two attribute keys and at least two attribute values corresponding to the at least two attribute keys, at least two business logic values of the Internet of Things data corresponding to the at least two attribute values are obtained respectively.
[0091] According to the IoT data access processing device provided by the present invention, storing the dynamic attribute data in a column storage database specifically includes: Binary serialization and compression processing are performed on the dynamic attribute data to obtain compressed dynamic attribute data; The compressed dynamic attribute data is stored in a column storage database.
[0092] According to the IoT data access processing device provided by the present invention, the obtaining of the business logic value of the IoT data corresponding to the attribute value based on the mapping conversion coefficient and mapping conversion offset corresponding to the attribute key and the attribute value corresponding to the attribute key specifically includes: The mapping conversion coefficient and mapping conversion offset corresponding to the attribute key and the attribute value corresponding to the attribute key are input into a preset business logic value algorithm model for processing to obtain the business logic value of the Internet of Things data corresponding to the attribute value; wherein the business logic value algorithm model is pre-configured.
[0093] According to the IoT data access processing device provided by the present invention, the device type based on the IoT device, obtaining the mapping conversion coefficient and the mapping conversion offset corresponding to the attribute key in the relational storage database, specifically includes: The device type of the Internet of Things device is used as an index to search the relational storage database to obtain a mapping conversion coefficient and a mapping conversion offset corresponding to the index.
[0094] The IoT data access processing device provided by the present invention obtains the attribute key for querying IoT data stored in a key-value pair vectorized group storage mode; based on the device type of the IoT device, obtains the mapping conversion coefficient and mapping conversion offset corresponding to the attribute key in the relational storage database; based on the attribute key and the mapping relationship corresponding to the preset key-value pair vectorized group, obtains the attribute value corresponding to the attribute key in the column storage database; the column storage database stores the dynamic attribute data of the IoT device during operation; the attribute value is the physical value of the dynamic attribute data; based on the mapping conversion coefficient and mapping conversion offset corresponding to the attribute key and the attribute value corresponding to the attribute key, obtains the business logic value of the IoT data corresponding to the attribute value. The method of the present invention greatly reduces the storage space, can effectively improve the efficiency and accuracy of IoT data reading, and thus greatly reduces the operating cost.
[0095] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3As shown, the electronic device (i.e., pipeline damage point diagnosis and detection equipment) may include: a processor (processor) 301, a communication interface (CommunicationsInterface) 304, a memory (memory) 302 and a communication bus 303, wherein the processor 301, the communication interface 304, and the memory 302 communicate with each other through the communication bus 303. The processor 301 can call the logic instructions in the memory 302 to execute the Internet of Things data access processing method, which includes: obtaining an attribute key for querying the Internet of Things data stored in a key-value pair vectorized group storage method; based on the device type of the Internet of Things device, obtaining the mapping conversion coefficient and mapping conversion offset corresponding to the attribute key from the relational storage database; wherein the relational storage database stores static attribute data of the Internet of Things device; based on the attribute key and the mapping relationship corresponding to the preset key-value pair vectorized group, obtaining the attribute value corresponding to the attribute key from the column storage database; the column storage database stores the dynamic attribute data of the Internet of Things device during operation; the attribute value is the physical numerical value of the dynamic attribute data; based on the mapping conversion coefficient and mapping conversion offset corresponding to the attribute key and the attribute value corresponding to the attribute key, obtain the business logic value of the Internet of Things data corresponding to the attribute value.
[0096] In addition, the logic instructions in the above-mentioned memory 302 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0097] On the other hand, the present application also provides a computer program product, which includes a computer program, which can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the Internet of Things data access processing method provided by the above-mentioned methods, which includes: obtaining an attribute key for querying Internet of Things data stored in a key-value pair vectorized group storage manner; based on the device type of the Internet of Things device, obtaining a mapping conversion coefficient and a mapping conversion offset corresponding to the attribute key from a relational storage database; wherein the relational storage database stores static attribute data of the Internet of Things device; based on the attribute key and the mapping relationship corresponding to the preset key-value pair vectorized group, obtaining the attribute value corresponding to the attribute key from a column storage database; the column storage database stores dynamic attribute data of the Internet of Things device during operation; the attribute value is the physical numerical value of the dynamic attribute data; based on the mapping conversion coefficient and mapping conversion offset corresponding to the attribute key and the attribute value corresponding to the attribute key, obtain the business logic value of the Internet of Things data corresponding to the attribute value.
[0098] On the other hand, the present application also provides a computer-readable storage medium, which includes a stored program, wherein the program executes the Internet of Things data access processing method provided by the above methods when it is running, and the method includes: obtaining an attribute key for querying the Internet of Things data stored in a key-value pair vectorized group storage manner; based on the device type of the Internet of Things device, obtaining a mapping conversion coefficient and a mapping conversion offset corresponding to the attribute key from a relational storage database; wherein the relational storage database stores static attribute data of the Internet of Things device; based on the attribute key and the mapping relationship corresponding to the preset key-value pair vectorized group, obtaining the attribute value corresponding to the attribute key from a column storage database; the column storage database stores dynamic attribute data of the Internet of Things device during operation; the attribute value is the physical numerical value of the dynamic attribute data; based on the mapping conversion coefficient and mapping conversion offset corresponding to the attribute key and the attribute value corresponding to the attribute key, obtain the business logic value of the Internet of Things data corresponding to the attribute value.
[0099] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0100] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A method for accessing and processing Internet of Things data, characterized in that: include: Get the attribute key used to query IoT data stored in key-value vectorized group storage; Based on the device type of the IoT device, obtain a mapping conversion coefficient and a mapping conversion offset corresponding to the attribute key from a relational storage database; wherein the relational storage database stores static attribute data of the IoT device; Based on the mapping relationship between the attribute key and the preset key-value pair corresponding to the quantized group, the attribute value corresponding to the attribute key is obtained from the column storage database; the column storage database stores the dynamic attribute data of the IoT device during operation; the attribute value is the physical value of the dynamic attribute data; Based on the mapping conversion coefficient and the mapping conversion offset corresponding to the attribute key and the attribute value corresponding to the attribute key, the business logic value of the Internet of Things data corresponding to the attribute value is obtained.
2. The method for accessing and processing Internet of Things data according to claim 1, characterized in that: Before obtaining the attribute key for querying IoT data stored in a key-value vectorized group storage, it also includes: Obtain static attribute data of the IoT device, and store the static attribute data of the IoT device in a relational storage database; wherein the static attribute data includes a device type identifier, a mapping conversion coefficient, and a mapping conversion offset; The dynamic attribute data of the IoT device during operation is obtained, and the dynamic attribute data is stored in a column storage database in a key-value pair quantized group storage manner; wherein the dynamic attribute data is the collected dynamically changing attribute data; the key-value pair quantized group includes an attribute key vector and an attribute value vector, the attribute key vector includes the physical values of static attribute data corresponding to multiple attribute keys, and the attribute value vector includes the physical values of dynamic attribute data corresponding to multiple attribute values.
3. The method for accessing and processing Internet of Things data according to claim 1, characterized in that: Also includes: In a case where the attribute key is at least two attribute keys corresponding to at least two IoT data respectively, based on the device type of the IoT device, obtaining at least two mapping conversion coefficients and at least two mapping conversion offsets corresponding to the at least two attribute keys from a relational storage database; Based on the mapping relationship between the at least two attribute keys and the preset key-value pairs corresponding to the quantized group, obtaining at least two attribute values corresponding to the at least two attribute keys in the column storage database; Based on at least two mapping conversion coefficients and at least two mapping conversion offsets corresponding to the at least two attribute keys and at least two attribute values corresponding to the at least two attribute keys, at least two business logic values of the Internet of Things data corresponding to the at least two attribute values are obtained respectively.
4. The method for accessing and processing Internet of Things data according to claim 2, characterized in that: The storing the dynamic attribute data in a column storage database specifically includes: Binary serialization and compression processing are performed on the dynamic attribute data to obtain compressed dynamic attribute data; The compressed dynamic attribute data is stored in a column storage database.
5. The Internet of Things data access processing method according to claim 1, characterized in that: The obtaining, based on the mapping conversion coefficient and the mapping conversion offset corresponding to the attribute key and the attribute value corresponding to the attribute key, the business logic value of the Internet of Things data corresponding to the attribute value specifically includes: The mapping conversion coefficient and mapping conversion offset corresponding to the attribute key and the attribute value corresponding to the attribute key are input into a preset business logic value algorithm model for processing to obtain the business logic value of the Internet of Things data corresponding to the attribute value; wherein the business logic value algorithm model is pre-configured.
6. The Internet of Things data access processing method according to claim 1, characterized in that: The device type based on the IoT device, obtaining the mapping conversion coefficient and the mapping conversion offset corresponding to the attribute key from the relational storage database, specifically includes: The device type of the Internet of Things device is used as an index to search the relational storage database to obtain a mapping conversion coefficient and a mapping conversion offset corresponding to the index.
7. An IoT data access processing device, characterized in that: include: An attribute key acquisition unit, used for acquiring an attribute key for querying IoT data stored in a key-value pair vectorized group storage manner; A conversion parameter acquisition unit, used to acquire a mapping conversion coefficient and a mapping conversion offset corresponding to the attribute key from a relational storage database based on the device type of the Internet of Things device; wherein the relational storage database stores static attribute data of the Internet of Things device; An attribute value search unit is used to obtain an attribute value corresponding to the attribute key from a column storage database based on the mapping relationship between the attribute key and the preset key-value pair corresponding to the quantized group; the column storage database stores dynamic attribute data of the IoT device during operation; the attribute value is a physical value of the dynamic attribute data; The attribute value processing unit is used to obtain the business logic value of the Internet of Things data corresponding to the attribute value based on the mapping conversion coefficient and mapping conversion offset corresponding to the attribute key and the attribute value corresponding to the attribute key.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the Internet of Things data access processing method as described in any one of claims 1 to 6 is implemented.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the Internet of Things data access processing method as described in any one of claims 1 to 6 is implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the Internet of Things data access processing method as described in any one of claims 1 to 6 is implemented.