Distributed data storage and retrieval system and method for industrial Internet of Things
Through the distributed data storage and retrieval system, the intelligent gateway acquisition module and multi-dimensional data processing framework are used to realize classified storage and efficient retrieval of industrial IoT data, solving the problems of data confusion and low retrieval efficiency, and improving the data processing speed and accuracy.
Patent Information
- Application Number
- CN202410912404.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-09
- Publication Date
- 2024-11-01
AI Technical Summary
It is difficult for the prior art to effectively store and retrieve complex data generated by the industrial Internet of Things, especially the classified storage and efficient retrieval of multiple types of data, resulting in data confusion and inefficient retrieval.
Adopt distributed data storage and retrieval system, including intelligent gateway acquisition module, data processing framework, data storage framework and data index framework. Through multi-dimensional data processing and classified storage, data communication protocol and acquisition driver are dynamically configured, and multiple data types are supported. storage and indexing, and use binary tree, B tree and other index methods to achieve rapid query.
It realizes the classification storage and efficient retrieval of a variety of industrial equipment data, accurately identify data types, reduces manual classification time, improves data processing speed, and is highly integrated in storage and indexing functions, has strong anti-interference and is easy to maintain.
Smart Images

Figure CN118885518A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and more particularly to a distributed data storage and retrieval system and method for industrial Internet of Things. Background Art
[0002] The Internet of Things (IoT) is a popular technology that is emerging both internationally and domestically. It is widely used in many fields such as intelligent transportation, basic measurement, land and resources survey, industrial detection, etc., and has great development and application prospects. However, there are still some limitations in the research and industrialization of the Industrial Internet of Things. The core problem is that the research on distributed storage and high-efficiency search technology for massive heterogeneous data is relatively limited.
[0003] Regarding data storage methods, in most cases, the acquired data is stored in a single repository with a large enough capacity for unified storage, such as a data lake. This method of storing data in a single repository can easily cause data confusion, resulting in the data not having relatively good classified storage characteristics, which seriously affects the classified storage measures of the data in the repository.
[0004] Regarding data indexing, although current indexing engine technologies such as directory indexing, keyword indexing, and self-media indexing have become relatively mature and have been very successfully applied, they are mainly aimed at static or slowly changing web page information that mainly consists of text information. They are not suitable for industrial Internet of Things environments where data changes frequently in time series and involves multiple types of data.
[0005] Therefore, how to propose a distributed data storage and retrieval system and method for the Industrial Internet of Things, realize a unified mechanism to store the complex data generated by the Industrial Internet of Things, and realize efficient retrieval of the stored data is an urgent problem that technicians in this field need to solve. Summary of the invention
[0006] In view of this, the present invention provides a distributed data storage and retrieval system and method for the industrial Internet of Things, which realizes a unified mechanism to store complex data generated by the industrial Internet of Things and realizes efficient retrieval of the stored data. In order to achieve the above purpose, the present invention adopts the following technical solutions:
[0007] A distributed data storage and retrieval system for industrial Internet of Things, comprising: industrial equipment, an intelligent gateway acquisition module and a multi-dimensional data storage and retrieval device, wherein the multi-dimensional data storage and retrieval device comprises a data processing framework, a data storage framework and a data indexing framework connected in sequence;
[0008] The intelligent gateway acquisition module is used to collect and convert the industrial equipment data in the data bus and package it to send it to the data processing framework;
[0009] The data processing framework is used to receive the identified data and classify and process the data;
[0010] The data storage framework is used to perform classified storage processing on the classified data;
[0011] The data index framework is used to configure storage indexes for data after classified storage processing, and to display data according to required query indexes.
[0012] Optionally, the intelligent gateway acquisition module dynamically configures data communication protocols and acquisition drivers according to different types of industrial equipment; during data acquisition, the dynamic configuration function in the corresponding driver is called according to the device information and acquisition information to obtain data; the acquisition cycle is customized according to the industrial equipment dimension, and the use of expressions for calculation is supported; after the data acquisition is completed, the data is spliced and sent to the multi-dimensional data storage and retrieval device.
[0013] Optionally, the data processing framework includes a collection data type identification module, a data classification module, a type configuration unit module, a data processing module and a storage configuration unit module which are connected in sequence.
[0014] Optionally, the collected data type identification module is used to identify the type of data collected by the intelligent gateway collection module according to data usage frequency, data format type and data usage occasion, and upload the data to the data classification module.
[0015] Optionally, the data classification module is used to call the configuration method in the module to classify the data after the data type identification module uploads the data, and upload different types of data to the data processing module through the type configuration unit function;
[0016] The data processing module is used to receive the classified data, and call the corresponding processing method to process the data according to the configured type, and upload the data in different storage formats to the data storage framework through the storage configuration unit module.
[0017] Optionally, the data storage framework includes a data storage module and a database connected in sequence, and the data storage module is used to store data in sequence to a corresponding data repository according to a storage unit configured for different data, and the different data include structured, unstructured, memory, relational, non-relational, time series and file type data.
[0018] Optionally, the storage mode of the data storage module is divided into five types of databases according to data usage frequency, usage location, business function, data type, and usage source, including: memory cache, memory database, hard disk database, file database, and external integrated publishing database;
[0019] According to the relationship between data and data characteristics, data storage methods are divided into relational data and non-relational data, and external integrated publishing databases are divided into publish-subscribe and queue information.
[0020] Optionally, the data index framework includes a storage index configuration function module, a data storage connection module and a data display module which are connected in sequence.
[0021] Optionally, the data storage connection module is used to configure the specific location of each data in the database through a storage index, and feed it back to the data storage connection module;
[0022] Indexing methods include binary tree, B-tree, B+ tree, B* tree, etc.
[0023] Optionally, a distributed data storage and retrieval method for industrial Internet of Things includes:
[0024] Industrial equipment, intelligent gateway acquisition module and multi-dimensional data storage and retrieval device, the multi-dimensional data storage and retrieval device includes a data processing framework, a data storage framework and a data index framework connected in sequence; the intelligent gateway acquisition module converts the industrial equipment data acquisition in the data bus and packages it to the data processing framework; the data processing framework receives the identified data and classifies the data; the data storage framework classifies and stores the classified data; the data index framework configures storage indexes for the classified and stored data, and displays the data according to the required query index.
[0025] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a distributed data storage and retrieval system and method for industrial Internet of Things, which has the following beneficial effects:
[0026] The present invention proposes a distributed data storage and retrieval system for an industrial Internet of Things, comprising: industrial equipment, an intelligent gateway acquisition module and a multidimensional data storage and retrieval device, wherein the multidimensional data storage and retrieval device comprises a data processing framework, a data storage framework and a data index framework connected in sequence; the intelligent gateway acquisition module is used to acquire and convert industrial equipment data in a data bus and package and send the data to the data processing framework; the data processing framework is used to receive the identified data and classify the data; the data storage framework is used to classify and store the classified data; the data index framework is used to configure a storage index for the classified and stored data and display the data according to the required query index. The present invention: 1) realizes classified storage of data of various industrial equipment, and can add more equipment that needs to store data by adjusting the configuration, and has strong versatility; 2) accurately identifies data types, analyzes data characteristics, and stores them in a designated database, and the data classification is reasonable and accurate; 3) performs distributed storage according to the type, usage frequency, and usage location of the collected industrial equipment data, and can also enter the designated database for retrieval according to the type of data when retrieving data, thereby improving the speed of industrial Internet of Things data processing; 4) there will be log records when each data is stored, and the speed and accuracy are high when indexing data; 5) the storage and indexing functions are highly integrated in the multi-dimensional data storage and indexer, which has strong anti-interference ability, small maintenance workload, and convenient maintenance. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0028] Figure 1 This is a structural framework diagram of a distributed data storage and retrieval system for the industrial Internet of Things provided by the present invention.
[0029] Figure 2 A schematic diagram of the data storage workflow provided by the present invention.
[0030] Figure 3 A schematic diagram of the index data workflow provided by the present invention.
[0031] Figure 4 This is a schematic diagram of the binary tree principle provided by the present invention. DETAILED DESCRIPTION
[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0033] The embodiment of the present invention discloses a distributed data storage and retrieval system of the industrial Internet of Things, including: industrial equipment, an intelligent gateway acquisition module and a multi-dimensional data storage and retrieval device, wherein the multi-dimensional data storage and retrieval device includes a data processing framework, a data storage framework and a data indexing framework connected in sequence;
[0034] The intelligent gateway acquisition module is used to collect and convert the industrial equipment data in the data bus and package it to send it to the data processing framework;
[0035] The data processing framework is used to receive the identified data and classify and process the data;
[0036] The data storage framework is used to perform classified storage processing on the classified data;
[0037] The data index framework is used to configure storage indexes for data after classified storage processing, and to display data according to required query indexes.
[0038] In a specific implementation, a distributed data storage and retrieval system applied to industrial Internet of Things, such as Figure 1 As shown, including:
[0039] Intelligent gateway acquisition module, acquisition data type identification module, data classification module, type configuration unit functional module, data processing module, storage configuration unit functional module, data storage module, database, storage index configuration functional module, data storage connection module, data display module.
[0040] The intelligent gateway acquisition module is used to convert and package all industrial equipment data in the data bus and send them to the multi-dimensional data storage and indexer; the acquisition data type identification module is used to analyze the data sent in a package, identify the type of each data and pass it to the next module; the data classification and processing module receives the identified data, and stores the data in different types of libraries by configuring the type unit and the storage unit; the data storage connection module finds the required data by configuring the storage index, and retrieves the data from the library where the data is located, and sends it to the data display module to display the data. The present invention can realize the distributed data storage and indexing function in the intelligent production line, avoid the data mixing problem caused by the storage of a single data type, reduce the manual time required for data classification, increase the speed of calling data, and improve production efficiency.
[0041] Furthermore, the built-in driver of the intelligent gateway acquisition module is a dynamic configuration data, which needs to be dynamically configured according to the different types of industrial equipment. When collecting data, the method in the corresponding driver is called according to the device information and the collection information. The collection cycle can be customized by the device dimension, and the expression is supported for basic calculation. After the data collection is completed, it is spliced and sent to the edge end.
[0042] Among them, the dynamically configurable data types include numeric type, Boolean type and character type. The numeric type includes integer type (Int), short integer type (Short), long integer type (Long), floating point type (Float), double precision floating point type (Double), Boolean type includes True (1) and False (0), and character type includes single character type (Char) and string type (String).
[0043] The dynamic expansion method of industrial equipment types is achieved by expanding the acquisition driver of the configuration equipment. Since the acquisition communication protocols of different equipment are different, it is necessary to flexibly expand the driver of different protocols. The present invention determines the functional structure of the protocol driver by defining a general driver interface class. The interface definition is as follows:
[0044]
[0045]
[0046]
[0047] All drivers that need to be extended must inherit this interface class. The extended driver class contains three types of attributes and methods: DiverInfoAttribute (responsible for identifying the driver class), ConfigParameterAttribute (responsible for identifying the driver attributes), and MethodAttribute (identifying the acquisition method in the driver). Examples are as follows:
[0048]
[0049]
[0050] The value returned by the method defined by MethodAttribute is the data collected by the device. In this way, the data collected by the device can be dynamically configured and expanded.
[0051] Furthermore, after completing the data collection on the device side, the collected data is serialized in JSON format according to the device + timestamp + point value, the data is spliced, and the serialized and spliced data is sent to the edge side. Each device has an independent thread for data transmission, and the data of different devices are independent during the upload process on the collection side.
[0052] Furthermore, the collected data type identification module is used to identify the type of data collected by the gateway according to data usage frequency, data format type, data usage occasion, etc., and upload the data to the data classification module.
[0053] The function of the data classification module is: after the data type recognition module uploads data, it calls the configuration method in the module to classify the data, and uploads different types of data to the data processing module through the type configuration unit function.
[0054] Among them, the configuration method in the module realizes the classification of data. The specific configuration classification method is carried out in the form of a binary tree, such as Figure 4 As shown, for the acquired data, first determine whether the data is real-time data (such as equipment status, processing program number, etc.) or non-real-time data (data used for subsequent statistical analysis, such as order number, processing time, production line rhythm, etc.), then determine whether it is relational data or non-relational data, and then store the corresponding data type based on whether it is structured.
[0055] The storage method of each type of data is configured through an XML file. After obtaining the data type, you need to read the XML file to obtain the specific storage data method and database name and location.
[0056] Furthermore, the data processing module is used to receive the classified data, and process the data by calling the corresponding processing method according to the configured type unit, and upload the data in different storage formats to the data storage module through the storage configuration unit function.
[0057] The function of the data processing module is to perform logical operations, merge multiple collected items or numerical operations on the collected data according to the configured rules, so as to convert the collected signals into business-meaningful data. Specifically, it includes the following forms:
[0058] Variable and value operations (single-point data processing configuration)
[0059]
[0060] Multi-point operation (multi-point data processing configuration)
[0061]
[0062] In addition, both single-point bit values and multi-point bits also include logical operations, including value range truncation, AND, OR, and NOT.
[0063] Furthermore, the data storage module is used to store data in sequence to corresponding data storage according to the storage units configured for different data, including structured, unstructured, memory, relational, non-relational, time series, files and other types.
[0064] The data storage methods are divided into five types according to the frequency of data use, place of use, business function, data type, source of use, etc., namely: memory cache, memory database, hard disk database, file database and external integrated release. According to the relationship between data and data characteristics, the data storage methods are divided into relational data and non-relational data, and the external integrated distribution database is divided into publish-subscribe and queue information. Furthermore, the data storage connection module is used to configure the storage index for the specific location of each data in the database and feed it back to the data storage connection module. Indexing methods include but are not limited to binary trees, B trees, B+ trees, and B* trees.
[0065] In a specific implementation, in view of the existing data classification, storage and indexing methods, most of which are based on manual work and have low efficiency, a distributed data storage and retrieval method applied to the industrial Internet of Things is proposed, which includes the following steps:
[0066] Step 1: If Figure 2 As shown, data is stored in categories:
[0067] S11: The stored data comes from sensors, I / O in hardware acquisition, PLC in point acquisition, industrial equipment, configuration software, quality inspection system, integrated interface and logistics system, etc. The data is parsed and packaged and sent to the data type recognition module through the driver that has been configured in the intelligent gateway.
[0068] S12: The source data is identified by the configured method, the data is configured to the corresponding type when passing through the type configuration unit function module, and then the configured data is passed to the data processing module for in-depth processing.
[0069] S13: The data processing module needs to further process the source data, identify the type unit in the data through the built-in method, call the corresponding method to process the data, perform the corresponding type configuration when the data passes through the storage configuration unit function module, and then pass the data to the data storage module.
[0070] S14: When the data is transferred to the data storage module, the corresponding storage unit is identified and the data is placed in one or more corresponding databases. Thus, a complete data distributed storage process ends.
[0071] Step 2: If Figure 3 As shown, data retrieval:
[0072] S21: Each data has a corresponding address when it is stored in the database. The index configuration in the data storage and indexer is called to locate the data address and pass the data to the storage data connection module.
[0073] S22: receiving the data sent by the data storage module, and connecting the data to the data display module.
[0074] S23: The data display module processes the received data into two parts, namely, equipment index analysis and production line index analysis, which are displayed on the web page interface through corresponding methods. Thus, a complete data indexing process ends.
[0075] In a specific embodiment, a distributed data storage and retrieval method for an industrial Internet of Things includes: industrial equipment, an intelligent gateway acquisition module, and a multidimensional data storage and retriever, wherein the multidimensional data storage and retriever includes a data processing framework, a data storage framework, and a data index framework connected in sequence; the intelligent gateway acquisition module acquires and converts industrial equipment data in a data bus and packages and sends the data to the data processing framework; the data processing framework receives the identified data and classifies the data; the data storage framework classifies and stores the classified data; the data index framework configures a storage index for the classified and stored data, and displays the data according to the required query index.
[0076] The present invention (1) realizes the classified storage of data of various industrial equipments, and can add more equipments that need to store data by adjusting the configuration, and has strong versatility; (2) accurately identifies the data type, analyzes the data characteristics, and stores it in the specified database, and the data classification is reasonable and accurate: (3) There will be a log record when each data is stored, and the speed and accuracy are high when indexing data; (4) The storage and indexing functions are highly integrated in the multi-dimensional data storage and indexer, with strong anti-interference ability, small maintenance workload, and convenient maintenance. Among them, the dynamic expansion method of industrial equipment types and data is realized by expanding the acquisition driver of the configuration equipment. Since the acquisition communication protocols of different equipment are different, it is necessary to flexibly expand the drivers of different protocols. Then, the acquisition method in the driver is expanded through the MethodAttribute in the expanded acquisition driver class, and the data returned by the acquisition method is the more data that needs to be stored after adjusting the configuration.
[0077] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0078] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A distributed data storage and retrieval system for industrial Internet of Things, It is characterized in that include: Industrial equipment, intelligent gateway acquisition module and multi-dimensional data storage and retrieval device, the multi-dimensional data storage and retrieval device includes a data processing framework, a data storage framework and a data indexing framework connected in sequence; The intelligent gateway acquisition module is used to collect and convert the industrial equipment data in the data bus and package it to send it to the data processing framework; The data processing framework is used to receive the identified data and classify and process the data; The data storage framework is used to perform classified storage processing on the classified data; The data index framework is used to configure storage indexes for data after classified storage processing, and to display data according to required query indexes.
2. According to the distributed data storage and retrieval system of the industrial Internet of Things according to claim 1, It is characterized in that The intelligent gateway acquisition module dynamically configures data communication protocols and acquisition drivers according to different types of industrial equipment; When collecting data, the dynamic configuration function in the corresponding driver is called according to the device information and collection information to obtain data; Customize the collection cycle through industrial equipment dimensions and support the use of expressions for calculations; After data collection is completed, it is spliced and sent to the multi-dimensional data storage and retrieval device.
3. According to the distributed data storage and retrieval system of the industrial Internet of Things in claim 1, It is characterized in that The data processing framework includes a collection data type identification module, a data classification module, a type configuration unit module, a data processing module and a storage configuration unit module which are connected in sequence.
4. A distributed data storage and retrieval system for industrial Internet of Things according to claim 3, It is characterized in that The collected data type identification module is used to identify the type of data collected by the intelligent gateway collection module according to the data usage frequency, data format type and data usage occasion, and upload the data to the data classification module.
5. A distributed data storage and retrieval system for industrial Internet of Things according to claim 4, It is characterized in that The data classification module is used to call the configuration method in the module to classify the data after the data is uploaded by the acquisition data type identification module, and upload different types of data to the data processing module through the type configuration unit function; The data processing module is used to receive the classified data, and call the corresponding processing method to process the data according to the configured type, and upload the data in different storage formats to the data storage framework through the storage configuration unit module.
6. The distributed data storage and retrieval system of the industrial Internet of Things according to claim 1, It is characterized in that The data storage framework includes a data storage module and a database connected in sequence. The data storage module is used to store data in sequence to corresponding data repositories according to storage units configured for different data. The different data include structured, unstructured, memory, relational, non-relational, time series and file type data.
7. A distributed data storage and retrieval system for industrial Internet of Things according to claim 6, It is characterized in that The storage mode of the data storage module is divided into five types of databases according to the frequency of data use, place of use, business function, data type, and source of use, including: memory cache, memory database, hard disk database, file database, and external integrated publishing database; According to the relationship between data and data characteristics, data storage methods are divided into relational data and non-relational data, and external integrated publishing databases are divided into publish-subscribe and queue information.
8. The distributed data storage and retrieval system of the industrial Internet of Things according to claim 1, It is characterized in that The data index framework includes a storage index configuration function module, a data storage connection module and a data display module which are connected in sequence.
9. A distributed data storage and retrieval system for industrial Internet of Things according to claim 8, It is characterized in that The data storage connection module is used to configure the specific location of each data in the database through the storage index configuration function, and feed back to the data storage connection module; Indexing methods include binary tree, B-tree, B+ tree and B* tree.
10. A distributed data storage and retrieval method for industrial Internet of Things, It is characterized in that include: Industrial equipment, intelligent gateway acquisition module and multi-dimensional data storage and retrieval device, the multi-dimensional data storage and retrieval device includes a data processing framework, a data storage framework and a data index framework connected in sequence; the intelligent gateway acquisition module converts the industrial equipment data acquisition in the data bus and packages it to the data processing framework; the data processing framework receives the identified data and classifies the data; the data storage framework classifies and stores the classified data; the data index framework configures storage indexes for the classified and stored data, and displays the data according to the required query index.
Citation Information
Patent Citations
Storage system of distributed real-time database
CN107229639A
Data concentration system of industrial Internet of Things and data aggregation, processing and storage method
CN112422663A
Industrial data acquisition and analysis system and method
CN115208914A
Edge Intelligence Platform, and Internet of Things Sensor Streams System
US20170060574A1
