Distributed storage and retrieval method and system for spatiotemporal big data for Industry 4.0

Abstract

The present invention provides a spatiotemporal big data distributed storage and retrieval method for Industry 4.0, characterized by comprising: a data receiving step: receiving spatiotemporal big data generated in real time by a narrowband Internet of Things; a spatiotemporal big data layer storage step: storing the spatiotemporal big data layer in ElasticSearch; a spatiotemporal big data layer retrieval step: retrieving the spatiotemporal big data layer; a spatiotemporal big data storage step: storing the spatiotemporal big data in ElasticSearch; a spatiotemporal big data retrieval step: retrieving the spatiotemporal big data in ElasticSearch; a spatiotemporal process big data storage step: storing the spatiotemporal process big data in ElasticSearch; a spatiotemporal process big data retrieval step: retrieving the spatiotemporal process big data in ElasticSearch. The present invention makes full use of the high performance of real-time search of ElasticSearch, and greatly improves the spatiotemporal big data storage and retrieval performance.

Description

Technical Field

[0001] The present invention relates to the field of big data storage and retrieval technology, and in particular to a spatiotemporal big data distributed storage and retrieval method and system for Industry 4.0. Background Art

[0002] Spatiotemporal big data is big data with both time and space dimensions. On September 4, 2019, the Ministry of Industry and Information Technology compiled the "Guiding Opinions on the Development of Industrial Big Data (Draft for Comments)". The "Draft for Comments" requires the promotion of the application of technologies such as 5G and NB-IoT in industrial scenarios. Narrow Band Internet of Things (NB-IoT) is an important branch of the Internet of Everything (IoT) network, and the era of the Internet of Everything has arrived. A huge change in the era of the Internet of Everything is that compared with traditional big data, big data with time and space dimensions has increased dramatically. Exploring more effective distributed storage and retrieval methods for spatiotemporal big data has become an important issue that needs to be solved urgently.

[0003] In the field of big data storage technology, the industry basically uses Hadoop distributed file system HDFS for distributed storage. Neither HBASE nor HIVE takes into account the temporal and spatial characteristics of big data, nor optimizes the distributed storage of spatiotemporal big data.

[0004] In the field of big data retrieval technology, there are usually three solutions in the industry: solutions based on the open source project ElasticSearch, solutions based on the open source project Solr, and solutions for self-developed retrieval algorithms. Self-developed retrieval algorithms have high development and maintenance costs, and are not necessarily higher than the retrieval performance of open source projects. Solr is widely used in the Hadoop ecosystem. Solr focuses on text search, while Elasticsearch is often used for query, filtering, and group analysis and statistics. The technical positioning of Elasticsearch also makes this query more efficient. In addition, although Solr queries quickly, it is slow to update the index (i.e., slow insertion and deletion). For the massive spatiotemporal big data generated in real time by the narrowband Internet of Things in the era of the Internet of Everything, the Solr distributed retrieval solution cannot provide real-time retrieval services. Since the implementation technical solution of the present invention not only needs to provide efficient text search, but also requires complex time series search and aggregation capabilities, Elasticsearch is a more reasonable and effective choice.

[0005] Patent document CN105426491A (application number 201510818539.2) discloses a retrieval method and system for spatiotemporal geographic big data. When storing spatiotemporal data, a spatiotemporal code is generated according to the spatiotemporal data and then stored in a database. The spatiotemporal data includes spatial attributes and time attributes. When generating the spatiotemporal code, the range of the leaf nodes of the binary tree of the three dimensions of longitude, latitude and time is calculated in sequence, and the binary codes of longitude, latitude and time are cross-coded; when performing a spatiotemporal range query, the corresponding spatiotemporal code is converted and generated according to the time and space range to be queried, and then the query result is obtained by performing string fuzzy matching with the spatiotemporal code stored in the database. The present invention is based on the index provided by ElasticSearch, while the patent is based on the index provided by Oracle NoSQL. The present invention uses GUID to uniquely identify resource directories, spatiotemporal big data layers, spatiotemporal big data, and spatiotemporal process big data, while the patent cross-codes the binary codes of longitude, latitude and time. The retrieval method of the present invention is based on the Search APIs provided by ElasticSearch, while the patent creates its own retrieval method.

[0006] Patent document CN103198126B (application number 201310121937.X) discloses a method for managing spatiotemporal data of the Internet of Things, including a storage method for sensor sampling data and a retrieval method for sensor sampling data. The storage method for sampling data includes: the sensor issues a storage request; the control node generates a unique ID number of the sensor in the storage system, and establishes a mapping relationship between the sensor and the storage node; the sensor sends sampling data to the corresponding storage node according to the mapping relationship; the storage node stores the sampling time, sampling data, etc. The retrieval method for sampling data includes: the client transmits retrieval information to the control node; the control node determines whether the sensor ID is empty after receiving the retrieval information; the control node extracts the mapping relationship between the sensor and the storage node; the control node sends the retrieval information to the corresponding storage node based on the extracted mapping relationship; the storage node extracts the sampling data within the corresponding time point or time range through the time information, etc. The present invention is based on the distributed storage and retrieval provided by ElasticSearch, which is different from the patent. The retrieval method of the present invention is based on the SearchAPIs provided by ElasticSearch, while the patent creates its own retrieval algorithm.

[0007] Patent document CN103425789A (application number 201310382312.9) discloses a method and device for querying spatiotemporal data, the method comprising: obtaining statement information input by a user, when a spatiotemporal keyword identifier is identified in the statement information, judging the type of the statement information according to the type identifier keyword following the spatiotemporal keyword identifier; when the type of the statement information is a query category, extracting spatial information keywords and temporal information keywords from the statement information, and performing query processing according to the spatial information keywords and temporal information keywords. The device comprises: an acquisition judgment module, an extraction module, and a query processing module. The present invention is based on the distributed storage and retrieval provided by ElasticSearch, which is different from the patent. The retrieval method of the present invention is based on the Search APIs provided by ElasticSearch, while the patent creates its own retrieval algorithm, which adds spatiotemporal elements by expanding on the basis of the traditional SQL language.

[0008] Patent document CN109947354A (application number: 201711383412.8) discloses an Agent-based decentralized heterogeneous spatiotemporal data storage system and method, the system comprising: a storage area establishment module, a storage area scheduling module and a storage area management module; the storage area establishment module is used to establish a number of storage areas, the storage areas are directories, services or devices with storage capabilities, and each storage area runs an Agent service; the storage scheduling module is used to retrieve a suitable storage area as the storage area of ​​the data according to the storage attributes of the spatiotemporal data, and run the Agent service of the storage area to perform related operations; the storage area management module is used to uniformly manage the storage areas.

[0009] The present invention optimizes the time and space characteristics of massive spatiotemporal big data generated in real time by narrowband Internet of Things in the era of Internet of Everything, and proposes a distributed storage and retrieval method for spatiotemporal big data to solve the problems arising from the application of 5G, NB-IoT and other technologies in industrial scenarios. Summary of the invention

[0010] In view of the defects in the prior art, the purpose of the present invention is to provide a distributed storage and retrieval method and system for spatiotemporal big data for Industry 4.0.

[0011] The distributed storage and retrieval method of spatiotemporal big data for Industry 4.0 provided by the present invention includes:

[0012] Data receiving step: receiving the spatiotemporal big data generated in real time by the narrowband Internet of Things;

[0013] Steps for storing spatiotemporal big data layers: Store spatiotemporal big data layers in ElasticSearch;

[0014] Steps for retrieving spatiotemporal big data layers: Retrieve spatiotemporal big data layers;

[0015] Spatiotemporal big data storage steps: store spatiotemporal big data in ElasticSearch;

[0016] Steps for retrieving spatiotemporal big data: Retrieve spatiotemporal big data in ElasticSearch;

[0017] Steps for storing spatiotemporal process big data: Store spatiotemporal process big data in ElasticSearch;

[0018] Steps for retrieving spatiotemporal process big data: Retrieve spatiotemporal process big data in ElasticSearch.

[0019] Preferably, the step of storing the spatiotemporal big data layer includes:

[0020] Step A1: construct a spatiotemporal big data layer JSON object and a GUID of the spatiotemporal big data layer JSON object, set the layer type of the spatiotemporal big data layer JSON object to a resource directory or a layer type including points, lines, and surfaces, and set the resource path of the spatiotemporal big data layer JSON object;

[0021] Step A2: Construct an ElasticSearch IndexRequest object, set the GUID of the spatiotemporal big data layer JSON object as the id of the ElasticSearch IndexRequest object, and set the spatiotemporal big data layer JSON object as the source of the ElasticSearch IndexRequest object;

[0022] Step A3: Call the index or indexAsync function of ElasticSearch with the ElasticSearch IndexRequest object as a parameter;

[0023] Step A4: Store the spatiotemporal big data layer in ElasticSearch.

[0024] Preferably, the spatiotemporal big data layer retrieval step includes:

[0025] Step B1: Construct ElasticSearch Search Request object and Search Source Builder object;

[0026] Step B2: Sort the Search Source Builder objects in ascending order according to the resource path, and set the query conditions and the maximum number of records to be returned;

[0027] Step B3: Call the source function of the ElasticSearch SearchRequest object with the Search Source Builder object as a parameter;

[0028] Step B4: Take the ElasticSearch Search Request object as a parameter, call the search or searchAsync function of ElasticSearch, and retrieve the resource directory and spatiotemporal big data layer in ElasticSearch;

[0029] Step B5: loop the list of spatiotemporal big data layers retrieved: if the resource type of the JSON object is a resource directory, find the parent node of the JSON in the resource tree according to the resource path, add a child node to the parent node, and set the child node as the current resource directory; if the resource type of the JSON object is other types, add a leaf node to the current resource directory of the resource tree; when the loop of the retrieved list ends, the resource tree is established;

[0030] Step B6: Perform search based on the established resource tree.

[0031] Preferably, the spatiotemporal big data storage step includes:

[0032] Step C1: receiving spatiotemporal big data generated in real time by the narrowband Internet of Things;

[0033] Step C2: construct a spatiotemporal big data JSON object, set the GUID of the spatiotemporal big data JSON object, the GUID of the spatiotemporal big data layer of the spatiotemporal big data JSON object, the time timestamp and spatial location geo_point of the spatiotemporal big data JSON object;

[0034] Step C3: construct an ElasticSearch IndexRequest object, set the GUID of the spatiotemporal big data JSON object as the id of the ElasticSearch IndexRequest object, and set the spatiotemporal big data JSON object as the source of the ElasticSearch IndexRequest object;

[0035] Step C4: Call the index or indexAsync function of ElasticSearch with the ElasticSearch IndexRequest object as a parameter;

[0036] Step C5: Store spatiotemporal big data in ElasticSearch.

[0037] Preferably, the spatiotemporal big data retrieval step includes:

[0038] Step D1: Construct ElasticSearch SearchRequest object and SearchSourceBuilder object;

[0039] Step D2: Set the query conditions, sorting method, and maximum number of returned records of the SearchSourceBuilder object;

[0040] Step D3: Call the source function of the ElasticSearchSearchRequest object with the SearchSourceBuilder object as a parameter;

[0041] Step D4: Call the search or searchAsync function of ElasticSearch with the ElasticSearch SearchRequest object as a parameter;

[0042] Step D5: Retrieve spatiotemporal big data in ElasticSearch.

[0043] Preferably, the spatiotemporal process big data storage step includes:

[0044] Step E1: receiving spatiotemporal big data generated in real time by the narrowband Internet of Things;

[0045] Step E2: construct a spatiotemporal process big data JSON object, set the GUID of the spatiotemporal process big data JSON object, set the spatiotemporal big data layer GUID of the spatiotemporal process big data JSON object, set the time timestamp and spatial location geo_point of the spatiotemporal process big data JSON object;

[0046] Step E3: construct an ElasticSearch IndexRequest object, set the GUID of the spatiotemporal process big data JSON object as the id of the ElasticSearch IndexRequest object, and set the spatiotemporal process big data JSON object as the source of the ElasticSearch IndexRequest object;

[0047] Step E4: Call the index or indexAsync function of ElasticSearch with the ElasticSearch IndexRequest object as a parameter;

[0048] Step E5: Store the spatiotemporal process big data in ElasticSearch.

[0049] Preferably, the spatiotemporal process big data retrieval step includes:

[0050] Step F1: Construct ElasticSearch SearchRequest object and SearchSourceBuilder object;

[0051] Step F2: Set the query conditions, sorting method, and maximum number of returned records of the SearchSourceBuilder object;

[0052] Step F3: Call the source function of the ElasticSearchSearchRequest object with the SearchSourceBuilder object as a parameter;

[0053] Step F4: Call the search or searchAsync function of ElasticSearch with the ElasticSearch SearchRequest object as a parameter;

[0054] Step F5: Retrieve spatiotemporal process big data in ElasticSearch.

[0055] Preferably, the resource directory is a type of spatiotemporal big data layer, and the difference between the resource directory and the spatiotemporal big data layer is only that the value of the type field of the resource directory JSON object and the spatiotemporal big data layer JSON object is different; before storing the spatiotemporal big data layer, check whether the resource directory has been stored;

[0056] The spatiotemporal big data layer and resource directory classify and manage the spatiotemporal big data.

[0057] Preferably, the spatiotemporal big data and the spatiotemporal big data layer are in a many-to-one relationship;

[0058] The space-time process big data and the space-time big data are in a many-to-one relationship.

[0059] The spatiotemporal big data distributed storage and retrieval system for Industry 4.0 provided by the present invention includes:

[0060] Data receiving module: receiving the spatiotemporal big data generated in real time by the narrowband Internet of Things;

[0061] Spatiotemporal big data layer storage module: stores spatiotemporal big data layers in ElasticSearch;

[0062] Spatiotemporal big data layer retrieval module: retrieve spatiotemporal big data layers;

[0063] Spatiotemporal big data storage module: stores spatiotemporal big data in ElasticSearch;

[0064] Spatiotemporal big data retrieval module: retrieve spatiotemporal big data in ElasticSearch;

[0065] Spatiotemporal process big data storage module: stores spatiotemporal process big data in ElasticSearch;

[0066] Spatiotemporal process big data retrieval module: retrieve spatiotemporal process big data in ElasticSearch.

[0067] Compared with the prior art, the present invention has the following beneficial effects:

[0068] 1. Hadoop is the most popular offline data processing platform, while ElasticSearch is committed to providing real-time search. The present invention stores spatiotemporal big data in ElasticSearch instead of Hadoop, making full use of the high performance of ElasticSearch real-time search, greatly improving the spatiotemporal big data storage and retrieval performance. At the same time, in view of the defect that Solr is fast in query but slow in updating indexes (i.e., slow in insertion and deletion), ElasticSearch has been improved to greatly improve the performance of updating indexes, meeting the needs of real-time storage and retrieval of spatiotemporal big data.

[0069] 2. The present invention makes full use of the ElasticSearch cluster for distributed storage and retrieval of spatiotemporal big data. When the amount of spatiotemporal big data is small, the number of ElasticSearch nodes can be reduced. When the spatiotemporal big data reaches the level of massive data, the number of ElasticSearch nodes can be increased to thousands or even tens of thousands without sacrificing the storage and retrieval performance of spatiotemporal big data.

[0070] 3. The present invention introduces the concepts of spatiotemporal big data layers and resource directories to classify and manage spatiotemporal big data. Spatiotemporal big data with the same attributes are assigned to one spatiotemporal big data layer, which greatly improves the storage and retrieval performance, further improving the storage and retrieval performance of spatiotemporal big data.

[0071] 4. The present invention creatively abstracts the spatiotemporal process big data, stores the spatiotemporal big data reflecting the latest status separately from the spatiotemporal process big data reflecting the process, and optimizes the storage and retrieval specifically for the different characteristics of the two types of data, further improving the storage and retrieval performance of the spatiotemporal big data and the spatiotemporal process big data.

[0072] 5. The purpose of the present invention is to overcome the shortcomings in the traditional big data field and provide a distributed storage and retrieval method for spatiotemporal big data, which makes full use of the characteristics of time and space, improves the storage and retrieval efficiency of massive spatiotemporal big data, and thus meets the storage and retrieval requirements of massive spatiotemporal big data generated in real time by narrowband Internet of Things. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0074] Figure 1 It is a relationship diagram between resource directory, spatiotemporal big data layer, spatiotemporal big data, and spatiotemporal process big data;

[0075] Figure 2 Flowchart for storing spatiotemporal big data;

[0076] Figure 3 Storing flow chart for spatiotemporal process big data. DETAILED DESCRIPTION

[0077] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0078] The distributed storage and retrieval method of spatiotemporal big data for Industry 4.0 provided by the present invention includes:

[0079] Based on the ElasticSearch distributed storage system, the distributed storage system is divided into three parts: spatiotemporal big data layer and resource directory, spatiotemporal big data and spatiotemporal process big data, including: storage method and retrieval method of spatiotemporal big data layer and resource directory, storage method and retrieval method of spatiotemporal big data, and storage method and retrieval method of spatiotemporal process big data. The introduction of spatiotemporal big data layer and resource directory is to better manage spatiotemporal big data and classify spatiotemporal big data. The process data is abstracted separately as spatiotemporal process big data in order to optimize storage and retrieval according to the characteristics of process data. For each object, spatiotemporal process big data reflects the time, space, and state changes of the object, such as the vehicle driving trajectory within a certain time period, the elbow joint movement trajectory of an industrial robot completing a specific industrial process, etc.; spatiotemporal big data reflects the latest state of each object, and each data update overwrites the old data, that is, only the latest data is retained, such as the current position of the vehicle and the current position of the elbow joint of the industrial robot.

[0080] The resource directory is a special spatiotemporal big data layer, and the layer type of the spatiotemporal big data layer JSON object is a resource directory; after receiving the spatiotemporal big data (including time, location, status and other attributes) generated in real time by the narrowband Internet of Things, the spatiotemporal big data and the spatiotemporal process big data are stored in the ElasticSearch distributed storage system at the same time according to the storage method of spatiotemporal big data and the storage method of spatiotemporal process big data in the technical solution of the present invention.

[0081] The steps of storing the spatiotemporal big data layer and resource directory include: storing the spatiotemporal big data layer and resource directory in ElasticSearch instead of in Hadoop; each spatiotemporal big data layer or resource directory is stored as a JSON (JavaScript Object Notation, a lightweight data exchange format) object; the JSON object has attributes such as layer type (resource directory, point, line, surface, etc.), resource path, unique GUID, and layer parameters; setting the JSON object to ElasticSearch The source of IndexRequest and the id of ElasticSearchIndexRequest are set to the GUID of the JSON object. As a resource, the spatiotemporal big data layer organizes resources in a tree directory structure and is stored in the resource path field of the JSON object using a unified resource path expression. The above-mentioned unified resource path expression is defined as / first-level path GUID / second-level path GUID / …… / N-level path GUID (for example, / 5F9619FF-8B86-D011-B42D-00C04FC964FF / 6F9619FF-8B86-D011-B42D-00C04FC964FF / 7F9619FF-8B86-D011-B42D-00C04FC964FF). The resource directory is stored as a special spatiotemporal big data layer (layer type is resource directory), which specifically includes the following steps:

[0082] Step 101, check whether the resource directories at all levels in the uniform resource path expression have been created. If all the resource directories at all levels in the uniform resource path expression have been created, jump to step 107;

[0083] Step 102, constructing a spatiotemporal big data layer JSON object, setting the GUID of the JSON object, setting the layer type of the JSON object to a resource directory, and setting the resource path of the JSON object;

[0084] Step 103, construct an ElasticSearch IndexRequest object, the parameter is the spatiotemporal big data layer index name, and the corresponding API call is as follows:

[0085] IndexRequest request = new IndexRequest (spatial-temporal big data layer index name);

[0086] Step 104, setting the GUID of the spatiotemporal big data layer JSON object constructed in step 102 to the id of the ElasticSearch IndexRequest object constructed in step 103;

[0087] Step 105, setting the spatiotemporal big data layer JSON object constructed in step 102 as the source of the ElasticSearch IndexRequest object constructed in step 103;

[0088] Step 106, using the ElasticSearch IndexRequest object constructed in step 103 as a parameter, calling the index or indexAsync function of ElasticSearch, storing the spatiotemporal big data layer in the ElasticSearch distributed storage system, and jumping to step 101;

[0089] Step 107, constructing a spatiotemporal big data layer JSON object, setting the GUID of the JSON object, setting the layer type of the JSON object, and setting the resource path of the JSON object;

[0090] Step 108, construct an ElasticSearch IndexRequest object;

[0091] Step 109, setting the GUID of the spatiotemporal big data layer JSON object constructed in step 107 to the id of the ElasticSearch IndexRequest object constructed in step 108;

[0092] Step 110, setting the spatiotemporal big data layer JSON object constructed in step 107 as the source of the ElasticSearch IndexRequest object constructed in step 108;

[0093] Step 111, using the ElasticSearch IndexRequest object constructed in step 108 as a parameter, calling the index or indexAsync function of ElasticSearch, and storing the spatiotemporal big data layer in the ElasticSearch distributed storage system.

[0094] The steps for storing or creating resource directories and spatiotemporal big data layers are the same. The only difference is that the values ​​of the type fields of the resource directory JSON object and the spatiotemporal big data layer JSON object are different. The former's type is directory, and the latter's type is point, Polyline, Polygon, etc.

[0095] The retrieval method of the spatiotemporal big data layer adopts the Search APIs provided by ElasticSearch to provide a distributed retrieval service for the spatiotemporal big data layer, and sorts in ascending order according to the resource path during retrieval. The specific steps include:

[0096] Step 201, construct an ElasticSearchSearchRequest object, the parameter is the spatiotemporal big data layer index name, and the corresponding API call is as follows:

[0097] SearchRequestsearchRequest = newSearchRequest (spatial-temporal big data layer index name);

[0098] Step 202, construct a SearchSourceBuilder object, the corresponding API call is as follows:

[0099] SearchSourceBuildersourceBuilder=newSearchSourceBuilder();

[0100] Step 203, set the SearchSourceBuilder object to sort in ascending order according to the resource path, and set other parameters such as the query condition and the maximum number of returned records. The corresponding API call is as follows:

[0101] sourceBuilder.query(QueryBuilders.matchAllQuery());

[0102] sourceBuilder.sort(newFieldSortBuilder(“ResourcePath”).order(SortOrder.ASC));

[0103] Step 204, using the SearchSourceBuilder object constructed in step 202 as a parameter, calls the source function of the SearchRequest object constructed in step 201. The corresponding API call is as follows:

[0104] searchRequest.source(sourceBuilder);

[0105] Step 205, using the ElasticSearch SearchRequest object constructed in step 201 as a parameter, calling the search or searchAsync function of ElasticSearch, and retrieving the resource directory and spatiotemporal big data layer from the ElasticSearch distributed storage system. The corresponding API call is as follows:

[0106] SearchResponsesearchResponse=client.search(searchRequestRequestOptions.DEFAULT);

[0107] Step 206, looping the resource directory and spatiotemporal big data layer retrieved in step 205, if the resource type of the JSON object is a resource directory, first find its parent node in the resource tree according to the resource path, and add a child node to the parent node, the child node is set as the current resource directory; if the resource type of the JSON object is other types such as point, line, surface, etc., then add a leaf node to the current resource directory in the resource tree. When the list retrieved in step 205 is looped, the resource tree is established.

[0108] The storage steps of the spatiotemporal big data include: the spatiotemporal big data is stored in ElasticSearch instead of Hadoop; each spatiotemporal big data is stored as a JSON object; the object has attributes such as spatiotemporal big data layer GUID, unique GUID, time timestamp, spatial location geo_point and other parameters; the JSON object is set as the source of ElasticSearch IndexRequest, and the id of ElasticSearch IndexRequest is set to the GUID of the JSON object; the spatiotemporal big data and the spatiotemporal big data layer are in a many-to-one relationship, and all spatiotemporal big data under each spatiotemporal big data layer are stored in an index of ElasticSearch; the naming rule of the above ElasticSearch index is prefix + spatiotemporal big data layer GUID, prefix + spatiotemporal big data layer GUID + suffix, or spatiotemporal big data layer GUID + suffix (for example, layer_8F9619FF_8B86_D011_B42D_00C04FC964FF), which specifically includes the following steps:

[0109] Step 301, after receiving the spatiotemporal big data (including time, location, status and other attributes) generated in real time by the narrowband Internet of Things, construct a spatiotemporal big data JSON object, set the GUID of the JSON object, set the spatiotemporal big data layer GUID of the JSON object, set the time timestamp and spatial location geo_point of the JSON object;

[0110] Step 302, construct an ElasticSearch IndexRequest object, the parameters of the IndexRequest constructor are prefix + spatiotemporal big data layer GUID, prefix + spatiotemporal big data layer GUID + suffix, or spatiotemporal big data layer GUID + suffix (for example, layer_8F9619FF_8B86_D011_B42D_00C04FC964FF);

[0111] IndexRequest constructor, corresponding API calls have the following three methods:

[0112] IndexRequest request = new IndexRequest ("arbitrary prefix" + spatiotemporal big data layer GUID);

[0113] IndexRequest request = new IndexRequest ("any prefix" + spatiotemporal big data layer GUID + "any suffix");

[0114] IndexRequest request = new IndexRequest (Spatial-temporal big data layer GUID + "arbitrary suffix");

[0115] Step 303, set the GUID of the spatiotemporal big data JSON object constructed in step 301 to the id of the ElasticSearch IndexRequest object constructed in step 302. The corresponding API call is as follows:

[0116] request.id (GUID of spatiotemporal big data JSON object);

[0117] Step 304, set the spatiotemporal big data JSON object constructed in step 301 as the source of the ElasticSearch IndexRequest object constructed in step 302. The corresponding API call is as follows:

[0118] request.source (spatiotemporal big data JSON object, XContentType.JSON)

[0119] Step 305, using the ElasticSearch IndexRequest object constructed in step 302 as a parameter, calling the index or indexAsync function of ElasticSearch, and storing the spatiotemporal big data in the ElasticSearch distributed storage system.

[0120] The retrieval method of the spatiotemporal big data adopts the Search APIs provided by ElasticSearch to provide a distributed retrieval service for spatiotemporal big data, which supports the retrieval of the time dimension of spatiotemporal big data, the retrieval of the space dimension of spatiotemporal big data, and the simultaneous retrieval of the time dimension and space dimension of spatiotemporal big data.

[0121] Step 401, construct an ElasticSearch SearchRequest object. If all spatiotemporal big data layers are to be searched, no parameters need to be passed into the SearchRequest constructor. If the search is limited to a certain spatiotemporal big data layer, the parameter of the SearchRequest constructor is the spatiotemporal big data index name. If the search is limited to multiple spatiotemporal big data layers, the parameters of the SearchRequest constructor are spatiotemporal big data index name 1, spatiotemporal big data index name 2, and spatiotemporal big data index name n. The spatiotemporal big data index name refers to "any prefix" + spatiotemporal big data layer GUID, "any prefix" + spatiotemporal big data layer GUID + "any suffix", or spatiotemporal big data layer GUID + "any suffix". This parameter is consistent with the parameter of the IndexRequest constructor in step 302.

[0122] Step 402, construct a SearchSourceBuilder object.

[0123] Step 403, set the query conditions, sorting method, maximum number of returned records and other parameters of the SearchSourceBuilder object. The query conditions can be spatial query conditions such as a certain area (such as Pudong New Area) or attribute query conditions such as a certain status (such as online status).

[0124] Step 404, using the SearchSourceBuilder object constructed in step 402 as a parameter, calling the source function of the SearchRequest object constructed in step 401.

[0125] Step 405, using the ElasticSearch SearchRequest object constructed in step 401 as a parameter, calling the search or searchAsync function of ElasticSearch to retrieve spatiotemporal big data from the ElasticSearch distributed storage system.

[0126] The storage step of the spatiotemporal process big data includes: the spatiotemporal process big data is stored in ElasticSearch instead of Hadoop; each spatiotemporal process big data is stored as a JSON object; the JSON object is set as the source of ElasticSearch IndexRequest, and the id of ElasticSearchIndexRequest is set as the GUID of the JSON object; the spatiotemporal process big data and the spatiotemporal big data are in a many-to-one relationship, and all spatiotemporal process big data under each spatiotemporal big data are stored in an index of ElasticSearch; the naming rule of the above ElasticSearch index is prefix + spatiotemporal big data GUID, prefix + spatiotemporal big data GUID + suffix, or spatiotemporal big data GUID + suffix (for example, poi_8F9619FF_8B86_D011_B42D_00C04FC964FF), which specifically includes the following steps:

[0127] Step 501, after receiving the spatiotemporal big data (including time, location, status and other attributes) generated in real time by the narrowband Internet of Things, construct a spatiotemporal process big data JSON object, set the GUID of the JSON object, set the spatiotemporal big data layer GUID of the JSON object, set the time timestamp and spatial location geo_point of the JSON object;

[0128] Step 502, construct an ElasticSearch IndexRequest object, the parameters of the IndexRequest constructor are prefix + spatiotemporal big data GUID, prefix + spatiotemporal big data GUID + suffix, or spatiotemporal big data GUID + suffix (for example, layer_8F9619FF_8B86_D011_B42D_00C04FC964FF);

[0129] Step 503, setting the GUID of the spatiotemporal process big data JSON object constructed in step 501 as the id of the ElasticSearch IndexRequest object constructed in step 502;

[0130] Step 504, setting the spatiotemporal process big data JSON object constructed in step 501 as the source of the ElasticSearch IndexRequest object constructed in step 502;

[0131] Step 505, using the ElasticSearch IndexRequest object constructed in step 502 as a parameter, calling the index or indexAsync function of ElasticSearch, and storing the spatiotemporal process big data into the ElasticSearch distributed storage system.

[0132] The retrieval method of the spatiotemporal process big data adopts the Search APIs provided by ElasticSearch to provide a distributed retrieval service for the spatiotemporal process big data, which supports the retrieval of the time dimension of the spatiotemporal process big data, the retrieval of the space dimension of the spatiotemporal process big data, and the simultaneous retrieval of the time dimension and space dimension of the spatiotemporal process big data.

[0133] Step 601, construct an ElasticSearch SearchRequest object. If all spatiotemporal big data are to be searched, the SearchRequest constructor does not need to pass in parameters; if the search is limited to a certain spatiotemporal big data, the parameter of the SearchRequest constructor is the spatiotemporal process big data index name; if the search is limited to multiple spatiotemporal big data, the parameters of the SearchRequest constructor are spatiotemporal process big data index name 1, spatiotemporal process big data index name 2, and spatiotemporal process big data index name n. Among them, the spatiotemporal process big data index name refers to "any prefix" + spatiotemporal big data GUID, "any prefix" + spatiotemporal big data GUID + "any suffix", or spatiotemporal big data GUID + "any suffix", and this parameter is consistent with the parameter of the IndexRequest constructor in step 502.

[0134] Step 602, construct a SearchSourceBuilder object.

[0135] Step 603, set the query conditions, sorting method, maximum number of returned records and other parameters of the SearchSourceBuilder object. The query conditions can be spatial query conditions such as a certain area (such as Pudong New Area), attribute query conditions such as a certain status (such as online status), or time query conditions such as a certain time period (such as within a month, last year).

[0136] Step 604, using the SearchSourceBuilder object constructed in step 602 as a parameter, calling the source function of the SearchRequest object constructed in step 601.

[0137] Step 605, using the ElasticSearch SearchRequest object constructed in step 601 as a parameter, calling the search or searchAsync function of ElasticSearch, and retrieving the spatiotemporal process big data from the ElasticSearch distributed storage system.

[0138] The Java High Level REST Client officially provided by ElasticSearch can be directly used and the coding can be implemented according to the detailed steps provided by the present invention.

[0139] Those skilled in the art know that, in addition to implementing the system, device and its various modules provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, the system, device and its various modules provided by the present invention can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing the method and structures within the hardware component.

[0140] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A distributed storage and retrieval method for spatiotemporal big data for Industry 4.0, characterized in that: include: Data receiving step: receiving the spatiotemporal big data generated in real time by the narrowband Internet of Things; Steps for storing spatiotemporal big data layers: Store spatiotemporal big data layers in ElasticSearch; Steps for retrieving spatiotemporal big data layers: Retrieve spatiotemporal big data layers; Spatiotemporal big data storage steps: store spatiotemporal big data in ElasticSearch; Steps for retrieving spatiotemporal big data: Retrieve spatiotemporal big data in ElasticSearch; Steps for storing spatiotemporal process big data: Store spatiotemporal process big data in ElasticSearch; Steps for retrieving big data of spatiotemporal processes: Retrieve big data of spatiotemporal processes in ElasticSearch; The step of storing the spatiotemporal big data layer includes: Step A1: construct a spatiotemporal big data layer JSON object and a GUID of the spatiotemporal big data layer JSON object, set the layer type of the spatiotemporal big data layer JSON object to a resource directory or a layer type including points, lines, and surfaces, and set the resource path of the spatiotemporal big data layer JSON object; Step A2: Construct an ElasticSearchIndexRequest object, set the GUID of the spatiotemporal big data layer JSON object as the id of the ElasticSearch IndexRequest object, and set the spatiotemporal big data layer JSON object as the source of the ElasticSearch IndexRequest object; Step A3: Call the index or indexAsync function of ElasticSearch with the ElasticSearch IndexRequest object as a parameter; Step A4: Store the spatiotemporal big data layer in ElasticSearch; The spatiotemporal big data layer retrieval step includes: Step B1: Construct ElasticSearch Search Request object and Search Source Builder object; Step B2: Sort the Search Source Builder objects in ascending order according to the resource path, and set the query conditions and the maximum number of records to be returned; Step B3: Call the source function of the ElasticSearch SearchRequest object with the Search Source Builder object as a parameter; Step B4: Take the ElasticSearch Search Request object as a parameter, call the search or searchAsync function of ElasticSearch, and retrieve the resource directory and spatiotemporal big data layer in ElasticSearch; Step B5: loop the list of spatiotemporal big data layers retrieved: if the resource type of the JSON object is a resource directory, find the parent node of the JSON in the resource tree according to the resource path, add a child node to the parent node, and set the child node as the current resource directory; if the resource type of the JSON object is other types, add a leaf node to the current resource directory of the resource tree; when the loop of the retrieved list ends, the resource tree is established; Step B6: Perform search based on the established resource tree.

2. According to the distributed storage and retrieval method of spatiotemporal big data for Industry 4.0 according to claim 1, the spatiotemporal big data storage step comprises: Step C1: receiving spatiotemporal big data generated in real time by the narrowband Internet of Things; Step C2: construct a spatiotemporal big data JSON object, set the GUID of the spatiotemporal big data JSON object, the GUID of the spatiotemporal big data layer of the spatiotemporal big data JSON object, the time timestamp and spatial location geo_point of the spatiotemporal big data JSON object; Step C3: construct an ElasticSearchIndexRequest object, set the GUID of the spatiotemporal big data JSON object as the id of the ElasticSearch IndexRequest object, and set the spatiotemporal big data JSON object as the source of the ElasticSearchIndexRequest object; Step C4: Call the index or indexAsync function of ElasticSearch with the ElasticSearch IndexRequest object as a parameter; Step C5: Store spatiotemporal big data in ElasticSearch.

3. According to the distributed storage and retrieval method of spatiotemporal big data for Industry 4.0 according to claim 1, the spatiotemporal big data retrieval step comprises: Step D1: Construct ElasticSearch SearchRequest object and SearchSourceBuilder object; Step D2: Set the query conditions, sorting method, and maximum number of returned records of the SearchSourceBuilder object; Step D3: Call the source function of the ElasticSearch SearchRequest object with the SearchSourceBuilder object as a parameter; Step D4: Call the search or searchAsync function of ElasticSearch with the ElasticSearch SearchRequest object as a parameter; Step D5: Retrieve spatiotemporal big data in ElasticSearch.

4. According to the distributed storage and retrieval method of spatiotemporal big data for Industry 4.0 according to claim 1, the spatiotemporal process big data storage step comprises: Step E1: receiving spatiotemporal big data generated in real time by the narrowband Internet of Things; Step E2: construct a spatiotemporal process big data JSON object, set the GUID of the spatiotemporal process big data JSON object, set the spatiotemporal big data layer GUID of the spatiotemporal process big data JSON object, set the time timestamp and spatial location geo_point of the spatiotemporal process big data JSON object; Step E3: construct an ElasticSearchIndexRequest object, set the GUID of the spatiotemporal process big data JSON object as the id of the ElasticSearch IndexRequest object, and set the spatiotemporal process big data JSON object as the source of the ElasticSearch IndexRequest object; Step E4: Call the index or indexAsync function of ElasticSearch with the ElasticSearch IndexRequest object as a parameter; Step E5: Store the spatiotemporal process big data in ElasticSearch.

5. According to the distributed storage and retrieval method of spatiotemporal big data for Industry 4.0 according to claim 1, the spatiotemporal process big data retrieval step comprises: Step F1: Construct ElasticSearch SearchRequest object and SearchSourceBuilder object; Step F2: Set the query conditions, sorting method, and maximum number of returned records of the SearchSourceBuilder object; Step F3: Call the source function of the ElasticSearch SearchRequest object with the SearchSourceBuilder object as a parameter; Step F4: Call the search or searchAsync function of ElasticSearch with the ElasticSearch SearchRequest object as a parameter; Step F5: Retrieve spatiotemporal process big data in ElasticSearch.

6. According to the distributed storage and retrieval method of spatiotemporal big data for Industry 4.0 of claim 1, the resource directory is a type of spatiotemporal big data layer, and the difference between the resource directory and the spatiotemporal big data layer is only that the value of the type field of the resource directory JSON object and the spatiotemporal big data layer JSON object is different; before storing the spatiotemporal big data layer, check whether the resource directory has been stored; The spatiotemporal big data layer and resource directory classify and manage the spatiotemporal big data.

7. According to the distributed storage and retrieval method of spatiotemporal big data for Industry 4.0 according to claim 1, the spatiotemporal big data and the spatiotemporal big data layer are in a many-to-one relationship; The space-time process big data and the space-time big data are in a many-to-one relationship.

8. A spatiotemporal big data distributed storage and retrieval system for Industry 4.0, characterized in that: include: Data receiving module: receiving the spatiotemporal big data generated in real time by the narrowband Internet of Things; Spatiotemporal big data layer storage module: stores spatiotemporal big data layers in ElasticSearch; Spatiotemporal big data layer retrieval module: retrieve spatiotemporal big data layers; Spatiotemporal big data storage module: stores spatiotemporal big data in ElasticSearch; Spatiotemporal big data retrieval module: retrieve spatiotemporal big data in ElasticSearch; Spatiotemporal process big data storage module: stores spatiotemporal process big data in ElasticSearch; Spatiotemporal process big data retrieval module: retrieve spatiotemporal process big data in ElasticSearch; The spatiotemporal big data layer storage module includes: Construct a spatiotemporal big data layer JSON object and the GUID of the spatiotemporal big data layer JSON object, set the layer type of the spatiotemporal big data layer JSON object to a resource directory or a layer type including points, lines, and surfaces, and set the resource path of the spatiotemporal big data layer JSON object; Construct an ElasticSearchIndexRequest object, set the GUID of the spatiotemporal big data layer JSON object as the id of the ElasticSearch IndexRequest object, and set the spatiotemporal big data layer JSON object as the source of the ElasticSearch IndexRequest object; Call the index or indexAsync function of ElasticSearch with the ElasticSearchIndexRequest object as a parameter; Store spatiotemporal big data layers in ElasticSearch; The spatiotemporal big data layer retrieval module includes: Construct ElasticSearch Search Request object and Search Source Builder object; Sort the Search Source Builder objects in ascending order according to the resource path, and set the query conditions and the maximum number of records to be returned; Call the source function of the ElasticSearch Search Request object with the Search Source Builder object as a parameter; Take the ElasticSearch Search Request object as a parameter, call the search or searchAsync function of ElasticSearch, and retrieve the resource directory and spatiotemporal big data layer in ElasticSearch; The list of retrieved spatiotemporal big data layers is looped: if the resource type of the JSON object is a resource directory, the parent node of the JSON is found in the resource tree according to the resource path, and a child node is added to the parent node, and the child node is set as the current resource directory; if the resource type of the JSON object is other types, a leaf node is added to the current resource directory of the resource tree; when the loop of the retrieved list ends, the resource tree is established; Search based on the established resource tree.

Images