Data management system and method for mobile internet of things cloud edge fusion scene

By adopting cloud-edge convergence architecture and mini Rosetta filters in the IoT data management platform, network congestion problems in IoT device data flow management are solved, efficient data management and query are achieved, and the upper limit of the number of devices is increased .

CN118939627APending Publication Date: 2024-11-12SHANGHAI JIAOTONG UNIV
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202310540030.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-12
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing IoT data management platform is difficult to efficiently manage the data flow of a large number of mobile IoT devices in scenarios where network bandwidth is limited, resulting in network congestion and system bottlenecks, limiting the upper limit of the number of devices.

Method used

Adopt a cloud-edge converged data management system for mobile Internet of Things to collect and store data through edge nodes, and optimize data query and migration using global index management module and mini Rosetta filters, reduce network overhead and improve data management efficiency.

Benefits of technology

Effectively reduce network overhead and query latency, improve service quality of data management, and efficiently manage mobile IoT devices in an environment where network, storage and computing resources are limited, avoid network blockage, and improve system scalability .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939627A_ABST
    Figure CN118939627A_ABST
Patent Text Reader

Abstract

The invention provides a data management system and method for a mobile internet of things cloud edge fusion scene. The data management system comprises an edge node end and a cloud data center end, the edge node end comprises a collection task deployment module and a data collection thread module; the cloud data center end comprises a data stream metadata management module and a global index management module; the collection task deployment module deploys a corresponding data collection thread by querying data stream metadata maintained by the data stream metadata management module and receiving mobile edge device data, and the data collection thread module analyzes the corresponding metadata and stores the metadata to a local database to finish the data collection. And the global index management module returns node information existing in the corresponding data according to the query instruction. According to the invention, the problem of network congestion caused by limited network bandwidth between the central node and the edge node is avoided as much as possible, and better service quality is provided for data management under the mobile edge network.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data management system and method for a mobile Internet of Things cloud-edge fusion scenario. Background Art

[0002] With the rapid development of mobile IoT technology, the number of mobile IoT devices has increased rapidly. In order to perform various management tasks on mobile IoT devices, it is necessary to build a mobile IoT data management platform to collect and analyze the status information uploaded by the devices in real time. Many existing IoT data management platforms choose to centrally manage all IoT devices through cloud data centers, which can provide highly scalable, highly fault-tolerant, and low-cost computing and storage services. However, centralized data management methods will put great pressure on the network bandwidth of the data center. In scenarios where network bandwidth is limited, this will become a bottleneck for the system and limit the scalability of the system.

[0003] Many existing studies have also proposed using edge data storage architecture to improve such problems. By distributing and storing data on edge nodes and providing data query and analysis functions on edge nodes, network overhead and query latency can be effectively reduced. However, these solutions generally lack the ability to efficiently store, range query, and aggregate analysis of streaming data, and cannot efficiently support data stream management tasks in mobile IoT scenarios.

[0004] The present invention is aimed at the scenario of mobile Internet of Things management. The service needs to manage the message data streams continuously uploaded by a large number of mobile Internet of Things devices. This scenario has the characteristics of a large number of messages, continuous data writing, device mobility, and complex data query. The real-time uploading and analysis and processing of large amounts of data put forward higher requirements on the network, storage and computing resources of the cloud data center. When resources are insufficient, the service quality of various real-time data analysis applications that rely on data will be affected. Especially in some environments where network bandwidth is limited and network congestion is prone to occur, such as unmanned aerial vehicle self-organizing network environment and marine satellite network, data transmission becomes the bottleneck of the system, limiting the upper limit of the number of devices that can be managed simultaneously.

[0005] Therefore, the market is in urgent need of a method and system for efficient cloud-edge fusion data management for mobile Internet of Things with lower network overhead. Summary of the invention

[0006] In view of the defects in the prior art, the object of the present invention is to provide a data management system and method for mobile Internet of Things cloud-edge fusion scenarios.

[0007] A data management system for a mobile Internet of Things cloud-edge fusion scenario provided by the present invention includes: an edge node end and a cloud data center end;

[0008] The edge node end includes a collection task deployment module and a data collection thread module; the cloud data center end includes a data flow metadata management module and a global index management module;

[0009] The collection task deployment module deploys the corresponding data collection thread by querying the data stream metadata maintained by the data stream metadata management module and receiving the mobile edge device data. The data collection thread module parses the corresponding metadata and stores it in the local database. The global index management module returns the node information where the corresponding data exists according to the query instruction.

[0010] Preferably, the edge node also includes a query module, a sub-query module and a data migration module;

[0011] The query module receives the data query instruction and distributes the query task to the sub-query module for execution. At the same time, the sub-query module counts the resource usage and regularly summarizes the statistical timestamps and corresponding statistical data and sends them to the data migration module for record;

[0012] The data migration module identifies the hot data using the hot data identification model configured by the administrator according to the record content each time the overhead is updated, and requests the data migration module to migrate the data slices that cause performance bottlenecks.

[0013] Preferably, the collection task deployment module queries the data flow group to which the data flow belongs according to the accessed data flow resource address, and establishes an independent data collection thread for each accessed data flow group to collect messages of all data flows belonging to the data flow group.

[0014] Preferably, the data collection thread module includes a data monitoring thread, a database writing thread and an index updating thread;

[0015] The data monitoring thread monitors the messages uploaded by the user through the message queue;

[0016] The database writing thread periodically writes data in batches to the local database. If there is no corresponding database for the data shard, a table and index are created based on the metadata of the data stream group.

[0017] The index update thread periodically flushes the newly written data summary information back to the global index management module.

[0018] Preferably, the data stream metadata management module records basic configuration information of the data stream, maintains metadata of all data streams, and provides a metadata query interface for querying metadata;

[0019] The basic configuration information includes datagram format information, message subscription address and data storage table name;

[0020] The input of the metadata query interface is the resource address of the data stream, and the output is the data stream group to which the data stream belongs and the metadata configuration of the data stream group.

[0021] Preferably, the global index management module provides a query interface and an information update interface, and the information update interface is used to receive data information newly written by the edge node end;

[0022] When the query module performs a range-filtered search, the query module sends the data-filtering-related fields in the query body to the query interface, and the query interface returns the data slices of all query-related data and the node numbers where the data slices are located.

[0023] Preferably, the range filtering search is performed using a mini-Rosetta filter, and the mini-Rosetta filter includes a multi-layer Bloom filter.

[0024] A data management method for a mobile Internet of Things cloud-edge fusion scenario provided by the present invention includes:

[0025] Collection task deployment steps: query and obtain data stream metadata, receive mobile edge device data, and deploy corresponding data collection threads;

[0026] Data collection thread step: parse the corresponding data stream metadata and store it in the local database;

[0027] Global index management steps: Return the node information where the corresponding data exists according to the query instruction.

[0028] Preferably, it also includes:

[0029] Data query step: receive data query instructions and distribute query tasks and execute sub-query tasks, count resource usage, and regularly summarize statistical timestamps and corresponding statistical data;

[0030] Data migration step: when the query overhead of each data shard is updated, the hot data in the statistical data is identified through the hot data model, and then it is determined whether there is hot data. If so, the data shard that causes the performance bottleneck is migrated to the cloud data center for processing; if not, it is not migrated for processing. Preferably, the global index management step includes local data indexing at the edge node end and global data indexing at the cloud data center end;

[0031] The local data index of the edge node is managed by the database of each edge node. When the data collection thread writes the data of the data stream group to the edge database for the first time, the data collection thread will create a table and index according to the metadata of the data stream group.

[0032] The global data index at the cloud data center end is processed by a mini-Rosetta filter, and the mini-Rosetta filter includes a multi-layer Bloom filter.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. The present invention utilizes the advantages of low communication delay and sufficient bandwidth between edge nodes and mobile Internet of Things devices, and the characteristics of stable connection between cloud data centers and all edge nodes and sufficient computing and storage resources, and tries to avoid the problem of network congestion caused by limited network bandwidth between central nodes and edge nodes, so as to provide better service quality for data management under mobile edge networks.

[0035] 2. The present invention aims to reduce the network overhead caused by data storage and query of mobile Internet of Things devices. All data submitted by users will be stored locally in the edge node first, so that there is no inter-node communication overhead caused by data transmission when writing. At the same time, the data on each edge node are data in the same area, and the data are clustered in spatial and temporal dimensions. When users need to query the status information of other nearby devices, there will be good locality. At the same time, each edge node also optimizes the efficiency of local query execution through indexing.

[0036] 3. The present invention uses the Mini Rosetta filter. When executing a query, the cloud data center verifies the edge node storing relevant data based on the summary information of the data segment within the query range and sends a query request to it. This can not only more effectively find the query-related data, but also ensure the requirement of low network overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0038] Figure 1 Schematic diagram of the system architecture of the present invention.

[0039] Figure 2 Schematic diagram of the triggering process of hotspot data migration in the present invention. DETAILED DESCRIPTION

[0040] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0041] The status data sent by the IoT device of the present invention will be stored preferentially on the nearby edge nodes. The edge nodes only upload data summary information based on the compressed mini-Rosetta filter to the cloud data center to build a global data index, which reduces the network transmission overhead of data collection and provides the ability to search the global data range. The cloud data center stores and manages metadata to accelerate data query. The load-aware data migration steps on the edge nodes ensure service stability and improve service quality, thereby more efficiently meeting the needs of mobile IoT device management in edge environments with limited network, computing, and storage resources.

[0042] According to the present invention, a data management system for a mobile Internet of Things cloud-edge fusion scenario includes: an edge node end and a cloud data center end. Among them, the edge node end includes a collection task deployment module and a data collection thread module, and the cloud data center end includes a data stream metadata management module and a global index management module. Specifically, the edge node end is responsible for data collection, data query and data migration. The user device connects to the edge node and uploads the data message, and the edge node listens to the data and writes it to the local database. Users can also send query requests to the edge node. When the edge node is short of storage resources, computing resources and other resources, the data migration module will select appropriate data shards and migrate them to the data center for storage. The cloud data center end is connected to all edge nodes through the network and has sufficient storage and computing resources to store data and process query requests.

[0043] More specifically, the collection task deployment module deploys the corresponding data collection thread by querying the data flow metadata maintained by the data flow metadata management module and receiving the mobile edge device data. The data collection thread module parses and stores the corresponding metadata to the local database, and the global index management module returns the node information of the corresponding data according to the query instruction. The edge node end also includes a query module, a sub-query module and a data migration module. The query module receives the data query instruction and distributes the query task to the sub-query module for execution. At the same time, the sub-query module counts the resource occupancy, and regularly summarizes the statistical timestamps and corresponding statistical data and sends them to the data migration module for record; the data migration module uses the hot data identification model configured by the administrator to identify the hot data according to the record content at each overhead update, and will cause the data sharding of the performance bottleneck, and request the data migration module to migrate the data sharding, where the performance bottleneck refers to the situation that the edge node also has limited computing resources and network bandwidth resources. Due to the imbalance of the device distribution itself, some nodes may be connected to more users. At this time, the number of queries carried by the node will also increase accordingly, which will bring challenges to the computing and network bandwidth resources of the node, thereby affecting the service quality of the query. If there is a data skew problem and the data stored on a node becomes hot data, then that node will need to process a large number of subqueries, which can easily lead to increased query latency or even node crashes.

[0044] Furthermore, the collection task deployment module queries the cloud data center data flow metadata management module for the data flow group to which the data flow belongs based on the accessed data flow resource address, and establishes an independent data collection thread for each accessed data flow group to collect messages from all data flows belonging to the data flow group. If there is already a data collection thread for collecting data for the same data flow group, the existing data collection thread will be reused. The collection task deployment module will monitor the time of the last data flow connection event of the data collection thread of the data flow. When all data flows under a data flow group are disconnected, the collection task deployment module will actively release the resources occupied by the collection thread responsible for collecting the data flow group, saving the limited memory resources of the edge node.

[0045] The data collection thread module parses each message according to the metadata of the data stream group, and writes the parsed data in batches into the local database. Specifically, it includes a data listening thread, a database writing thread, and an index update thread. Among them, the data listening thread listens to the messages uploaded by the user through the message queue. The message uploaded by the user through the message queue is a piece of text in the form of a string. The data collection thread will be responsible for listening to these messages, parsing the text into structured or semi-structured data, and writing it into the database in batches. During the parsing process, the structure of the data and the type of the field are obtained according to the metadata configuration of the data stream group of the data, that is, by querying the data stream metadata management module of the cloud data center. The database writing thread periodically writes data in batches to the local database. If there is no corresponding database for the data shard, a table and index are created according to the metadata of the data stream group. The index update thread periodically writes the summary information of the newly written data back to the global index management module.

[0046] The data migration module stores the query data shards and the amount of resources used during the query service execution process. That is to say, in the sub-query module of each edge node, the query data shards and the amount of resources used for each sub-query are recorded and regularly summarized and sent to the data migration module. The data migration module will store the historical query overhead of each data shard, and use the hot data identification model configured by the administrator to identify hot data each time the overhead is updated. If there is a data shard that causes a performance bottleneck, the data migration module is requested to migrate the shard. The specific description of the process of data migration triggering is as follows:

[0047] The subquery module will count the resource usage such as the cumulative execution time, the amount of data transmitted, and the memory occupied by all data shard subqueries in each smaller time slot, and send the statistical data to the data migration module with the statistical timestamp for recording. The data migration module will use a list to record the usage of each resource in each time period for each data shard. Whenever the overhead data of a data shard is updated, the data migration module will pass the list of overhead data of each resource to the load sensing interface. The load sensing interface in the present invention provides a load statistics based on a sliding window. The model will traverse the resource occupancy statistical records of the specified data shard, delete the invalid records outside the window according to the given time window length, and separately accumulate the resource occupancy of each resource in the statistical records within the window. If a resource occupies too much in a recent period of time and exceeds the set threshold, the migration of the data shard will be triggered.

[0048] The data stream metadata management module records the basic configuration information of the data stream, maintains the metadata of all data streams, and provides a metadata query interface for querying metadata. The basic configuration information includes datagram format information, message subscription address, and data storage table name. The input of the metadata query interface is the resource address of the data stream, and the output is the data stream group to which the data stream belongs and the metadata configuration of the data stream group. In this way, subsequent services can obtain the storage, parsing, and query configuration information of the data stream group to which it belongs based on the resource address provided by the datagram, so as to correctly provide the services required by users.

[0049] In the present invention, data submitted by all user devices will be uploaded in the form of messages through message queues, and the subject of the message will be used to distinguish various types of messages. All data under the same subject will be regarded as a data stream, and all messages in each data stream have the same message format. Usually, a data stream corresponds to all data continuously uploaded by a device or a sensor. Since edge nodes usually have good network connections with nearby devices, devices will push data directly to the message queues on nearby edge nodes connected to them. Each data stream has a fixed address, and the collection relationship of data streams is determined by the prefix of the address.

[0050] The metadata of a data stream group must contain at least the following fields:

[0051] Resource Path: The longest common prefix of all data flow resource addresses in the data flow group. When a user data flow accesses, the management platform can identify the data flow group to which it belongs based on its prefix.

[0052] Collection Name: The name of the table in the database that stores the data of this data stream group. For each data stream group, the node will use a separate database table to store it for more efficient index query.

[0053] PayloadType: The format specification of the data. Currently all messages are encoded in JSON format. More dedicated message formats can be expanded in the future.

[0054] TimeField: A field in a datagram that indicates the time when the data was generated. This field will be indexed separately, and all data will be stored in shards in the database based on the time provided by this field.

[0055] Fields: The type of each field. Currently, only strings, integers, and floating-point numbers are supported. In order to better implement global indexes, the left and right boundaries of each field and the minimum query interval during query are also required.

[0056] The global index management module returns the node information where the relevant data exists according to the user's query conditions. It provides query interfaces and information update interfaces. The information update interface is used to receive newly written data information from the edge node. That is to say, every preset period of time, if the edge node has written new data, the edge node will summarize the newly written data information and report it to the global index module through the information update interface. When the query module performs a range filtering search, the query module sends the data filtering related fields in the query body to the query interface, and the query interface returns the data shards of all query-related data and the node numbers where the data shards are located.

[0057] Among them, in order to meet the needs of range query in the scenario of IoT device data analysis, the range filtering search uses Mini Rosetta filter for search. Mini Rosetta filter includes multi-layer Bloom filter. This Mini Rosetta improves the problem of high prefix tree height in the original Rosetta filter implementation, which leads to increased total filter space and low transmission efficiency, and adds support for string type prefix search to additionally support prefix matching query of resource address. This two-stage data index supports range search and ensures low network overhead requirements, takes up little space, has high query efficiency and supports range search requirements.

[0058] Specifically, a typical mini-Rosetta filter consists of multiple layers of Bloom filters, denoted as

[0059] RF={BF1,BF2…,BF l}, which provides three methods: data insertion Insert, data search Check and filter merging InsertAll.

[0060] The lnsert(val) method inserts a value into the filter. The process of inserting numeric data is slightly different from that of string data. If the inserted data is a number, it needs to be converted to an unsigned integer type ui. When defining the type information of the message field in the metadata of the data stream group, the maximum value of the field field is provided. i,max Minimum field i,min The minimum query interval of the and field field i,g The conversion method is to limit the value within the interval, subtract the minimum value, and then divide it by the minimum query interval. The conversion formula is as follows:

[0061]

[0062] After the conversion is completed, inserting the data will insert a certain length prefix of the binary complement of ui into the Bloom filter of each layer, and the length is the number of layers k of the Bloom filter multiplied by the prefix growth parameter α.

[0063] For string types, MiniRosetta uses delimiters to divide the string and calculates the length of the string prefix that needs to be included in each layer to support prefix matching search of data stream addresses.

[0064] According to the above data insertion process, we can get the number of layers l required for each Mini Rosetta filter. For numeric data, the number of layers is the encoding length of the maximum value of the data in binary after conversion to an unsigned integer divided by the prefix growth parameter. In this case, the length of l is:

[0065]

[0066] For string types, l is the maximum level of the string. For prefix matching queries of resource address fields, l is the maximum length of data stream address nesting. For ordinary string fields of strict matching queries, l is 1, which degenerates into an ordinary Bloom filter.

[0067] The CheckNum(left, right) method is used to determine whether a number in the search range has been inserted into the filter. If it returns true, it is possible that some values ​​that meet the search range have been inserted into the filter. If it returns false, there must be no values ​​in the filter that meet the search range.

[0068] The CheckStr(str) process for searching string type data is relatively simple. It only needs to check whether the prefix to be queried exists in each layer in turn.

[0069] Furthermore, the method of using the Mini Rosetta filter to index global data is described in detail as follows:

[0070] First, all database tables are sharded by time, divided into equal time periods of every hour or every day. When the data collection thread on each edge node receives data, it calls the messages written within a period of time a data segment. The messages in a data segment must belong to the same data segment. Otherwise, the data segment is split into multiple sub-segments, and each sub-segment must belong to the same data segment.

[0071] Then, each query field value of the message in the segment needs to be written into the meta information of the segment.

[0072] Finally, the metadata of the segment is flushed back to the data center, and the global index management module of the cloud data center uses the database to back up the metadata records of each data segment. The metadata of the data segment here is similar to the metadata of the data shard, which is the sum of the metadata of all data segments in the shard.

[0073] When checking whether the data required for the query exists in the shard, it is necessary to check the metadata of all segments, which is time-consuming. Therefore, the global index module maintains an aggregated shard-level metadata in memory. Aggregation means merging the metadata of all data segments under the same shard. The merge will use the InsertAll operation to merge the MiniRosetta filters of the same field in the metadata of all data segments into a single filter according to each field of the shard metadata. When any node performs a query, if the target node of the query is not specified, it is necessary to obtain the information of the node where the data is located from the cloud data center according to the query range conditions.

[0074] Users build a global data index by uploading data summary information based on the compressed Mini Rosetta filter at the edge node to the data center. The data center stores and manages the data stream metadata. At the same time, a load-aware data migration solution is used on the edge node to ensure service stability.

[0075] The present invention aims to provide a data indexing, data migration and data management method in a cloud-edge fusion scenario.

[0076] A data management method for a mobile Internet of Things cloud-edge fusion scenario provided by the present invention includes:

[0077] Collection task deployment steps: query and obtain data stream metadata, receive mobile edge device data, and deploy corresponding data collection threads.

[0078] Data collection thread step: Parse the corresponding data stream metadata and store it in the local database.

[0079] Global index management step: Return the node information where the corresponding data exists according to the query instruction. In order to meet the needs of data analysis of IoT devices, most queries are range queries. The global index management step includes local data index on the edge node side and global data index on the cloud data center side. The local data index on the edge node side is managed by the database of each edge node. When the data collection thread writes the data of the data stream group to the edge database for the first time, the data collection thread will create tables and indexes based on the metadata of the data stream group. The index is used to efficiently find data in a single database and perform local query operations faster. The global data index on the cloud data center side is processed by the Mini Rosetta filter. The Mini Rosetta filter includes a multi-layer Bloom filter, which is an improvement based on the Rosetta filter. The problem that the height of the prefix tree in the original Rosetta filter implementation is very high, which increases the total space of the filter and reduces the efficiency of transmission is improved, and prefix search support for string types is added to additionally support prefix matching queries for resource addresses.

[0080] Data query step: Receive data query instructions and distribute query tasks and execute sub-query tasks, and count resource usage, and regularly summarize statistical timestamps and corresponding statistical data.

[0081] Data migration steps: Each time the query overhead of a data shard is updated, the hot data in the statistical data is identified through the hot data model to determine whether there is hot data. If so, the data shard that causes the performance bottleneck is migrated to the cloud data center for processing; if not, it is not migrated for processing.

[0082] Specifically, since cloud data centers have sufficient computing and storage resources, they tend to migrate to cloud data centers. Subsequent subqueries will be forwarded to cloud data centers for execution to avoid affecting the normal operation of edge nodes. Although data migration will bring a lot of network bandwidth overhead, since there is usually not too much hot data, data migration is not a frequent behavior, so this overhead is usually acceptable. The process triggered by hot data migration is as follows: Figure 2 As shown, the subquery module will count the resource usage such as the cumulative execution time, the amount of data transmitted, and the memory occupied by all data shard subqueries in each smaller time slot, and send the statistical data to the data migration module with the statistical timestamp for recording. The data migration module will use a list to record the usage of each resource in each time period for each data shard. Whenever the overhead data of a data shard is updated, the data migration module will pass the list of overhead data of each resource to the load sensing interface. The load sensing interface in the present invention provides a load statistics based on a sliding window. The model will traverse the resource occupancy statistical records of the specified data shard, delete the invalid records outside the window according to the given time window length, and separately accumulate the resource occupancy of each resource in the statistical records within the window. If a resource occupies too much in a recent period of time and exceeds the set threshold, the migration of the data shard will be triggered.

[0083] Those skilled in the art know that, in addition to implementing the system, device and its various modules provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, the system, device and its various modules provided by the present invention can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing the method and structures within the hardware component.

[0084] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A data management system for mobile Internet of Things cloud-edge fusion scenarios, characterized in that: include: Edge node side and cloud data center side; The edge node end includes a collection task deployment module and a data collection thread module; The cloud data center end includes a data stream metadata management module and a global index management module; The collection task deployment module deploys the corresponding data collection thread by querying the data stream metadata maintained by the data stream metadata management module and receiving the mobile edge device data. The data collection thread module parses the corresponding metadata and stores it in the local database. The global index management module returns the node information where the corresponding data exists according to the query instruction.

2. The data management system for mobile Internet of Things cloud-edge fusion scenarios according to claim 1 is characterized in that: The edge node also includes a query module, a sub-query module and a data migration module; The query module receives the data query instruction and distributes the query task to the sub-query module for execution. At the same time, the sub-query module counts the resource usage and regularly summarizes the statistical timestamps and corresponding statistical data and sends them to the data migration module for record; The data migration module identifies the hot data using the hot data identification model configured by the administrator according to the record content each time the overhead is updated, and requests the data migration module to migrate the data slices that cause performance bottlenecks.

3. The data management system for mobile Internet of Things cloud-edge fusion scenarios according to claim 1 is characterized in that: The collection task deployment module queries the data stream group to which the data stream belongs according to the accessed data stream resource address, and establishes an independent data collection thread for each accessed data stream group to collect messages of all data streams belonging to the data stream group.

4. The data management system for mobile Internet of Things cloud-edge fusion scenarios according to claim 1 is characterized in that: The data collection thread module includes a data monitoring thread, a database writing thread and an index updating thread; The data monitoring thread monitors the messages uploaded by the user through the message queue; The database writing thread periodically writes data in batches to the local database. If there is no corresponding database for the data shard, a table and index are created based on the metadata of the data stream group. The index update thread periodically flushes the newly written data summary information back to the global index management module.

5. The data management system for mobile Internet of Things cloud-edge fusion scenarios according to claim 1 is characterized in that: The data stream metadata management module records basic configuration information of the data stream, maintains metadata of all data streams, and provides a metadata query interface for querying metadata; The basic configuration information includes datagram format information, message subscription address and data storage table name; The input of the metadata query interface is the resource address of the data stream, and the output is the data stream group to which the data stream belongs and the metadata configuration of the data stream group.

6. The data management system for mobile Internet of Things cloud-edge fusion scenarios according to claim 1 is characterized in that: The global index management module provides a query interface and an information update interface, and the information update interface is used to receive data information newly written by the edge node; When the query module performs a range-filtered search, the query module sends the data-filtering-related fields in the query body to the query interface, and the query interface returns the data slices of all query-related data and the node numbers where the data slices are located.

7. The data management system for mobile Internet of Things cloud-edge fusion scenarios according to claim 6 is characterized in that: The range filtering search is performed using a mini-Rosetta filter, and the mini-Rosetta filter includes a multi-layer Bloom filter.

8. A data management method for mobile Internet of Things cloud-edge fusion scenarios, characterized in that: include: Collection task deployment steps: query and obtain data stream metadata, receive mobile edge device data, and deploy corresponding data collection threads; Data collection thread step: parse the corresponding data stream metadata and store it in the local database; Global index management steps: Return the node information where the corresponding data exists according to the query instruction.

9. The data management method for the mobile Internet of Things cloud-edge fusion scenario according to claim 8 is characterized in that: Also includes: Data query step: receive data query instructions and distribute query tasks and execute sub-query tasks, count resource usage, and regularly summarize statistical timestamps and corresponding statistical data; Data migration step: when the query overhead of each data shard is updated, the hot data in the statistical data is identified through the hot data model, and then it is determined whether there is hot data. If so, the data shard that causes the performance bottleneck is migrated to the cloud data center for processing; If it does not exist, it will not be migrated.

10. The data management method for mobile Internet of Things cloud-edge fusion scenarios according to claim 8 is characterized in that: The global index management step includes local data indexing at the edge node end and global data indexing at the cloud data center end; The local data index of the edge node is managed by the database of each edge node. When the data collection thread writes the data of the data stream group to the edge database for the first time, the data collection thread will create a table and index according to the metadata of the data stream group; The global data index at the cloud data center end is processed by a mini-Rosetta filter, and the mini-Rosetta filter includes a multi-layer Bloom filter.

Citation Information

Patent Citations

  • Data processing method, device and equipment in distributed edge storage system

    CN108121511A

  • Hot spot data management method, device and system

    CN110569233A

  • Cloud platform architecture based on big data and data processing method thereof

    CN113377577A

  • Data storage device and stored data migration method

    WO2022062015A1