An Internet of Things-based big data distributed storage method, device and medium

Abstract

The application discloses a big data distributed storage method and device based on Internet of Things, and relates to the technical field of distributed storage. The method comprises the following steps: establishing a naming rule, sequentially combining object attribute identifiers in a data writing request of Internet of Things, generating an Internet of Things naming identifier, and encapsulating the Internet of Things naming identifier and a data load into a naming writing request packet; performing naming analysis on the Internet of Things naming identifier in the naming writing request packet, determining a target shard identifier, and generating a shard routing result packet; writing the data load into a shard data segment corresponding to the target shard identifier according to a shard node set in the shard routing result packet, and performing index registration on the Internet of Things naming identifier, generating a distributed index item and registering it as available. The application realizes the efficiency and flexibility of data storage and routing by performing naming analysis and generating a shard routing result packet based on a target shard identifier.

Description

Technical Field

[0001] This invention relates to the field of distributed storage technology, and in particular to a method, device and medium for distributed storage of big data based on the Internet of Things. Background Technology

[0002] With the rapid development of IoT technology, an increasing number of devices are generating massive amounts of data. The data generated by IoT devices is diverse, including sensor data, control information, and environmental data. The storage, processing, and analysis of this data have become core issues for IoT applications. Distributed storage, due to its flexibility and scalability, has become the mainstream solution for IoT big data storage. Current distributed storage methods typically rely on sharding technology, dividing data into multiple fragments and storing them on different nodes to improve storage and access efficiency. However, ensuring that the sharded data can be efficiently routed and accessed, while guaranteeing data consistency and reliability, remains a pressing problem to be solved.

[0003] When existing distributed storage is applied in IoT environments, it often relies on static sharding and routing strategies, making it difficult to dynamically adjust according to real-time data demands. This means that as the number of IoT devices increases, existing storage and routing strategies may face efficiency bottlenecks and struggle to cope with constantly changing data loads. Current technologies typically use fixed routing tables and naming resolution mechanisms, but lack flexibility and dynamic adjustment capabilities, resulting in storage efficiency and data access speeds failing to meet requirements in high-volume and large-scale IoT environments. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a distributed big data storage method based on the Internet of Things to solve the problems of lack of flexibility and dynamic adjustment capabilities.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a distributed big data storage method based on the Internet of Things, comprising,

[0008] Establish a naming rule to orderly combine the set of object attribute identifiers in the IoT data write request to generate an IoT naming identifier, and encapsulate the IoT naming identifier and data payload into a named write request package; the data payload is a set of fields that carry the actual business data in the IoT data write request.

[0009] Perform name resolution on the IoT name identifier in the name write request packet to determine the target fragment identifier and generate a fragment routing result packet;

[0010] Based on the set of fragment nodes in the fragment routing result packet, the data payload is written into the fragment data segment corresponding to the target fragment identifier, and the IoT naming identifier is indexed and registered, generating a distributed index item and registering it as available. The available status of the distributed index item is encapsulated into a searchable submission receipt packet.

[0011] Based on the retrievability submission receipt packet, the resolvability and routableness of the IoT naming identifier in the distributed index item are verified. When the retrievability verification is successful, the distributed storage write is confirmed to be complete. When the retrievability verification fails, the fragmentation routing adjustment process is triggered.

[0012] As a preferred embodiment of the IoT-based distributed storage method for big data described in this invention, the establishment of naming rules involves inserting fixed separators into adjacent levels of the object attribute identifier set to form a hierarchical string, and then registering the hierarchical string as a naming rule after performing character set unification and null value padding.

[0013] As a preferred embodiment of the IoT-based distributed storage method for big data described in this invention, the step of encapsulating the IoT naming identifier and data payload into a naming write request packet specifically involves:

[0014] According to the naming rules, the set of object attribute identifiers is sorted and ordered to form a sequence of object attribute identifiers;

[0015] The object attribute identifier sequence is formatted uniformly to obtain a standardized sequence;

[0016] The standardized sequence is concatenated to generate an IoT naming identifier, and the IoT naming identifier is encapsulated with the data payload to generate a naming write request packet.

[0017] As a preferred embodiment of the IoT-based distributed storage method for big data described in this invention, the generation of the fragmented routing result packet specifically involves:

[0018] Establish a naming resolution table and write the mapping relationship between IoT naming identifiers and target fragment identifiers; establish a fragment routing table and register the routing domain information of the target fragment identifier;

[0019] The IoT naming identifier in the naming request packet is split hierarchically according to the hierarchical boundaries of the naming rules to generate hierarchical fragments;

[0020] The hierarchical fragments are concatenated level by level according to the naming rules to generate a prefix sequence;

[0021] Perform prefix matching on the prefix sequence to obtain the matching prefix;

[0022] In the naming resolution table, the shard mapping record is located by the hit prefix to determine the target shard identifier;

[0023] Based on the target fragment identifier, select a fragment node set from the fragment routing table, and encapsulate the target fragment identifier and the fragment node set to generate a fragment routing result packet.

[0024] As a preferred embodiment of the IoT-based distributed storage method for big data described in this invention, the determination of the target shard identifier involves performing table entry location processing in the naming resolution table using the hit prefix as the retrieval key, retrieving the shard mapping record associated with the hit prefix, and extracting the target shard identifier from the shard mapping record.

[0025] As a preferred embodiment of the IoT-based distributed big data storage method of the present invention, the step of generating distributed index items and registering them as available specifically involves:

[0026] Perform node ordering on the fragment node set in the fragmented routing result packet to obtain the first node and successor node in the writing chain order;

[0027] At the first node, the data segment corresponding to the target fragment identifier is appended to the fragment, generating a write fragment, and the data payload is stored in the write fragment;

[0028] The write segment is passed to the successor node hop by hop in the write chain order. At the successor node, the data segment corresponding to the target segment identifier is synchronously written and appended, and the write completion status is output.

[0029] Based on the write completion status, perform write completion confirmation on the successor node and generate a write completion message;

[0030] The write completion message is sent back to the first node hop by hop in the write chain order, triggering the index registration of the IoT naming identifier, generating a distributed index entry, and registering it as available.

[0031] As a preferred embodiment of the IoT-based distributed big data storage method of the present invention, the verification of the resolvability and routableness of IoT naming identifiers in distributed index entries specifically involves:

[0032] The searchability submission receipt is decapsulated to obtain the availability status of the IoT naming identifier and distributed index item;

[0033] Using the IoT naming identifier as the retrieval key, the distributed index item is matched and located, and the index target fragment identifier is read from the distributed index item;

[0034] Perform an existence check on the index target shard identifier and generate the existence check result;

[0035] Perform consistency verification on the availability status of distributed index items and the target shard identifier of the index, and generate consistency verification results;

[0036] When the existence verification result is passed and the consistency verification result is passed, a searchable verification success flag is generated, and the distributed storage write is confirmed to be complete.

[0037] If either the existence verification result or the consistency verification result fails, a searchable verification failure flag is generated, and the fragmentation routing adjustment process is triggered.

[0038] As a preferred embodiment of the IoT-based big data distributed storage method of the present invention, the distributed storage write completion means that the data payload corresponding to the IoT naming identifier has been written to the data segment corresponding to the target segment identifier, and the distributed index item registration has been completed, and the successful verification has been marked and confirmed.

[0039] The triggered fragmentation routing adjustment process refers to re-executing name resolution and target fragmentation identifier determination based on the searchable verification failure flag, and generating a new fragmentation routing result packet to replace the original fragmentation node set.

[0040] In a second aspect, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements any step of the Internet of Things-based distributed storage method for big data as described in the first aspect of the present invention.

[0041] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the Internet of Things-based distributed storage method for big data as described in the first aspect of the present invention.

[0042] The beneficial effects of this invention are as follows: By performing naming resolution on IoT naming identifiers and generating fragmented routing result packets based on target fragment identifiers, high efficiency and flexibility in data storage and routing are achieved. Dynamic naming identifier resolution accurately determines the target fragments of data, ensuring reasonable data distribution and efficient storage path selection. During fragmented routing, the precise matching of fragment identifiers with the target fragment node set based on the naming resolution table optimizes data access speed and avoids bottlenecks in traditional static routing strategies. Adaptive routing adjustment allows data to flexibly adapt to the ever-changing IoT environment, achieving immediate response and processing capabilities for large-scale data loads, significantly improving storage efficiency and system responsiveness, while enhancing the scalability and dynamic adjustment capabilities of distributed storage to meet rapidly growing data storage demands. Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart of a distributed big data storage method based on the Internet of Things.

[0045] Figure 2 A flowchart for performing name resolution.

[0046] Figure 3 This is a flowchart for adjusting the sharding route.

[0047] Figure 4 This is a flowchart for retrievability verification. Detailed Implementation

[0048] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0049] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0050] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0051] Reference Figures 1-4 This is one embodiment of the present invention, which provides a distributed big data storage method based on the Internet of Things, including the following steps:

[0052] S1. Establish naming rules, orderly combine the set of object attribute identifiers in the IoT data write request to generate IoT naming identifiers, and encapsulate the IoT naming identifiers and data payloads into a named write request package; the data payload is a set of fields that carry the actual business data in the IoT data write request.

[0053] IoT data write requests originate from real-time data collected by IoT devices (such as smart cameras), including a set of object attribute identifiers and a data payload.

[0054] A hierarchical string is formed by inserting fixed separators at adjacent levels of the object attribute identifier set. After unifying the character set and padding with null values ​​on the hierarchical string, it is registered as a naming rule, specifically:

[0055] In the object attribute identifier set, the identifiers are arranged in the order of tenant identifier, device identifier, geographic domain identifier, timestamp, and data type identifier. The tenant identifier is used to distinguish the data boundaries of different tenants and is placed at the beginning of the hierarchy string. The device identifier is used to distinguish different IoT terminals under the same tenant and is placed after the tenant identifier. The geographic domain identifier is used to distinguish different access areas under the same tenant and is placed after the device identifier. The timestamp is used to distinguish data objects generated by the same device at different times and is placed after the geographic domain identifier. The data type identifier is used to distinguish different sets of business fields at the same point in time and is placed after the timestamp. The tenant identifier, device identifier, geographic domain identifier, timestamp, and data type identifier are arranged in the same order in all named write request packets.

[0056] After the arrangement is completed, a fixed delimiter of the same character form is inserted between adjacent object attribute identifiers to form a hierarchical string; all characters in the hierarchical string are compared with the allowed character set, and characters that do not belong to the allowed character set are replaced with the corresponding uniform character form, so that the characters in the hierarchical string only retain letters, numbers and fixed delimiters.

[0057] Further explanation: The allowed character set is directly given by the character set range in the naming rule registration. The character set range is explicitly defined as the set of characters allowed to appear in the hierarchical string when the naming rule is registered. The character set range includes letters, numbers, and fixed separators under the unified encoding rule. The character set range is also used as the judgment boundary for character replacement rules. The character set range is set based on the value distribution of the IoT data naming identifiers already in the naming write request packet. The character set range includes character types that appear clearly and consistently in the IoT data naming identifiers, that is, character types that always appear in the IoT data naming identifiers without change, and are limited to letters, numbers, and fixed separators. The fixed separator is selected based on the unified character form within the character set range specified in the naming rule, ensuring that the separators inserted between each level of the object attribute identifier set are consistent with the naming rule.

[0058] When a missing field exists in the object attribute identifier set, a fixed-length placeholder is written at the corresponding position to complete the null value padding. Null value padding uses a fixed-length placeholder at the corresponding position. The fixed length is set based on the historical length distribution of the object attribute identifiers, selecting the maximum value length with a coverage ratio of over 99%. For example, the maximum value length comes from length sample data of at least 1000 object attribute identifiers. Selecting at least 1000 length sample data of object attribute identifiers ensures that the data has sufficient representativeness and statistical validity, accurately reflects the length distribution characteristics of object attribute identifiers, avoids errors or inconsistencies caused by length differences, and conforms to naming rules. The naming rules require that the object in the IoT data writing request be named according to the naming rules. The arrangement of attribute identifier sets, character set ranges, character replacement rules, and separator formats ensure that named identifiers conform to unified specifications and maintain consistency. After character replacement and null value padding are completed, the arrangement positions of tenant identifiers, device identifiers, geographic domain identifiers, time stamps, data type identifiers, fixed separator formats, character replacement rules, and placeholder formats are recorded as naming rules and registered as the naming rules used when generating IoT named identifiers. The character replacement rules use the letters, numbers, and fixed separators registered in the allowed character set as the replacement set. When a character not included in the allowed character set appears in the hierarchical string, the missing character is replaced with a fixed separator, so that the characters in the hierarchical string after replacement only contain letters, numbers, and fixed separators.

[0059] According to the naming rules, the object attribute identifier set is sorted and ordered to form an object attribute identifier sequence. The sorting and ordering is performed according to the order of tenant identifier, device identifier, geographic domain identifier, timestamp, and data type identifier registered in the naming rules. The corresponding positions are read from the object attribute identifier set and written to them in sequence. Specifically:

[0060] According to the tenant identifier, device identifier, geographic domain identifier, time stamp, and data type identifier arrangements registered in the naming rules, the corresponding object attribute identifiers are read sequentially from the object attribute identifier set and written into the corresponding arrangement positions of the object attribute identifier sequence. When there are missing object attribute identifiers in the object attribute identifier set, a placeholder is written into the corresponding arrangement position of the object attribute identifier sequence to maintain the continuity of the arrangement positions. After all arrangement positions are written, the object attribute identifier sequence is output in the registration order.

[0061] The tenant identifier, device identifier, data type identifier, and timestamp in the object attribute identifier sequence are standardized to obtain a standardized sequence, specifically:

[0062] The tenant identifier, device identifier, data type identifier, and timestamp are read sequentially from the object attribute identifier sequence, and the tenant identifier, device identifier, data type identifier, and timestamp are compared item by item with the character set range registered in the naming rules.

[0063] When all characters in the tenant identifier, device identifier, data type identifier, and timestamp are within the character set range, maintain the original character format of the tenant identifier, device identifier, data type identifier, and timestamp; when characters outside the character set range appear in the tenant identifier, device identifier, data type identifier, and timestamp, replace the characters outside the character set range with fixed separators; when all letters in the tenant identifier, device identifier, data type identifier, and timestamp conform to the letter writing format registered in the naming rules, maintain the original letter writing format; when different letter writing formats appear in the tenant identifier, device identifier, data type identifier, and timestamp, unify the letters to the letter writing format registered in the naming rules; when the timestamp uses the time expression method registered in the naming rules, maintain the original timetamp expression method; when the timestamp uses different time expression methods, convert the timestamp to the time expression method registered in the naming rules.

[0064] After character replacement, standardization of letter writing format, and standardization of time expression method are completed, the tenant identifier, device identifier, data type identifier, and time stamp are output as a standardized sequence.

[0065] The standardized sequence is concatenated to generate an IoT naming identifier, and the IoT naming identifier and data payload are encapsulated to generate a naming write request packet. The concatenation is to arrange the object attribute identifiers registered according to the naming rules, and to concatenate the tenant identifier, device identifier, geographic domain identifier, time stamp and data type identifier in the standardized sequence in sequence, and insert fixed separators between adjacent fields to form the IoT naming identifier.

[0066] To further explain, the IoT naming identifier is a string-type identifier field, while the data payload is a set of business fields. The two are stored side by side in different fields in the naming write request packet, distinguished by field identifiers. There is no uniformity of numerical scale or unit conversion. Only structured encapsulation is performed to maintain their original semantics and value forms.

[0067] S2. Perform naming resolution on the IoT naming identifier in the naming write request packet to determine the target fragment identifier, and generate a fragment routing result packet based on the target fragment identifier.

[0068] Establish a naming resolution table and write the mapping relationship between IoT naming identifiers and target fragment identifiers. Establish a fragment routing table and register the routing domain information of the target fragment identifier, specifically:

[0069] A name resolution table is created in the storage space of the distributed storage routing service, and an IoT name identifier field and a target fragment identifier field are set in the name resolution table. The IoT name identifier is written to the IoT name identifier field of the name resolution table, and the target fragment identifier corresponding to the IoT name identifier is written to the target fragment identifier field of the name resolution table, thereby completing the mapping relationship between the IoT name identifier and the target fragment identifier. The correspondence between the IoT name identifier and the target fragment identifier comes from the fragment configuration registration of the distributed storage routing service. The fragment configuration registration is used in the storage space of the distributed storage routing service to record the binding relationship between the IoT name identifier and the target fragment identifier.

[0070] A sharded routing table is created in the storage space of the distributed storage routing service. The target shard identifier field, routing domain information field, and sharded node set field are set in the sharded routing table. The target shard identifier is written to the target shard identifier field of the sharded routing table. Simultaneously, the routing domain information corresponding to the target shard identifier is written to the routing domain information field of the sharded routing table, and the sharded node set corresponding to the target shard identifier is written to the sharded node set field of the sharded routing table. This completes the registration of the routing domain information and the sharded node set for the target shard identifier. The sharded node set originates from the sharded node list registration in the distributed storage routing service. The sharded node list registration, located in the storage space of the distributed storage routing service, records the binding relationships between sharded node identifiers and their corresponding network domain identifiers and access domain identifiers. The sharded node identifier is the identifier used to uniquely identify the sharded node in the sharded node list registration. The network domain identifier is used to identify the network domain where the sharded node identifier resides, and the access domain identifier is used to identify the access boundary corresponding to the sharded node identifier.

[0071] The IoT naming identifier in the naming request packet is hierarchically split according to the hierarchical boundaries of the naming rules to generate hierarchical fragments, specifically:

[0072] The IoT naming identifier is read from the naming write request packet, and the string of the IoT naming identifier is scanned according to the fixed delimiter positions registered in the naming rules. The string of the IoT naming identifier is cut at each fixed delimiter position, so that the string of the IoT naming identifier is divided into multiple consecutive field fragments. Each field fragment is output as a hierarchical fragment in sequence, and the arrangement order of the hierarchical fragments is kept consistent with the arrangement positions of the tenant identifier, device identifier, geographic domain identifier, time stamp, and data type identifier registered in the naming rules, thereby forming a set of hierarchical fragments for subsequent prefix sequence generation.

[0073] The hierarchical fragments are concatenated level by level according to the naming rules to generate a prefix sequence, specifically:

[0074] Read the hierarchical fragments corresponding to the tenant identifier positions sequentially from the hierarchical fragment set, and insert a fixed separator after the tenant identifier hierarchical fragment; continue reading the hierarchical fragments corresponding to the device identifier positions, and insert a fixed separator after the device identifier hierarchical fragments; continue reading the hierarchical fragments corresponding to the geographic domain identifier positions, and insert a fixed separator after the geographic domain identifier hierarchical fragments; continue reading the hierarchical fragments corresponding to the time stamp positions, and insert a fixed separator after the time stamp hierarchical fragments; continue reading the hierarchical fragments corresponding to the data type identifier positions and form a complete concatenation string.

[0075] After the first round of splicing, the spliced ​​string containing only the tenant identifier level fragment is retained as the first prefix. After the second round of splicing, the spliced ​​string containing both the tenant identifier level fragment and the device identifier level fragment is retained as the second prefix. After the third round of splicing, the spliced ​​string containing the tenant identifier level fragment, the device identifier level fragment, and the geographic domain identifier level fragment is retained as the third prefix. After the fourth round of splicing, the spliced ​​string containing the tenant identifier level fragment, the device identifier level fragment, the geographic domain identifier level fragment, and the time stamp level fragment is retained as the fourth prefix. After the fifth round of splicing, the spliced ​​string containing the tenant identifier level fragment, the device identifier level fragment, the geographic domain identifier level fragment, the time stamp level fragment, and the data type identifier level fragment is retained as the fifth prefix, thus forming a prefix sequence composed of multiple prefixes.

[0076] Perform prefix matching on the prefix sequence to obtain the matching prefix, specifically:

[0077] Read the first, second, third, fourth and fifth prefixes in hierarchical order from the prefix sequence, and compare each of the first, second, third, fourth and fifth prefixes with the IoT naming identifier field in the naming resolution table for initial consistency.

[0078] To further explain, the IoT naming identifier is generated by sorting the set of object attribute identifiers in the IoT data write request, inserting fixed delimiters, and performing character set unification and null value padding. The IoT naming identifier field is a field stored in the IoT naming identifier and is used to store and reference the IoT naming identifier in the naming resolution table.

[0079] When the first prefix has a beginning-matching relationship with any IoT naming identifier field in the naming resolution table, the first prefix is ​​recorded as a candidate hit prefix; when the second prefix has a beginning-matching relationship with any IoT naming identifier field in the naming resolution table, the second prefix is ​​replaced with a candidate hit prefix; when the third prefix has a beginning-matching relationship with any IoT naming identifier field in the naming resolution table, the third prefix is ​​replaced with a candidate hit prefix; when the fourth prefix has a beginning-matching relationship with any IoT naming identifier field in the naming resolution table, the fourth prefix is ​​replaced with a candidate hit prefix; when the fifth prefix has a beginning-matching relationship with any IoT naming identifier field in the naming resolution table, the fifth prefix is ​​replaced with a candidate hit prefix; after completing the beginning-matching relationship comparison of the first, second, third, fourth, and fifth prefixes, the candidate hit prefix is ​​output as the hit prefix.

[0080] When none of the first, second, third, fourth, and fifth prefixes form a candidate hit prefix, output a null hit prefix flag and end the naming resolution; a null hit prefix flag means that there is no IoT naming identifier field entry in the naming resolution table that is consistent with the first, second, third, fourth, or fifth prefix in the starting position; when the null hit prefix flag is displayed, output a null target fragment identifier flag and output a null fragment routing result packet flag.

[0081] In the naming resolution table, the shard mapping record is located using the hit prefix to determine the target shard identifier. Determining the target shard identifier involves performing table entry location processing in the naming resolution table using the hit prefix as the search key, retrieving the shard mapping record associated with the hit prefix, and extracting the target shard identifier from the shard mapping record. Specifically:

[0082] The matched prefix is ​​input as the search key into the naming resolution table. The naming resolution table performs a start-of-prefix consistency lookup on the matched prefix in the IoT naming identifier field. This lookup involves comparing the consecutive characters at the beginning of the string in the IoT naming identifier field with the matched prefix character by character to confirm whether the string in the IoT naming identifier field starts with the matched prefix. When the string in the IoT naming identifier field and the matched prefix are consistent at the beginning, the corresponding entry is identified as a fragment mapping record, and the target fragment identifier field is read from the fragment mapping record as the target fragment identifier output. When the string in the IoT naming identifier field and the matched prefix are not consistent at the beginning, the naming resolution table continues to search for the next entry, and a start-of-prefix consistency lookup is performed on the IoT naming identifier field of the next entry until an entry with consistent beginning positions is found. In the naming resolution table, the same matched prefix corresponds to only one target fragment identifier field value.

[0083] To further clarify, when the prefix is ​​an empty marker, the fragment mapping record is not located and the target fragment identifier is not output.

[0084] Based on the target fragment identifier, a fragment node set is selected from the fragment routing table, and the target fragment identifier and the fragment node set are encapsulated to generate a fragment routing result packet, specifically:

[0085] The target shard identifier is input into the shard routing table as the search key. A consistency comparison is performed on the target shard identifier field of the shard routing table. The consistency comparison is to compare the target shard identifier string in the target shard identifier field with the target shard identifier of the search key character by character to confirm whether the target shard identifier string in the target shard identifier field is completely consistent with the target shard identifier of the search key.

[0086] When the target fragment identifier string in the target fragment identifier field is completely consistent with the search key target fragment identifier, the corresponding table entry is determined as a routing table entry, and the routing domain information field and fragment node set field are read from the routing table entry. The routing domain information field contains a network domain identifier and an access domain identifier, and the fragment node set field contains multiple fragment node identifiers. Each fragment node identifier carries the corresponding network domain identifier and the corresponding access domain identifier. After reading the fragment node set field from the routing table entry, a consistency check is performed between the network domain identifier corresponding to the fragment node identifier in the fragment node set field and the network domain identifier and access domain identifier corresponding to the fragment node identifier in the routing domain information field. Fragment node identifiers that pass the consistency check are registered as fragment node set selection results, and fragment node identifiers that fail the consistency check are not registered as fragment node set selection results. The same target fragment identifier field value in the fragment routing table corresponds to only one routing table entry.

[0087] When the target shard identifier string in the target shard identifier field does not completely match the target shard identifier of the search key, continue searching the target shard identifier field of the next entry in the shard routing table until an entry is found whose target shard identifier string completely matches the target shard identifier of the search key. When there is no entry in the target shard identifier field of the shard routing table that completely matches the target shard identifier of the search key, output a flag indicating that the shard node set is empty and end the shard node set selection. When the shard node set is marked as empty, do not read the shard node set field and do not register the shard node set selection result. When the shard node set is not marked as empty, register the shard node set as the shard node set selection result.

[0088] It should be noted that the target fragment identifier and the fragment node set are not data of the same type and format. The target fragment identifier is a string identifier field, and the fragment node set is a collection of fragment node identifiers. The target fragment identifier and the fragment node set are stored side by side in different fields in the fragment routing result packet. They are distinguished by field identifiers. There is no uniformity of numerical scale or unit conversion. Only structured encapsulation is performed to maintain the original semantics and value form of the target fragment identifier and the fragment node set, thereby generating the fragment routing result packet.

[0089] S3. Based on the fragment node set in the fragment routing result packet, write the data payload into the fragment data segment corresponding to the target fragment identifier, perform index registration on the IoT naming identifier, generate a distributed index item and register it as available, and encapsulate the available status of the distributed index item into a searchable submission receipt packet.

[0090] The fragment node set in the fragmented routing result packet is ordered to obtain the first and successor nodes in the writing chain order. The node ordering is based on the consistency sorting of the fragment node identifier strings. The order of the fragment node identifiers is determined by character consistency comparison to ensure that the node order in the writing chain is consistent. Specifically:

[0091] Within the shard node set, a consistency sort is performed based on the shard node identifier string. The sorted set of shard node identifiers is then registered as the write chain order. The consistency sorting involves reading characters from left to right within the shard node identifier string and performing character consistency comparisons. When the first differing character is found in the character consistency comparison, the order of the shard node identifier strings is determined by the order in which the first differing character appears in the allowed character set of the shard node identifier. The allowed character set of shard node identifiers is derived from the character set of shard node identifiers registered in the shard node list. When no differing character is found in the character consistency comparison, the shard node identifier string with the shorter length is determined as the prior shard node identifier. If duplicate shard node identifiers exist in the write chain order, only the first shard node identifier in the write chain order is retained, and subsequent duplicate shard node identifiers are removed. The first shard node identifier in the write chain order is registered as the first node. All shard node identifiers in the write chain order other than the first node are registered as successor nodes in the order of their appearance.

[0092] At the first node, a write append operation is performed on the data segment corresponding to the target fragment identifier, generating a write fragment, and the data payload is stored in the write fragment, specifically:

[0093] The append write position is located at the end of the data segment corresponding to the target fragment identifier of the first node; the data payload is written to the append write position in field order; the append write position, the data payload field set and the target fragment identifier are registered as a write fragment; the write fragment comes from the append write position and the data payload field set registration item generated by the first node's append write; the append write position is only used for the first node's write fragment registration, and subsequent nodes do not reuse the append write position.

[0094] The write segment is passed hop-by-hop to the successor node in the write chain order. At the successor node, a synchronous write append is performed on the data segment corresponding to the target segment identifier, and the write completion status is output. Specifically:

[0095] The write segment is encapsulated into a hop-by-hop write message and sent to the next successor node in the write chain order. The hop-by-hop write message sets the target fragment identifier field, the write segment field, and the write completion status field. The write completion status field is initially registered as an empty flag. The hop-by-hop write message is sent to the next successor node by reading the fragment node identifier corresponding to the fragment node communication address from the fragment node list registration item, and using the fragment node communication address as the target of the hop-by-hop write message.

[0096] The successor node receives hop-by-hop write messages and decapsulates them to obtain write fragments; it locates the append write position at the end of the data segment corresponding to the target fragment identifier of the successor node; it writes the set of data payload fields in the write fragment to the append write position; it registers the write completion status as the value of the write completion status field and writes it to the write completion status field of the hop-by-hop write message. The write completion status includes a write completion flag and a write failure flag; when the write failure flag appears, it stops sending hop-by-hop write messages to the next successor node.

[0097] After the write completion status field value is registered, the hop-by-hop write messages are sent in reverse order of the write chain to the previous shard node identifier and continuously transmitted back until the first node. The previous shard node identifier refers to the adjacent shard node identifier that is located before the sending shard node identifier in the write chain order. The first node receives the hop-by-hop write messages and reads the write completion status field value, and registers it as a write completion status field value transmission registration item.

[0098] Based on the write completion status, a write completion confirmation is performed on the successor node, generating a write completion message, specifically as follows:

[0099] The write completion status field values ​​of successor nodes are read sequentially according to the write chain order; when the write completion status field value of the successor node is a write completion flag, the shard node identifier of the successor node is registered as the completion node identifier; when the write completion status field value of the successor node is a write failure flag, the shard node identifier of the successor node is registered as the failure node identifier; the write completion status field value is obtained from the write completion status field value return registration item of the successor node's hop-by-hop write message.

[0100] When a failure node identifier exists, the write completion status is registered as a write failure flag and a write completion message is generated. The write completion message contains the target shard identifier, the write failure flag, and the failure node identifier. When a failure node identifier does not exist, the write completion status is registered as a write completion flag and a write completion message is generated. The write completion message contains the target shard identifier, the write completion flag, and a set of completion node identifiers.

[0101] The write completion message is sent back to the first node hop by hop in the write chain order, triggering the index registration of the IoT naming identifier, generating a distributed index entry, and registering it as available, specifically:

[0102] The write completion message is sent in reverse order of the write chain to the previous shard node identifier and continuously transmitted back until the first node; the first node receives the write completion message and reads the write completion flag and write failure flag; when the write completion flag exists, the index registration of the IoT naming identifier is performed. The index registration includes: creating a distributed index entry record in the distributed index storage space; writing the IoT naming identifier to the IoT naming identifier field of the distributed index entry; writing the target shard identifier to the target shard identifier field of the distributed index entry; writing the set of completed node identifiers to the shard node set field of the distributed index entry; writing the available status field of the distributed index entry to the available flag and registering it as available; the distributed index storage space is the storage space for the distributed index entry registration entries, and the distributed index storage space comes from the index registration entries in the storage space of the distributed storage router.

[0103] When a write failure flag exists, the index registration of the IoT naming identifier is not performed, and the available status field is written to the unavailable flag; the unavailable flag is used to indicate that the distributed index item is not registered as available.

[0104] The availability status of distributed index entries and IoT naming identifiers are encapsulated into structured receipt records, which serve as a searchable submission receipt package, specifically:

[0105] Set the IoT naming identifier field and availability status field in the searchable submission receipt package; write the IoT naming identifier into the IoT naming identifier field; when the write completion flag exists, write the availability flag into the availability status field; when the write failure flag exists, write the unavailable flag into the availability status field; register the structured receipt record as the searchable submission receipt package output.

[0106] S4. Based on the retrievability submission receipt packet, verify the resolvability and routableness of the IoT naming identifier in the distributed index item. When the retrievability verification is successful, confirm that the distributed storage write is completed. When the retrievability verification fails, trigger the fragmentation routing adjustment process.

[0107] The searchability submission receipt packet is decapsulated to obtain the availability status of the IoT naming identifier and distributed index item, specifically:

[0108] The IoT naming identifier field value is read from the searchable submission receipt package and registered as the receipt IoT naming identifier; the available status field value is read from the searchable submission receipt package and registered as the receipt available status flag; the receipt IoT naming identifier and the receipt available status flag originate from the searchable submission receipt package.

[0109] Using the IoT naming identifier as the search key, the distributed index entries are matched and located, and the target shard identifier is read from the distributed index entries, specifically:

[0110] The IoT name identifier of the receipt is input as the search key into the distributed index storage space; the distributed index storage space performs a consistency comparison on the search key in the IoT name identifier field of the distributed index item. The consistency comparison is a character-by-character alignment comparison, and the match is confirmed when all characters are completely consistent.

[0111] When the consistency comparison is successful, the matching table entry is registered as an index hit distributed index entry, and the value of the target shard identifier field is read from the index hit distributed index entry and registered as the index target shard identifier. At the same time, the value of the shard node set field is read and registered as the index shard node set, and the value of the availability status field is read and registered as the index availability status flag.

[0112] If the consistency comparison fails, continue to perform a consistency comparison on the IoT naming identifier field of the next table entry until the consistency comparison succeeds and the consistency comparison stops when the distributed index traversal ends.

[0113] When the distributed index item traversal is completed and the consistency comparison fails, the target shard identifier of the index is registered as an empty flag, the set of index shard nodes is registered as an empty flag, and the index availability status flag is registered as an empty flag.

[0114] Perform an existence check on the target shard identifier of the index and generate the existence check result, specifically:

[0115] When the target shard identifier of the index is an empty flag, the existence verification result is marked as failed; when the target shard identifier of the index is not an empty flag, the target shard identifier field of the shard routing table is read to form the target shard identifier field value sequence.

[0116] The target shard identifier field values ​​are read sequentially from the target shard identifier field value sequence, and a character-by-character alignment comparison is performed between the target shard identifier field values ​​and the index target shard identifier. The character-by-character alignment comparison is as follows: characters are read from left to right for both the target shard identifier field value and the index target shard identifier to form character sequences. First, the lengths of the character sequences are compared; if the character sequence lengths are inconsistent, the comparison result is recorded as inconsistent. If the character sequence lengths are consistent, the characters at the same position are compared character by character; if any character is inconsistent, the comparison result is recorded as inconsistent; if all characters at the same position are consistent, the comparison result is recorded as consistent. If any target shard identifier field value in the target shard identifier field value sequence is recorded as consistent with the index target shard identifier, the subsequent reading of target shard identifier field values ​​is immediately stopped, and the existence verification result is recorded as passed. When the target shard identifier field value sequence is traversed and all comparison results are recorded as inconsistent, the existence verification result is recorded as failed.

[0117] For the availability status of distributed index entries and the target shard identifier of the index, perform a consistency check and generate a consistency check result, specifically as follows:

[0118] When the index availability status flag or the receipt availability status flag is empty, the consistency verification result is registered as a failed flag. When both the index availability status flag and the receipt availability status flag are not empty, a character-by-character alignment comparison is performed on the index availability status flag and the receipt availability status flag. The character-by-character alignment comparison is as follows: characters are read from left to right for the index availability status flag and the receipt availability status flag respectively to form a character sequence. First, the length of the character sequence is compared. If the lengths of the character sequences are inconsistent, the consistency verification result is registered as a failed flag. If the lengths of the character sequences are consistent, the characters at the same character position are compared character by character. If any character is inconsistent, the consistency verification result is registered as a failed flag. If all characters at the same character position are consistent, the consistency verification result is registered as a passed flag.

[0119] When both the existence verification and consistency verification results are passed, a searchable verification success flag is generated, and the distributed storage write is confirmed to be complete. Distributed storage write completion means that the data payload corresponding to the IoT naming identifier has been written to the data segment corresponding to the target shard identifier, and the distributed index entry registration has been completed, confirmed by the searchable verification success flag. Specifically:

[0120] Read the values ​​of the existence verification result field and the consistency verification result field; when both the existence verification result field and the consistency verification result field are registered as passed, register a retrieved verification success mark in the retrieveable verification result field.

[0121] The successful retrieval verification mark, the IoT naming identifier of the receipt, the index target shard identifier, the index shard node set, and the receipt availability status mark are encapsulated into a distributed storage write completion confirmation record, and the distributed storage write completion confirmation record is registered as a distributed storage write completion confirmation output.

[0122] When either the existence verification result or the consistency verification result fails, a retrieval verification failure flag is generated, and fragmentation route adjustment processing is triggered. Triggering fragmentation route adjustment processing means, based on the retrieval verification failure flag, re-executing name resolution and target fragment identifier determination, and generating a new fragmentation route result packet to replace the original fragmentation node set. Specifically:

[0123] Read the values ​​of the existence verification result field and the consistency verification result field; when the existence verification result field value is registered as a failed flag, register a searchable verification failure flag in the searchable verification result field; when the consistency verification result field value is registered as a failed flag, register a searchable verification failure flag in the searchable verification result field; encapsulate the searchable verification failure flag and the receipt IoT naming identifier into a fragmented routing adjustment trigger record and output it; using the receipt IoT naming identifier as the naming parsing input, re-execute the hierarchical splitting, prefix sequence generation, prefix matching, and target fragment identifier determination operations, and generate a new fragmented routing result packet with the newly determined target fragment identifier; read the new fragmented node set from the new fragmented routing result packet, and replace the original fragmented node set with the new fragmented node set.

[0124] To further explain, resolvability refers to the ability to accurately determine and map to the corresponding fragment identifier based on the IoT naming identifier, thereby achieving clear location of data storage.

[0125] Routable refers to the ability to dynamically determine the data fragment path through the mapping relationship between IoT naming identifiers and target fragment identifiers. Dynamically determining the data fragment path is achieved in the naming resolution and fragment routing adjustment operations. By using the mapping relationship between IoT naming identifiers and target fragment identifiers, combined with the fragment node set in the fragment routing result packet, appropriate fragment nodes are dynamically selected for data storage and access, thereby achieving efficient data access and transmission.

[0126] It should be noted that the dynamic sharding routing adjustment mechanism in this step (i.e., triggering sharding routing adjustment processing) can effectively cope with the rapid growth in the number of devices and data volume in the IoT environment. Through flexible sharding routing strategies, it can dynamically adjust the data storage path in real time, ensuring high-efficiency storage and access speeds even during large-scale data processing. By intelligently selecting storage nodes, data is evenly distributed, avoiding performance degradation caused by excessive node load. As the number of devices increases, it can automatically expand and adapt to new storage needs, ensuring network stability and high availability. At the same time, it has good fault tolerance, automatically detecting and responding to node failures, ensuring continuous and reliable operation of data storage and routing. Overall, it ensures data storage and routing efficiency in large-scale IoT environments, flexibly copes with complex and changing data loads, and significantly improves storage responsiveness and scalability.

[0127] This embodiment also provides a computer device applicable to the Internet of Things-based distributed storage method for big data, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the Internet of Things-based distributed storage method for big data as proposed in the above embodiment.

[0128] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0129] This embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the distributed storage method for big data based on the Internet of Things as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0130] In summary, this invention achieves high efficiency and flexibility in data storage and routing by performing name resolution on IoT naming identifiers and generating fragmented routing result packets based on target fragment identifiers. Dynamic naming identifier resolution accurately determines the target fragments of data, ensuring reasonable data distribution and efficient storage path selection. During fragmented routing, the precise matching of fragment identifiers with target fragment node sets based on the naming resolution table optimizes data access speed and avoids bottlenecks in traditional static routing strategies. Adaptive routing adjustments allow data to flexibly adapt to the ever-changing IoT environment, achieving immediate response and processing capabilities for large-scale data loads, significantly improving storage efficiency and system responsiveness, while enhancing the scalability and dynamic adjustment capabilities of distributed storage to meet rapidly growing data storage demands.

[0131] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A distributed big data storage method based on the Internet of Things, characterized in that: include, Establish a naming rule to orderly combine the set of object attribute identifiers in the IoT data write request to generate an IoT naming identifier, and encapsulate the IoT naming identifier and data payload into a named write request package; the data payload is a set of fields that carry the actual business data in the IoT data write request. Perform name resolution on the IoT name identifier in the name write request packet to determine the target fragment identifier and generate a fragment routing result packet; Based on the set of fragment nodes in the fragment routing result packet, the data payload is written into the fragment data segment corresponding to the target fragment identifier, and the IoT naming identifier is indexed and registered, generating a distributed index item and registering it as available. The available status of the distributed index item is encapsulated into a searchable submission receipt packet. Based on the retrievability submission receipt packet, the resolvability and routableness of the IoT naming identifier in the distributed index item are verified. When the retrievability verification is successful, the distributed storage write is confirmed to be complete. When the retrievability verification fails, the fragmentation routing adjustment process is triggered. The generation of fragmented routing result packets specifically includes: Establish a naming resolution table and write the mapping relationship between IoT naming identifiers and target fragment identifiers; establish a fragment routing table and register the routing domain information of the target fragment identifier; The IoT naming identifier in the naming request packet is split hierarchically according to the hierarchical boundaries of the naming rules to generate hierarchical fragments; The hierarchical fragments are concatenated level by level according to the naming rules to generate a prefix sequence; Perform prefix matching on the prefix sequence to obtain the matching prefix; In the naming resolution table, the shard mapping record is located by the hit prefix to determine the target shard identifier; Based on the target fragment identifier, select a fragment node set from the fragment routing table, and encapsulate the target fragment identifier and the fragment node set to generate a fragment routing result packet.

2. The distributed big data storage method based on the Internet of Things as described in claim 1, characterized in that: The establishment of naming rules involves inserting fixed separators into adjacent levels of the object attribute identifier set to form a hierarchical string, and then registering the hierarchical string as a naming rule after performing character set unification and null value padding.

3. The distributed big data storage method based on the Internet of Things as described in claim 1, characterized in that: The process of encapsulating the IoT naming identifier and data payload into a naming write request packet specifically involves... According to the naming rules, the set of object attribute identifiers is sorted and ordered to form a sequence of object attribute identifiers; The object attribute identifier sequence is formatted uniformly to obtain a standardized sequence; The standardized sequence is concatenated to generate an IoT naming identifier, and the IoT naming identifier is encapsulated with the data payload to generate a naming write request packet.

4. The distributed big data storage method based on the Internet of Things as described in claim 1, characterized in that: The process of determining the target fragment identifier involves performing table entry location processing in the naming resolution table using the hit prefix as the search key, retrieving the fragment mapping record associated with the hit prefix, and extracting the target fragment identifier from the fragment mapping record.

5. The distributed big data storage method based on the Internet of Things as described in claim 1, characterized in that: The process of generating distributed index entries and registering them as available specifically involves: Perform node ordering on the fragment node set in the fragmented routing result packet to obtain the first node and successor node in the writing chain order; At the first node, the data segment corresponding to the target fragment identifier is appended to the fragment, generating a write fragment, and the data payload is stored in the write fragment; The write segment is passed to the successor node hop by hop in the write chain order. At the successor node, the data segment corresponding to the target segment identifier is synchronously written and appended, and the write completion status is output. Based on the write completion status, perform write completion confirmation on the successor node and generate a write completion message; The write completion message is sent back to the first node hop by hop in the write chain order, triggering the index registration of the IoT naming identifier, generating a distributed index entry, and registering it as available.

6. The distributed big data storage method based on the Internet of Things as described in claim 1, characterized in that: The verification of the resolvability and routableness of the IoT naming identifier in the distributed index item specifically involves... The searchability submission receipt is decapsulated to obtain the availability status of the IoT naming identifier and distributed index item; Using the IoT naming identifier as the retrieval key, the distributed index item is matched and located, and the index target fragment identifier is read from the distributed index item; Perform an existence check on the index target shard identifier and generate the existence check result; Perform consistency verification on the availability status of distributed index items and the target shard identifier of the index, and generate consistency verification results; When the existence verification result is passed and the consistency verification result is passed, a searchable verification success flag is generated, and the distributed storage write is confirmed to be complete. If either the existence verification result or the consistency verification result fails, a searchable verification failure flag is generated, and the sharding route adjustment process is triggered.

7. The distributed big data storage method based on the Internet of Things as described in claim 6, characterized in that: The completion of distributed storage writing means that the data payload corresponding to the IoT naming identifier has been written to the data segment corresponding to the target segment identifier, and the distributed index item registration has been completed, and the successful verification has been marked and confirmed. The triggering of fragmented routing adjustment process refers to re-executing name resolution and target fragment identifier determination based on the searchable verification failure flag, and generating a new fragmented routing result packet to replace the original fragmented node set.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the Internet of Things-based distributed storage method for big data as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the Internet of Things-based distributed storage method for big data as described in any one of claims 1 to 7.

Images