A decentralized storage and distributed storage system

Abstract

The application relates to the technical field of data storage and distributed information processing, in particular to a decentralized storage and distributed storage system, which comprises the following steps: a system receives sensor data, generates basic fields such as device identification, a timestamp, a sampling value and a data type, judges and outputs a data state mark according to the change amount of adjacent sampling values and a historical fluctuation threshold, splices the state mark, the device identification and the timestamp into a routing code, and generates a content identifier through hash calculation, shunts according to the state mark, writes normal data into a first storage area and writes abnormal data into a second storage area, for the abnormal data, the system takes the timestamp as the center, backtracks to search for normal data in a time window and establishes a pointer association, and comprehensively checks the uniqueness of the identifier, the consistency between fields and the writing sequence of the data, so that the processing is accurate; and the application facilitates quick retrieval of context data before and after an abnormal event.

Description

Technical Field

[0001] This invention relates to the field of data storage and distributed information processing technology, specifically a decentralized storage and distributed storage system. Background Technology

[0002] Decentralized and distributed storage technologies are common approaches for data addressing, distributed writing, and collaborative management in multi-node environments. They can organize massive amounts of data through content identification, node routing, and partitioned storage mechanisms, while also balancing storage scalability and the flexibility of data distribution among nodes. As sensor devices continue to connect and the scale of sampled data continues to grow, distributed storage systems are gradually evolving from general file storage to applications that provide categorized storage and rapid retrieval of continuous time-series data. However, when storing data with continuous sampling characteristics, existing technologies typically focus more on completing data on-chain addressing, node distribution, or ordinary partition writing, while neglecting the differentiated storage processing of data corresponding to changes in sampling state. This results in a lack of clear distinction between regular and abnormal data in terms of storage path and intra-node partitioning. At the same time, when it is necessary to trace the related regular data within the time range before and after an abnormal event, existing methods often struggle to balance cross-node retrieval efficiency, time window association integrity, and write consistency, leading to low efficiency in retrieving the context data of abnormal events. Summary of the Invention

[0003] The purpose of this invention is to provide a decentralized and distributed storage system that avoids the lack of clear distinction between regular and abnormal data in storage paths and intra-node partitions, as well as the impact of high-frequency abnormal writes on the archiving of regular data. It also makes it easier to control regular and abnormal data to enter different storage partitions, while taking into account cross-node retrieval efficiency, time window association integrity and write consistency, thereby efficiently retrieving context data before and after abnormal events.

[0004] The objective of this invention can be achieved through the following technical solutions: Receive sensor data streams and generate basic fields, including device identifier, sampling timestamp, sampled value, and data type identifier; Output a status flag based on the change between the current sampled value and the previous sampled value at the same sampling time on the same device; concatenate the status flag, device identifier, and sampling timestamp in a predetermined order to generate a routing code; Select the normal data channel or the abnormal data channel based on the status flag, and generate a content identifier based on the routing code to determine the target storage node; Map the regular data channels to the first storage area set, and map the abnormal data channels to the second storage area set; When the status is marked as abnormal, the set of regular data content identifiers within the preset backtracking time window is retrieved with the sampling timestamp of the abnormal data as the center, and an associated pointer field is established. The associated pointer field is used to record the set of regular data content identifiers. The system verifies the uniqueness of the content identifier, the consistency between the data indicated by the associated pointer field and the backtracking time window, and the order in which the data is written. The preset backtracking time window is a time interval of preset length centered on the sampling timestamp of the abnormal data.

[0005] Preferably, the absolute value of the difference between the current sampled value and the previous sampled value at the previous sampling time on the same device is used as the change amount, and the threshold is determined based on the fluctuation statistics of the device's historical regular data. The fluctuation statistics are statistics of the absolute value of the difference between adjacent sampled values ​​within a preset historical time period. When the change amount is not greater than the threshold, a normal status flag is output, and when the change amount is greater than the threshold, an abnormal status flag is output.

[0006] Preferably, the routing code is formed by concatenating the status flag, device identifier, and sampling timestamp in that order.

[0007] Preferably, data is sent to the normal data channel when the status is marked as normal, and to the abnormal data channel when the status is marked as abnormal.

[0008] Preferably, the data corresponding to the regular data channel is written to the first storage area set, and the data corresponding to the abnormal data channel is written to the second storage area set.

[0009] Preferably, the associated pointer field is an index set used to record the set of identifiers for regular data content.

[0010] Preferably, the uniqueness of the content identifiers within the preset time window is verified; the sampling time of the data corresponding to each content identifier in the associated pointer field is verified to be within the backtracking time window. Verify whether the order in which regular data is written to the first storage area set and abnormal data is written to the second storage area set conforms to the partition mapping rules; Verify whether the content identifier recorded in the associated pointer field of the abnormal data completely corresponds to the set of regular data content identifiers retrieved within the backtracking time window.

[0011] The beneficial effects of this invention are: 1. This invention performs unified parsing and fieldization of device identifiers, sampling times, sampling values, and data types in sensor data streams through the acquisition module, establishing a standardized data entry point for subsequent judgment and storage links. This enables continuous sampling data from different sources to form a consistent data structure before entering the decentralized storage system, thereby improving the execution stability of subsequent state recognition, route generation, and classification writing. 2. This invention, through a state judgment mechanism based on the change between the current sampled value and the previous sampled value, and combined with a threshold determined by the statistical results of historical regular data fluctuations, can distinguish between regular data and abnormal data according to the fluctuation characteristics of the device itself. This avoids the problem in the prior art of simply writing continuous sampled data without state differentiation processing, thereby enhancing the accuracy of data classification. 3. This invention generates routing codes based on status flags, device identifiers, and sampling times, and further generates content identifiers to determine the target storage node. Under a unified decentralized addressing system, it can distinguish data in different states at the addressing input stage, thereby distinguishing normal data from abnormal data not only in terms of logical classification, but also in terms of subsequent storage path selection, thus improving the data diversion capability in a distributed environment. 4. By selecting regular data channels or abnormal data channels based on status flags and mapping different channels to the first storage area set and the second storage area set respectively, this invention enables regular data and abnormal data to form a stable classification and partitioning writing relationship within the node, avoiding the impact of abnormal writing on regular archiving, and facilitating the subsequent priority retrieval of abnormal data and its related information, thereby improving the data organization efficiency and retrieval convenience within the node. Attached Figure Description

[0012] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a structural diagram of a decentralized and distributed storage system according to an embodiment. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] Please see Figure 1 A decentralized and distributed storage system includes a data acquisition module, a status judgment module, a routing code generation module, a data splitting module, a partition mapping module, a data association module, and a data verification module. The acquisition module receives sensor data streams and generates basic fields, including device identifiers. Sampling timestamp Sample values and data type identifier; The state determination module is used to determine the current sample value. Previous sample value at the same sampling time on the same device The change between Output status flags ; The routing code generation module is used to concatenate the code in a predetermined order. , and Generate routing code ; The data routing module is used to select the normal or abnormal data channel based on B, and to generate a content identifier based on K. To determine the target storage node; The partition mapping module is used to map regular and abnormal data channels to the first storage area set respectively. Second storage area set ; The data association module is used in At that time, retrieve abnormal data The central time interval Regular data within the preset backtracking time window W gather and establish a system for recording Associated pointer field , The preset positive time length is determined based on the physical quantity state change decay period of the service in which the equipment is located. The data verification module is used to verify the data. Uniqueness Indicator data and Consistency and write order are verified. The state determination module includes a change calculation unit, a threshold determination unit, and a state output unit; wherein, the change calculation unit is used to calculate the current sampled value. Previous sample value at the same sampling time on the same device The absolute value of the difference is taken as the change. ; The threshold determination unit is used to determine the threshold based on the fluctuation statistics of historical routine data of the equipment. The fluctuation statistics value is a statistical measure of the absolute value of the difference between adjacent sampled values ​​within a preset historical time period; the status output unit is used to... Time output ,exist Time output ; The routing code generation module is used to generate routing codes according to... , and The routing code is formed by concatenating the following sequences. .

[0015] This implementation uses the example of an edge node receiving sensor data streams; the sensor continuously outputs sampled data characterizing the objective operating status, and the sampled data carries at least a device identifier. Sampling timestamp Sample values and data type identifier ; The acquisition module is deployed on the data access side of the edge node, and its execution entity can be the receiving process in the edge computing node; The receiving process receives one or more data packets in each system sampling period and parses the device identifier from the data packets. Sampling timestamp Sample values and data type identifier After forming a basic field set, it is written into the unprocessed cache area of ​​the edge node; Edge nodes transform raw sensor data into a unified field format that can be directly called by subsequent judgment modules, ensuring that sensor data from different sources have a consistent data structure before entering the storage link. When the same device generates the current sample value Then, the status determination module retains the previous sampled value at the previous sampling time on the device. The data is retrieved, and the absolute difference between the two is calculated by the change calculation unit to obtain the change. ,Right now ; Since the above sampled values ​​reflect the objective physical quantities collected within the system's sampling period, therefore This reflects the magnitude of the device's state change between adjacent sampling periods; If a device is connected for the first time in the current period, the edge node first records the current sampled value of the device as a comparison benchmark for the future. After obtaining two consecutive samples, the change judgment is performed to ensure that the change calculation has clear preceding input. The threshold determination unit is used to provide the threshold required for state judgment. The threshold is derived from the statistical values ​​of historical regular data fluctuations of the same device. Specifically, the edge nodes extract the sampling sequence of data that has been determined to be regular data from a preset historical time period, calculate the absolute difference between adjacent sampling values, and then statistically analyze these absolute differences to obtain the historical fluctuation level. Preferably, the threshold The variation can be based on historical regular samples. The mean and standard deviation are used to determine the value, which is calculated by adding a factor of the standard deviation to the mean of historical changes; for example, suppose the mean of historical changes is... The standard deviation is The threshold determination logic is then specified as the formula:

[0016] in, The inherent quantization noise floor deviation value of the analog-to-digital conversion hardware of this type of sensor, which is pre-determined by the system; This is the preset magnification factor; This represents the mean of the changes in the historical sample. Standard deviation; The magnification factor is the preset magnification factor. , The preferred value range is 1.0 to 3.0. The same equipment has natural fluctuations under normal operating conditions. If the threshold is too small, normal fluctuations may be misjudged as abnormalities. If the threshold is too large, sudden changes in the state may be missed. Determining the threshold through the statistical results of historical normal data can make the state judgment match the fluctuation characteristics of the equipment itself. The status output unit receives and Then execute explicit conditional judgments; when When, the status output unit outputs This indicates that the current data belongs to the category of regular data; when When, the status output unit outputs This indicates that the current data is abnormal. Here It is the direct control variable for subsequent routing and traffic splitting. It has only two possible values, which makes it easy for edge nodes, target storage nodes and local partitions to adopt a unified judgment rule. In status flags After output, the route code generation module begins to work; the execution body of this module can be the route preprocessing process in the edge node, used to pre-inject state information into the address input string; specifically, the route code generation module follows... , and The routing code is formed by concatenating the data in a fixed order. ; In this embodiment, The order of their composition is as follows: state flags For equipment identification , is the sampling timestamp The status type is placed at the beginning, so that normal data and abnormal data are prefixed differently in subsequent addressing stages. Reintroducing equipment identification Used to distinguish the data sources of different devices; timestamps are introduced. This routing code is used to distinguish data packets from different sampling periods of the same device; this is how the routing code is formed. It retains both the device and time dimensions while also reflecting the differences in status; After obtaining the routing code Afterwards, the data splitting module... Perform hash calculations to generate content identifiers. Specifically, the data routing module can use a secure hash algorithm to generate 256 pairs of string-encoded routing codes. Perform calculations to obtain a fixed-length hash value as... ; Should As a content addressing identifier in distributed storage, it is used to determine the target storage node; the process of determining the target storage node is as follows: the system calculates based on the distributed hash table protocol. With each candidate storage node in the network The logical distance between them, such as the XOR distance, and the one with the smallest logical distance. One node was selected as the target storage node, among which It is a positive integer greater than or equal to 3; At the same time, the data splitting module also relies on status flags Select write channel: When At that time, the data enters the regular data channel; when At this time, the data enters the abnormal data channel; it can be seen that the status judgment result is used to directly control the path selection of the target storage link; Data generated by the same device in different states, although still using a unified decentralized addressing system, differs due to the input routing code. It contains different state prefixes, which can ultimately be distinguished at both the addressing level and the channel level; Once the data channel is determined, the partition mapping module maps the logical channel to different storage partitions on the node's local machine; for data that is determined to enter the regular data channel, the partition mapping module writes it into the first storage area set. ; For data entering the abnormal data channel, the partition mapping module writes it into the second storage area set. The first set of storage areas here The second storage area is used to hold continuously accumulated regular sampling data. Used to handle abnormal data with large variations; Regular data and abnormal data are written to different areas within the node to avoid the impact of high-frequency abnormal writes on the archiving of regular data, and also to facilitate the quick retrieval of related data based on abnormal events. When the status flag When this happens, the data association module is triggered to establish the association; the execution entity of this module can be the index processing process on the target storage node. The indexing process uses the sampling timestamp of the abnormal data. Centered on, the construction range is Retrospective time window Its total window length is Here, the positive time length The specific value is determined based on the physical quantity state change decay period of the service where the equipment is located. For example, for slowly varying physical quantities such as temperature, The time span can be set to 5 to 10 system sampling periods; for rapidly changing physical quantities such as voltage and current, The time span can be set to 1 to 3 system sampling periods to ensure that the correlation window can accurately and completely cover the correlation fluctuation data before and after the outbreak of the abnormal event; Considering the backtracking time window Includes For data in the future time period, the indexing process should preferably use an asynchronous delayed triggering mechanism; Once abnormal data is detected, the association establishment task first enters the waiting queue of the target node, waiting for the system time to pass. The time or the received timestamp is greater than After processing the regular data, the indexing process then officially begins the retrieval process; Because regular data and abnormal data have different prefixes during the routing code generation process, the content identifier generated by the secure hash algorithm... The technical characteristic that a single bit input flip results in an output bit flip rate greater than or equal to 50% means that when addressed by a distributed hash table, the data is hashed and distributed across different target storage nodes in the network. Therefore, it is impossible to directly traverse and retrieve physical data within the local storage area of ​​a single target node. Therefore, in this embodiment, the indexing process performs cross-node retrieval using a deterministic hashing method: the indexing process relies on the device identifier of the same device. Given the known system sampling period step size and window time range, iterate through all expected regular data sampling timestamps within the generated time interval and force a prefix. To reconstruct the regular routing code and calculate the theoretically existing regular data. gather; Through distributed hash networks to these theories Other target nodes send metadata probe requests to confirm whether regular data whose sampling time falls within the full window range is already in the first storage area set of their respective nodes. The probe truly survives and returns verified results. Summarize and form a valid set of regular data CIDs. ; Content identifiers assigned based on abnormal data The data association module generates an independent key-value pair mapping instruction in the distributed hash network metadata layer, which is independent of the physical payload of the abnormal data. This mapping instruction will handle abnormal data. As an index key, the set of regular data content identifiers As an associated pointer field Perform external storage mounting; Here Instead of modifying the physical structure of the abnormal data itself, it establishes an external multidimensional relationship graph for immutable content identifiers based on a decentralized network architecture. By adopting this method, a traceable correspondence is established between anomalous data and regular data within its time window, overcoming the barriers of physical node distribution. This allows for direct retrieval of anomalous events based on this information. Relevant routine data before and after positioning; After data writing and association establishment are completed, the data verification module performs pre-archiving verification; the verification includes at least three items. right Uniqueness is verified, that is, compared with the current value within a preset check range. Check if it already exists to prevent different data packets from using the same content identifier due to abnormal writing or duplicate submission; right The indicated data and the backtracking time window The consistency is verified, that is, the check is performed. Each recorded in Are the sampling times of the corresponding data all within the interval? Inside; The data write order is verified, that is, it is checked whether regular data has been written to the first storage area set. Whether abnormal data should be written to the second storage area set. This is to ensure that the aforementioned partition mapping rules are executed; only after the above verification passes will the current data packet and its index relationship be retained as a valid archiving result. Through the above processing, the edge nodes start from the sensor sample values, first forming a unified basic field, and then outputting a status flag based on the difference between adjacent samples. Then according to , and Generate routing code ,Finish Generate, channel split, local partition writing, record association between abnormal data and regular data within the window, and verify the writing results; This processing chain directly applies changes in objective sampled values ​​to decentralized addressing and intra-node partition writing, enabling data from the same device to enter different partitions via different storage paths when the state changes. It also retains regular data indexes within adjacent time ranges in abnormal situations, facilitating subsequent retrieval of context data before and after abnormal events.

[0017] In a preferred embodiment of the present invention, the data splitting module includes a channel selection unit, which is used for selecting data streams in a specific manner. The data is then sent to the regular data channel. Data will be sent to the abnormal data channel at that time; The partition mapping module includes a large-capacity partition mapping unit and a high-speed partition mapping unit; the large-capacity partition mapping unit is used to write data corresponding to the regular data channel into the first storage area set. The high-speed partition mapping unit is used to write the data corresponding to the abnormal data channel into the second storage area set. .

[0018] This implementation focuses on the cooperation between the data splitting module and the partition mapping module; the channel selection unit is set in the completion status judgment and After generation, its input must include at least a status flag. The current data packet and the routing code The obtained content identifier During execution, the target storage node or edge-side forwarding process first reads the status flags. Then select the data channel according to the binary judgment rule; when At that time, the channel selection unit will select the current data packet along with its... Send to the regular data channel; when At that time, the channel selection unit will select the current data packet along with its... Send to the abnormal data channel; the regular data channel is used to carry data whose change range does not exceed the threshold, while the abnormal data channel is used to carry data whose change range exceeds the threshold. Therefore, the channel selection action is to allocate the storage path for the objective operating status. After the channel selection is completed, the partition mapping module performs local persistent storage mapping on the selected channel; the large-capacity partition mapping unit and the high-speed partition mapping unit in the partition mapping module correspond to different types of data writing requirements; First storage area set Second storage area set It is a standardized physical volume pool that is pre-divided uniformly by each distributed storage node during the system's underlying initialization. This means that regardless of the content identifier... Addressing determines which specific physical node in the network to which data is allocated, and that node can provide logically consistent and structurally symmetrical categorized storage partitions; If the data packet originates from a regular data channel, the mass partition mapping unit writes the data to the first storage area set on the current node's local machine based on the channel identifier. Since regular data is typically generated repeatedly within consecutive sampling periods, therefore Used to carry continuously accumulated sampling records over a long period of time; If the data packet originates from an abnormal data channel, the high-speed partition mapping unit writes the data to the second storage area set on the current node based on the channel identifier. Abnormal data corresponds to sudden changes in sampled values. This type of data is used to retain it so that it can be read first during subsequent retrieval or verification; In this embodiment, the channel selection unit and the partition mapping module work together using a field passing method; when the channel selection unit outputs the current data packet, it simultaneously outputs the channel type to which the data packet belongs; After the partition mapping module reads the channel type, it no longer performs a new status check, but directly executes the write action based on the determined channel type; thus avoiding the same data packet being repeatedly checked in different processing stages and reducing the risk of inconsistent status. For example, within a system sampling period, the same device generates multiple sampling data packets, one of which, after status determination, obtains... The data packet is first sent to the regular data channel by the channel selection unit, and then written by the large-capacity partition mapping unit. ; Another data packet was obtained after status determination. In this case, the data packet is first sent to the abnormal data channel by the channel selection unit, and then written by the high-speed partition mapping unit. Both types of data are processed separately on both logical channels and physical partitions, allowing for quick location of the required data type based on the partition category during subsequent readings. In the above manner, the channel selection unit completes the conversion from status flag to logical channel, and the large-capacity partition mapping unit and the high-speed partition mapping unit complete the conversion from logical channel to physical storage area set; The preprocessing process maps and allocates data categories to logical storage paths, while the subsequent processing process allocates the distributed data to the physical storage area set for disk writing. After the two processes are executed in sequence, regular data and abnormal data form a stable partition writing relationship within the target node, which is beneficial for achieving data classification and storage within the node without changing the decentralized addressing result.

[0019] In a preferred embodiment of the present invention, the associated pointer field For recording routine data The set of indices; The data verification module includes an identifier verification unit, a time range verification unit, a storage order verification unit, and an association verification unit; The identifier verification unit is used to verify the identifier within a preset time window. Perform uniqueness verification; the time range verification unit is used for verification. Each of the Is the sampling time of the corresponding data within the backtracking time window? Inside; The storage order verification unit is used to verify the writing of regular data into the first storage area set. Furthermore, abnormal data is written to the second storage area set. Does the order conform to the partition mapping rules? The associated validation unit is used to validate the associated pointer field of abnormal data. Recorded in Does it completely correspond to the backtracking time window? The CID set of regular data obtained from internal retrieval .

[0020] This implementation focuses on the indexing process between abnormal data and regular data, as well as the verification process surrounding this index; the data association module uses status markers. As a trigger condition, the association processing process on the target storage node will only start the retrieval action when the status judgment module determines that the current data is abnormal data; The associated processing process reads the sampling timestamp of the current abnormal data. And generate a backtracking time window centered on this timestamp. ; The retrospective time window The time span is a pre-set positive time length. This limitation restricts the associated regular data to the time intervals immediately preceding and following the outlier data. Within, rather than infinitely extending to a longer historical sequence; After obtaining the backtracking time window Subsequently, to address the cross-node retrieval obstacles caused by content addressing resulting in regular and abnormal data being scattered across different storage nodes, the association processing process does not directly access the local first storage area set. Instead of performing a full scan, a forward state algorithm is used to perform targeted cross-node retrieval; The specific implementation steps are as follows: In order to ensure that the retrieval action can be completely obtained... Subsequent routine data processing will be handled by the correlation process once the system clock exceeds a certain threshold. That is, after confirming that all future data in the window should be ready, extract the corresponding device identifier. And with a preset system sampling period as the time step, in the interval Internal discretization generates a series of expected regular data timestamp sequences; Forced status marking The corresponding regular data routing code sequence is reconstructed according to the established generation rules, and a hash calculation is performed to obtain the target verification result. Collection; based on a distributed network protocol, verify the content identifier to the target. Each storage node pointed to by the set initiates a metadata verification request for a data packet with a size of less than 512 bytes; After receiving the request, each remote node sets up its local first storage area. Check if the corresponding one exists. And the The internal sampling timestamp of the corresponding data must be no less than and not greater than conditions; If the conditions are met, a hit confirmation message is returned to the abnormal data node; the associated processing process will send all data nodes that have received hit confirmation messages to the corresponding nodes. Extracting these data yields a set of regular data CIDs that have been confirmed to have been written to disk. ; It should be noted that the abnormal data written here is not the regular data content itself, but rather the corresponding data. Sets; associated processing processes will Write the associated pointer field of the current exception data ; As an index collection field, it is used to record the identifier of the regular data content associated with the current abnormal data; With this processing method, when reading abnormal data subsequently, only access... This allows you to locate regular data within the same time window without having to perform a window search on all regular data again. After the index is written, the data verification module checks the write results item by item; the identification verification unit checks the results within a preset time window. Perform uniqueness verification; the preset time window is a global verification cycle set by the system to prevent duplicate detection, independent of the aforementioned local backtracking time window for individual devices. ; The length of the global verification period is set to be greater than that of the distributed storage network. The maximum network latency for broadcast synchronization can be preset to, for example, the latency before the current verification time. to The interval is used to ensure that the anti-duplicate comparison covers all data blocks to be synchronized that may be in a state of network transmission delay; The verification process reads existing data within the preset time window. Set up and include the newly written data. With existing Compare them one by one; if a match is found If a record is used repeatedly for different data records, the uniqueness check will fail. Uniqueness check is used to prevent confusion of the identifier caused by repeated writing, repeated addressing, or abnormal retransmission. Time range verification unit Each of the The corresponding data execution time is checked; specifically, the verification process reads the data sequentially. Each recorded in Then check the reverse For the sampling timestamps of the regular data, determine whether they fall within the backtracking time window corresponding to the current abnormal data. That is, whether it satisfies the condition of being located in the interval Inside; if any If the sampling time of the corresponding data exceeds the backtracking time window range, then the associated pointer field is determined. The data contains records whose time context is inconsistent with the current abnormal data, so this check fails. The storage order verification unit is used to check whether the data write position across nodes conforms to the established global partition mapping rules; during verification, the verification process extracts the physical storage write receipt of abnormal data records in the local memory cache to confirm that the abnormal data has been successfully written to the second storage area set of this node. ; Simultaneously, the verification process extracts the partition status field of regular data records in each relevant remote node by parsing the confirmation message payload returned by the aforementioned metadata verification request, thereby confirming that each associated regular data has been written to the first storage area set of its target node. ; Sequence verification, in essence, is to check whether the discretely distributed data in a decentralized network architecture has been strictly written to the storage area in isolation along the predetermined channel mapping relationship, ensuring that no node writes data to an unexpected physical partition due to abnormal execution of local processes; If abnormal data is found to have been mistakenly written locally Or, it may be discovered in the remote message that some regular data has been mistakenly written to the target node. If the result is inconsistent with the system's established rules, it means that the physical execution result of the partition mapping is inconsistent with the system's established rules. In this case, the check should be judged as failed. Further checks on the associated verification unit Recorded in Does it completely correspond to the backtracking time window? The CID set of regular data obtained from internal retrieval ; Its execution process is as follows: First, based on the timestamp of the current abnormal data... Sum of positive time length Re-establishing the backtracking time window Then retrieve the regular data in that window again. The set will be retrieved and then compared with the set obtained from the previous retrieval. Recorded in The sets are compared; if they correspond completely, it means that the associated pointer field of the abnormal data is recorded completely. like The middle part is missing a certain element that should be associated with it. or includes outside a certain window If the result is not found, it means that the associated record is incomplete or inaccurate, and the validation will fail. In a specific data processing cycle, if some abnormal data has been successfully written to the second storage area set , and its The document contains several standard data entries within the window. Then the data verification module completes the process sequentially. Uniqueness check, time range check, partition mapping result check, and Integrity check; Only when all these checks pass can the abnormal data and its relationships be retained as valid archive results; if any one of the checks fails, the corresponding data record or related field must be rewritten or rebuilt. if If uniqueness, time range, or correlation integrity checks fail, the verification process will trigger a retry instruction, discarding the currently generated abnormal data. And wait for the latest synchronized data to be ready before re-initiating the window search and association establishment; If the partition mapping result fails the check, the verification process will generate a quantization remapping instruction, which will then be used to correct any errors. Abnormal data or accidental entry The regular data is forcibly extracted and rewritten to the target physical partition; Here, the quantization remapping instruction is specified as an execution rule with a defined data structure, which includes the following fields: the data to be migrated. The source address, destination physical partition address, and maximum number of retries for the erroneous physical partition; When the verification process executes this instruction, it first locks and extracts data into the memory buffer based on the source address, then initiates writing based on the target physical partition address. After the target partition returns a confirmation signal that the writing was successful, it sends a clear instruction to the source address to delete the erroneous record, and performs verification again after the migration is completed. Through the above-mentioned specific rewriting and reconstruction closed-loop mechanism, the temporal and indexing relationships between abnormal data and its preceding and following regular data are kept consistent. Through the above implementation method, abnormal data is no longer stored simply as a separate data block, but rather... With regular data in the preset window The data sets are associated; at the same time, the data verification module confirms the association result from four aspects: identifier, time range, partitioning rules, and association integrity. When analyzing abnormal events, you can directly rely on the abnormal data itself and its associated pointer fields. Read regular data within the corresponding time range to maintain consistency between the data retrieval process and the sampling time.

[0021] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A decentralized and distributed storage system, characterized in that, It includes a data acquisition module, a status judgment module, a routing code generation module, a data splitting module, a partition mapping module, a data association module, and a data verification module; The acquisition module is used to receive sensor data streams and generate basic fields, including device identifiers. Sampling timestamp Sample values and data type identifier; The state determination module is used to determine the current sample value. Previous sample value at the same sampling time on the same device The change between Output status flags ; The routing code generation module is used to concatenate in a predetermined order. , and Generate routing code ; The data splitting module is used to select a regular or abnormal data channel based on B, and based on Generate content identifier To determine the target storage node; The partition mapping module is used to map regular and abnormal data channels to the first storage area set respectively. Second storage area set ; The data association module is used to... At that time, retrieve abnormal data The central time interval As a preset backtracking time window Regular data within gather and establish a system for recording Associated pointer field , The preset positive time length is determined based on the physical quantity state change decay period of the service in which the equipment is located. The data verification module is used to verify... Uniqueness Indicator data and Consistency and write order are verified.

2. The decentralized storage and distributed storage system according to claim 1, characterized in that, The state determination module includes a change calculation unit, a threshold determination unit, and a state output unit; wherein, the change calculation unit is used to calculate the current sampled value. Previous sample value at the same sampling time on the same device The absolute value of the difference is taken as the change. ; The threshold determination unit is used to determine the threshold based on the fluctuation statistics of historical routine data of the device. The fluctuation statistics value is a statistical measure of the absolute value of the difference between adjacent sampled values ​​within a preset historical time period; the status output unit is used to... Time output ,exist Time output .

3. The decentralized storage and distributed storage system according to claim 1, characterized in that, The routing code generation module is used to... , and The routing code is formed by concatenating the following sequences. .

4. The decentralized storage and distributed storage system according to claim 1, characterized in that, The data splitting module includes a channel selection unit, which is used for... The data is then sent to the regular data channel. Data will be sent to the abnormal data channel at that time.

5. A decentralized storage and distributed storage system according to claim 1, characterized in that, The partition mapping module includes a large-capacity partition mapping unit and a high-speed partition mapping unit; The large-capacity partition mapping unit is used to write the data corresponding to the regular data channel into the first storage area set. The high-speed partition mapping unit is used to write the data corresponding to the abnormal data channel into the second storage area set. .

6. A decentralized storage and distributed storage system according to claim 1, characterized in that, The associated pointer field For recording the aforementioned routine data The set of indices.

7. A decentralized storage and distributed storage system according to claim 1, characterized in that, The data verification module includes an identifier verification unit, a time range verification unit, a storage order verification unit, and an association verification unit. The identifier verification unit is used to verify the identifier within a preset time window. Perform a uniqueness check; The time range verification unit is used to verify the Each of the Whether the sampling time of the corresponding data is within the backtracking time window Inside; The storage order verification unit is used to verify the writing of regular data into the first storage area set. Furthermore, abnormal data is written to the second storage area set. Does the order conform to the partition mapping rules? The association verification unit is used to verify the association pointer field of abnormal data. Recorded in Does it completely correspond to the backtracking time window? Regular data obtained from internal retrieval gather .

Images