A method and system for eliminating data redundancy of distributed industrial storage equipment

Abstract

The present invention discloses a method and system for eliminating data redundancy in distributed industrial storage equipment, and relates to the field of distributed storage technology. The method comprises: receiving an external data block and performing segmentation processing to obtain a plurality of data shards; based on the data shards, applying a dynamic hierarchical fingerprint algorithm to generate a load-associated fingerprint; matching the load-associated fingerprint with a global fingerprint library, determining whether the data shard is redundant data based on the matching result, and selecting a target storage node based on the load-associated fingerprint; storing the non-redundant data shard to the target storage node, and updating the global fingerprint library and the local fingerprint library of the target storage node. By fusing data features with the load status of the storage node to generate a load-associated fingerprint, the fingerprint is dynamically adjusted to match the node load trend, and intelligent data distribution is achieved, solving the problem of uneven data distribution, and achieving the effect of efficiently eliminating redundancy and balancing loads in distributed industrial storage.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage technology, and in particular to a method and system for eliminating data redundancy in distributed industrial storage equipment. Background Art

[0002] In distributed industrial storage scenarios, industrial equipment continuously generates massive amounts of data. This data urgently requires efficient management and redundancy control within distributed storage systems to ensure data security and stable system operation. In this process, data redundancy elimination technology becomes a core element in maintaining storage system performance.

[0003] Current mainstream data redundancy elimination technologies often use data fingerprint generation for matching detection to identify and eliminate duplicate data. However, these fingerprints focus solely on the data's characteristics and fail to consider the load status of the storage nodes.

[0004] This processing model disconnects data storage decisions from the actual operational status of nodes. Large amounts of similar data tend to be concentrated on a few nodes, overloading those nodes while leaving other nodes idle. This not only severely reduces the overall efficiency of the storage system but also threatens system stability, making it difficult to achieve a dynamic balance between data storage and node load. Summary of the Invention

[0005] The object of the present invention is to provide a method and system for eliminating data redundancy in distributed industrial storage devices to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a method for eliminating data redundancy in a distributed industrial storage device, comprising the following steps:

[0007] Receive external data blocks and perform segmentation processing to obtain several data fragments, providing standardized processing units for subsequent fingerprint generation;

[0008] Based on the data sharding, a dynamic hierarchical fingerprint algorithm is applied to generate a load-related fingerprint. The load-related fingerprint integrates the data sharding characteristics and the storage node load status to provide a basis for data redundancy detection and node selection.

[0009] Matching the load-associated fingerprint with a global fingerprint library, determining whether the data shard is redundant data based on the matching result, and selecting a target storage node based on the load-associated fingerprint;

[0010] Storing the non-redundant data slices in the target storage node, and updating the global fingerprint library and the local fingerprint library of the target storage node to reflect the latest storage status of the data;

[0011] Regularly analyze the load status of each storage node and the global fingerprint library, migrate data shards based on the analysis results, and synchronously update the relevant fingerprint library to achieve dynamic balancing of the system load.

[0012] Preferably, the receiving of the external data block and segmentation processing to obtain a plurality of data fragments, providing a standardized processing unit for subsequent fingerprint generation, includes:

[0013] Receive external data blocks in industrial scenarios;

[0014] The external data block is divided into a plurality of data slices according to a preset fixed size, and the preset fixed size is set according to industrial data characteristics and system processing requirements.

[0015] Preferably, based on the data sharding, a dynamic hierarchical fingerprint algorithm is applied to generate a load-associated fingerprint, wherein the load-associated fingerprint integrates the data sharding characteristics and the storage node load status to provide a basis for data redundancy detection and node selection, including:

[0016] Use SHA-256 hash algorithm to process the data fragments to generate basic fingerprints , introduce the timestamp hash fusion mechanism to generate timestamps for data Converted into a time signature code through an irreversible encryption algorithm , and the basic fingerprint Time signature Perform XOR operation to obtain enhanced basic fingerprint , the formula is: , where Represents exclusive OR operation;

[0017] Extract the real-time load parameters of the storage node, including CPU usage , memory usage , disk I / O rate , the real-time load parameters are standardized to obtain standardized load parameters , introduce reinforcement learning model to learn the weight coefficients of various real-time load parameters in different industrial scenarios , the load characteristic value is obtained by weighted summation , and adopt the fusion coefficient 、 The enhanced basic fingerprint and load characteristic value Fusion, generating load association fingerprints , the formula is:

[0018] ;

[0019] Where, is the number of real-time load parameters, is the load characteristic value , ;

[0020] Combined with the LSTM model to predict the load trend of the storage node within a preset time period in the future, and set the predictive load threshold interval , adjust the fusion coefficient according to the prediction results When the load is expected to exceed Increase , when the forecast load will be lower than Time decrease , to dynamically adjust the load association fingerprint .

[0021] Preferably, matching the load-associated fingerprint with a global fingerprint library, determining whether the data shard is redundant data according to the matching result, and selecting a target storage node based on the load-associated fingerprint includes:

[0022] Comparing the load-associated fingerprint with the fingerprints in the global fingerprint library, if the same fingerprint exists, determining that the corresponding data slice is redundant data; if the same fingerprint does not exist, determining that the corresponding data slice is non-redundant data;

[0023] For non-redundant data shards, based on the load correlation degree reflected by the load correlation fingerprint, a storage node with a lower load and a higher correlation degree with the load correlation fingerprint is selected as the target storage node. The correlation degree is the degree of matching between the storage node load state and the load characteristic part in the load correlation fingerprint. The matching degree is measured by the absolute value of the difference between the storage node load parameter and the load characteristic value. The smaller the absolute value of the difference, the higher the correlation degree.

[0024] Preferably, the storing of the non-redundant data shards to the target storage node and updating the global fingerprint library and the local fingerprint library of the target storage node to reflect the latest storage status of the data includes:

[0025] Storing the non-redundant data slices in the target storage node;

[0026] The load-associated fingerprint of the non-redundant data shard and its corresponding target storage node information are updated to the global fingerprint library. At the same time, the target storage node records the load-associated fingerprint to the local fingerprint library. The update of the global fingerprint library needs to be cross-checked by verification nodes of at least 3 different physical racks.

[0027] Preferably, the periodic analysis of the load status of each storage node and the global fingerprint library, migrating data shards according to the analysis results and synchronously updating the relevant fingerprint library to achieve dynamic system load balancing includes:

[0028] Periodically collecting load status information of each storage node and analyzing data shard storage distribution in combination with the global fingerprint library, wherein the periodicity is a preset time interval, and the preset time interval can be adjusted according to system load characteristics;

[0029] For storage nodes with excessive load, redundant data shards or data shards with low load correlation with the node are migrated to storage nodes with lower load according to the load correlation fingerprint of the data shards stored on them;

[0030] The global fingerprint database and the local fingerprint databases of the storage nodes involved in the migration are updated synchronously.

[0031] Preferably, it also includes:

[0032] When an extreme working condition is detected, a multi-level fingerprint degradation mechanism is used to process data fragmentation. After the system returns to normal, the local fingerprints generated during the degradation process are synchronized to the global fingerprint library to ensure the continuity and integrity of data processing;

[0033] The multi-level fingerprint degradation mechanism includes:

[0034] When downgrading to the first level, the basic fingerprint layer processing is retained, and the load association layer and dynamic adjustment layer processing are suspended;

[0035] A 128-bit simplified hash algorithm is used to generate fingerprints during the second-level downgrade;

[0036] Enable local cache fingerprint during level 3 downgrade.

[0037] The present invention also provides a data redundancy elimination system for distributed industrial storage devices, comprising:

[0038] A data processing module is configured to receive an external data block and perform segmentation processing to obtain a plurality of data shards, and then generate a load-related fingerprint based on the data shards using a dynamic hierarchical fingerprint algorithm. The load-related fingerprint integrates the characteristics of the data shards and the load status of the storage node;

[0039] The fingerprint management module is used to maintain the global fingerprint library and the local fingerprint library of each storage node, support matching the load-associated fingerprint with the global fingerprint library, and update the global fingerprint library and the local fingerprint library of the target storage node after data storage;

[0040] A node selection and scheduling module is used to determine whether a data shard is redundant data based on the matching result of the load-related fingerprint and the global fingerprint library, select a target storage node based on the load-related fingerprint, and regularly analyze the load status of each storage node and the global fingerprint library, and migrate data shards based on the analysis results;

[0041] The load monitoring module is used to monitor the load parameters of each storage node in real time and provide them to the data processing module to generate load-related fingerprints. It also provides a load status basis for node selection and data migration in the node selection and scheduling module.

[0042] Storage nodes are used to store non-redundant data shards and maintain local fingerprint libraries, responding to data read, write, and migration requests.

[0043] The present invention further provides an electronic device, wherein the electronic device is a physical device and comprises:

[0044] a processor and a memory, wherein the memory is communicatively connected to the processor;

[0045] The memory is used to store at least one executable instruction executed by the processor, and the processor is used to execute the executable instruction to implement the data redundancy elimination method of the distributed industrial storage device as described above.

[0046] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for eliminating data redundancy in the distributed industrial storage device as described above is implemented.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] By fusing data features with the load status of storage nodes to generate load-associated fingerprints, the fingerprints are dynamically adjusted to match the node load trends, achieving intelligent data distribution and solving the problem of uneven data distribution. Redundant data is accurately identified through fingerprint matching, and storage nodes are selected based on the node load status to improve storage efficiency and system performance. Regular optimization and adjustment and extreme working condition processing mechanisms are used to ensure stable system operation, achieving the effect of efficiently eliminating redundancy and balancing loads in distributed industrial storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A main flow chart of a method for eliminating data redundancy in a distributed industrial storage device provided by an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of the structure of a data redundancy elimination system for distributed industrial storage devices provided by an embodiment of the present invention;

[0051] Figure 3 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0053] The execution subject of the method of this embodiment is a terminal, which can be a mobile phone, tablet computer, PDA, notebook or desktop computer, etc. Of course, it can also be other devices with similar functions, and this embodiment is not limited.

[0054] See also Figure 1 The present invention provides a method for eliminating data redundancy in distributed industrial storage devices. The method is applied to eliminate data redundancy in industrial storage devices in complex industrial scenarios, comprising:

[0055] Step 100: Receive an external data block and perform segmentation processing to obtain a number of data slices, providing a standardized processing unit for subsequent fingerprint generation.

[0056] Specifically, step 100 includes:

[0057] Step 110: Receive external data blocks in an industrial scenario, where the external data blocks include industrial data such as sensor data and log data;

[0058] It's important to note that "external data blocks in industrial scenarios" encompass data generated by various types of equipment during the industrial production process, such as real-time operational data like temperature, pressure, and flow collected by sensors, production log data, and equipment status data. Receiving these data blocks is the initial step in data processing for distributed industrial storage devices. Their purpose is to introduce raw data from industrial sites into the storage system, providing processing targets for subsequent operations like data segmentation and fingerprint generation. By receiving this data, critical information from the industrial production process is captured and stored in a timely manner, providing a foundation for subsequent data analysis and redundancy elimination, thereby ensuring comprehensive management of production data by the industrial storage system.

[0059] Step 120: Segment the external data block into a plurality of data slices according to a preset fixed size, wherein the preset fixed size is set according to industrial data characteristics and system processing requirements;

[0060] The operation of "splitting external data blocks into several data shards of a preset fixed size" breaks down large external data blocks into smaller, easier-to-process units. This facilitates subsequent operations such as independent fingerprint generation and redundancy detection for each data shard, improving data processing efficiency and flexibility. Furthermore, uniformly sized data shards facilitate balanced data distribution and management across distributed storage nodes, avoiding storage and processing imbalances caused by varying data block sizes.

[0061] The "Preset Fixed Size" is determined based on the characteristics of industrial data (such as data volume, update frequency, read / write requirements), as well as the performance of the storage system (such as storage capacity, processing speed, and transmission bandwidth). For example, it can be set to 32KB, 64KB, or 128KB. For small sensor data (single data entry <1KB), choose 32KB to reduce the number of shards; for medium-sized log data (single data entry 1-10KB), choose 64KB; and for large device status snapshots (single data entry >10KB), choose 128KB. This size can be manually adjusted through the system initialization configuration interface, or the system will automatically recommend a size based on the first hour of data.

[0062] Step 200: Based on the data shards, a dynamic hierarchical fingerprint algorithm is applied to generate a load-associated fingerprint. The load-associated fingerprint integrates the data shard characteristics and the storage node load status to provide a basis for data redundancy detection and node selection.

[0063] Specifically, step 200 includes:

[0064] Step 210: Process the data fragments using the SHA-256 hash algorithm to generate a basic fingerprint. , introduce the timestamp hash fusion mechanism to generate timestamps from data Converted into a time signature code through an irreversible encryption algorithm , and the basic fingerprint Time signature Perform XOR operation to obtain enhanced basic fingerprint , the formula is: , where Represents exclusive OR operation;

[0065] Among them, the "SHA-256 hash algorithm" is a cryptographic hash function that can convert input data of any length into a hash value of fixed length (256 bits), which is the basic fingerprint. , which is unique and irreversible, and can accurately identify the content of the data shard. In the "timestamp hash fusion mechanism", "data generation timestamp " is the specific time information generated by data sharding, which is converted into a time signature through an irreversible encryption algorithm (such as the MD5 algorithm) , and then with the basic fingerprint Perform XOR operation to obtain enhanced basic fingerprint The purpose of this step is to add a temporal attribute to the fingerprint of the data shard, so as to avoid the same content data generated at different times being misjudged as redundant data, thereby improving the accuracy of redundancy detection. It is especially suitable for processing similar data generated periodically in industrial scenarios.

[0066] It should be noted that the characteristics of the XOR operation enable the enhanced basic fingerprint to retain the characteristic information of the original data segment and incorporate the time characteristics, ensuring that even if the data content is the same but the generation time is different, there will be differences in the fingerprint, providing a reliable basis for the subsequent accurate distinction and management of data at different time points.

[0067] Step 220: extract the real-time load parameters of the storage node, which include CPU usage. , memory usage , disk I / O rate , the real-time load parameters are normalized to obtain the normalized load parameters , introduce reinforcement learning model to learn the weight coefficients of various real-time load parameters in different industrial scenarios , the load characteristic value is obtained by weighted summation , and adopt the fusion coefficient 、 The enhanced basic fingerprint and load characteristic value Fusion, generating load-related fingerprints , the formula is:

[0068] ;

[0069] Where, is the number of real-time load parameters, is the load characteristic value , ;

[0070] Among them, "real-time load parameters" are key indicators that reflect the current operating status of the storage node, including CPU usage. It reflects the computing power usage of the node and the memory usage Reflects the usage of memory resources and disk I / O rate Indicates the speed at which the disk reads and writes data. Normalizing these parameters means converting them into standardized load parameters with a uniform magnitude (e.g., between 0 and 1). , in order to carry out subsequent weighted calculations. The "reinforcement learning model" automatically adjusts the weight coefficient by learning the impact of various load parameters on node performance under different industrial scenarios (such as production peak and trough periods). , so that the load characteristic value obtained by weighted summation is Can accurately reflect the actual load status of the node.

[0071] The reinforcement learning model uses the Q-learning algorithm. The state space is the real-time load parameters of the storage node (CPU usage, memory utilization, and disk I / O rate). The action space is the combination of weight coefficients for each load parameter (ranging from 0 to 1, with a step size of 0.1). The reward function is the node load balance (calculated as: 1 - standard deviation of each node's load / average load). The training sample is node load data from the past 30 days (specified by peak and off-peak production periods), with a sample size of at least 1,000 per scenario. The learning rate is set to 0.01, and the discount factor is set to 0.9.

[0072] In addition, it should be noted that through the fusion coefficient 、 Enhanced basic fingerprint and load characteristic value Fusion generates load-related fingerprints , realizing the organic combination of data sharding characteristics and node load status, so that the fingerprint can be used for redundancy detection and provide load correlation basis for node selection, thereby considering node load balancing in the data storage stage and avoiding excessive data concentration.

[0073] Step 230: Combine the LSTM model to predict the load trend of the storage node in the future preset time period and set the predictive load threshold interval , adjust the fusion coefficient according to the prediction results When the load is expected to exceed Increase , when the forecast load will be lower than Time decrease , to dynamically adjust the load association fingerprint ;

[0074] The preset time period is determined based on the update frequency of industrial scenario data and the node load fluctuation period, specifically 5 to 10 minutes. For example, for high-frequency production line sensor data (update frequency 1 time / second), the preset time period is set to 5 minutes; for low-frequency log data (update frequency 1 time / minute), the preset time period is set to 10 minutes. The LSTM model input is the time series of the load parameters of the storage node over the past hour (sampling interval 10 seconds), and the output is the predicted load mean value for the preset time period in the future. The model hidden layer dimension is set to 64, the number of training iterations is 100, and the loss function uses the mean squared error (MSE).

[0075] The LSTM model is a long-short-term memory network suitable for processing and predicting time series data. It is used here to predict the load trend of storage nodes in a preset time period (such as 5 to 10 minutes) in the future, and can perceive changes in node load in advance. It is a reasonable load range set according to the performance of the node and the stability of the system. Adjust the fusion coefficient according to the prediction results. , when the load is predicted to exceed Increase , so that the load correlation fingerprint highlights the high load state of the node and reduces it when the load is predicted to be lower than , making the data features account for a higher proportion in the fingerprint.

[0076] In addition, it should be noted that by dynamically adjusting the load-related fingerprint, it can adapt to future changes in node load, providing a more forward-looking basis for subsequent node selection, thereby achieving "predictive" load balancing and avoiding the performance loss caused by large-scale data migration due to excessive node load after data storage in the traditional way.

[0077] Step 300: Match the load-associated fingerprint with a global fingerprint library, determine whether the data shard is redundant data based on the matching result, and select a target storage node based on the load-associated fingerprint.

[0078] Specifically, step 300 includes:

[0079] Step 310: Compare the load-associated fingerprint with the fingerprints in the global fingerprint library. If the same fingerprint exists, the corresponding data slice is determined to be redundant data; if the same fingerprint does not exist, the corresponding data slice is determined to be non-redundant data.

[0080] The "global fingerprint library" is a collection of load-related fingerprints and corresponding storage information of all stored data slices in the storage system, which is the core basis for redundancy detection. Compare the data with the fingerprints in the global fingerprint database. This step checks whether the same fingerprint exists to determine whether the current data shard is already stored in the system. If the same fingerprint exists, it is considered redundant data, which can avoid repeated storage and save storage space. If the same fingerprint does not exist, it is considered non-redundant data and needs to be stored. The purpose of this step is to accurately identify redundant data, improve storage resource utilization, and ensure data uniqueness and integrity.

[0081] Step 320: For non-redundant data shards, based on the load correlation degree reflected by the load correlation fingerprint, select a storage node with a lower load and a higher correlation degree with the load correlation fingerprint as the target storage node. The correlation degree is the degree of match between the storage node load state and the load characteristic portion of the load correlation fingerprint. The matching degree is measured by the absolute value of the difference between the storage node load parameter and the load characteristic value. The smaller the absolute value of the difference, the higher the correlation degree.

[0082] The “load correlation degree” refers to the matching between the load characteristics reflected by the load correlation fingerprint of the data shard and the actual load status of each storage node. The absolute value of the difference is used to measure the correlation; the smaller the absolute value of the difference, the higher the correlation. This criterion makes node selection more quantitative and accurate. For non-redundant data shards, nodes with low load and high correlation are selected as target storage nodes. This not only avoids the performance degradation caused by storing data in already heavily loaded nodes, but also aligns data storage with node load characteristics, improving data read and write efficiency, thereby achieving balanced data distribution and optimizing system performance.

[0083] Optionally, a reasonable threshold for the absolute value of the correlation difference is 0.1-0.3 (within the range of standardized load parameters). When the absolute value of the difference is ≤0.1, the correlation is determined to be high; when it is 0.1 < the absolute value of the difference ≤0.3, the correlation is determined to be medium; when the absolute value of the difference is >0.3, the correlation is determined to be low. Node selection prioritizes nodes with high correlation and a load ratio <50%. If there is no such node, nodes with medium correlation and a load ratio <30% are selected.

[0084] Step 400: Store the non-redundant data slices in the target storage node, and update the global fingerprint library and the local fingerprint library of the target storage node to reflect the latest storage status of the data.

[0085] Specifically, step 400 includes:

[0086] Step 410: storing the non-redundant data shards in the target storage node;

[0087] Storing non-redundant data shards on target storage nodes is the core data storage operation. The target storage nodes are the nodes selected in step 320 above, which have low loads and high relevance to the data shards. This operation aims to securely and efficiently store new, non-redundant data on appropriate nodes, ensuring data accessibility and security. Storing data on appropriate nodes also lays the foundation for subsequent data management and load balancing, improving the overall efficiency of the storage system.

[0088] Step 420: Update the load-associated fingerprint of the non-redundant data shard and its corresponding target storage node information to the global fingerprint database. Meanwhile, the target storage node records the load-associated fingerprint to the local fingerprint database. The update of the global fingerprint database must be cross-checked by verification nodes of at least three different physical racks.

[0089] Among them, "global fingerprint library update" is to associate the load fingerprint of non-redundant data shards The target storage node and its corresponding target storage node information are added to the global fingerprint library, so that the global fingerprint library can reflect the latest storage status of the data in the system in real time, providing an accurate reference for subsequent fingerprint matching and data management. Record to local fingerprint database", which makes it easier for nodes to quickly manage and query locally stored data.

[0090] In addition, it should be noted that "global fingerprint library updates must be cross-checked by verification nodes in at least three different physical racks." This mechanism can ensure that the information updated to the global fingerprint library is true, accurate, and complete, avoiding distortion of fingerprint library information due to errors or malicious tampering of a single node, thereby improving the reliability and security of the global fingerprint library and ensuring data consistency of the entire storage system.

[0091] Among them, the verification node must meet the following requirements:

[0092] (1) Distributed across different physical racks (at least across 3 racks);

[0093] (2) No fault record in the past 7 days;

[0094] (3) CPU idle rate > 30%.

[0095] The cross-verification process is as follows: the target node sends a fingerprint update request (including an encrypted signature) → the verification node independently verifies the fingerprint validity (comparing the data shard hash value) → after at least two verification nodes return 'pass', the global fingerprint library is updated and a verification log is generated and stored in the blockchain (hash value chain).

[0096] Step 500: regularly analyze the load status of each storage node and the global fingerprint library, migrate data shards according to the analysis results, and synchronously update the relevant fingerprint library to achieve dynamic balancing of the system load.

[0097] Specifically, step 500 includes:

[0098] Step 510: Periodically collect the load status information of each storage node and analyze the data shard storage distribution in combination with the global fingerprint library. The periodicity is a preset time interval, which can be adjusted according to the system load characteristics.

[0099] The "preset time interval" is set based on the system's load characteristics (such as load fluctuation frequency and data update speed). For example, it can be set to 1 hour, 2 hours, etc., and can be adjusted based on actual conditions. "Collecting load status information for each storage node at the preset time interval" allows for timely monitoring of node load changes. By combining this with the global fingerprint database to analyze the data shard storage distribution, it provides a clear understanding of the balanced distribution of data across nodes, which nodes are overloaded, and which are underloaded, providing a basis for subsequent data migration. This step aims to provide accurate status information for dynamic system optimization and adjustment, ensuring that the system can promptly detect and resolve load imbalances.

[0100] Step 520: For a storage node with excessive load, redundant data shards or data shards with low load correlation with the node are migrated to a storage node with lower load based on the load correlation fingerprint of the data shards stored therein;

[0101] Among them, "overloaded storage nodes" refer to nodes whose load status exceeds the reasonable threshold set by the system. If not processed, it may cause node performance degradation or even failure. "According to the load associated fingerprint of the data shards stored , migrate redundant data shards or data shards with low correlation with the node load to storage nodes with lower load. Redundant data shards refer to shards with multiple copies in the system. Migrating them will not affect data availability. Migrating data shards with low correlation with the node load can reduce the load on the node and store the data shards on more suitable nodes. The purpose of this step is to relieve the pressure on overloaded nodes and achieve load balancing among nodes.

[0102] Step 530: Synchronously update the global fingerprint database and the local fingerprint databases of the storage nodes involved in the migration;

[0103] The "synchronous update of the global fingerprint database and the local fingerprint databases of the storage nodes involved in the migration" is necessary because the migration of data shards changes the data storage location. The fingerprint database must promptly reflect this change to ensure that the fingerprint database information is consistent with the actual data storage status. Failure to synchronize the update may cause errors in subsequent fingerprint matching, affecting data query and management. This step is intended to ensure the accuracy and consistency of fingerprint database information and ensure the normal and efficient operation of the entire storage system.

[0104] Step 600: When an extreme working condition is detected, a multi-level fingerprint degradation mechanism is used to process data fragmentation. After the system returns to normal, the local fingerprints generated during the degradation process are synchronized to the global fingerprint library to ensure data processing continuity and integrity.

[0105] "Extreme operating conditions" refer to sudden, unforeseen circumstances at industrial sites that could impact the normal operation of the storage system. These include sudden power outages that can disrupt node power supply, and network congestion that can affect data transmission efficiency and accuracy. "When an extreme operating condition is detected, a multi-level fingerprint degradation mechanism is triggered," enabling the system to take appropriate countermeasures to avoid data loss or system crashes caused by these conditions. This step aims to maintain the system's basic functionality and ensure data processing continuity in extreme situations.

[0106] Optionally, the triggering conditions for these extreme operating conditions include: network congestion (inter-node data transmission delay exceeding 100ms and lasting for more than 5 seconds), sudden power outage (node ​​power interruption exceeding 1 second), and computing resource overload (CPU utilization exceeding 95% for 10 seconds). Level 1 degradation corresponds to early stages of network congestion (delay of 50-100ms), level 2 degradation corresponds to severe network congestion (delay > 100ms) or CPU overload, and level 3 degradation corresponds to sudden power outage.

[0107] The multi-level fingerprint degradation mechanism includes:

[0108] When downgrading to the first level, the basic fingerprint layer processing is retained, and the load association layer and dynamic adjustment layer processing are suspended;

[0109] A 128-bit simplified hash algorithm is used to generate fingerprints during the second-level downgrade;

[0110] Enable local cache fingerprint during level 3 downgrade.

[0111] It should be noted that "retaining basic fingerprint layer processing during level one downgrade, and suspending load association layer and dynamic adjustment layer processing" is a measure taken when extreme working conditions have little impact on the system. It prioritizes ensuring that the data can generate basic fingerprints and store them, and temporarily ignores complex processing such as load association to save system resources. "Using a 128-bit simplified hash algorithm to generate fingerprints during level two downgrade" The 128-bit simplified hash algorithm has a small amount of computation and can quickly generate fingerprints when system computing resources are insufficient, ensuring that data can be identified and stored. "Enabling local cached fingerprints during level three downgrade" uses locally cached fingerprints for data processing to avoid data processing interruptions when the system cannot communicate with the global fingerprint library or has severely insufficient computing power.

[0112] In addition, the multi-level fingerprint degradation mechanism adopts different processing methods according to the severity of extreme working conditions. Its purpose is to maximize the protection of data processing and storage when system resources are limited, ensure that industrial data is not lost, and maintain the basic continuity of data processing.

[0113] The requirement to "synchronize the local fingerprints generated during the degradation process to the global fingerprint library after the system returns to normal" is because, under extreme operating conditions, the local fingerprints generated during the degradation process are only stored on the local node and not updated to the global fingerprint library. Synchronizing after system recovery ensures that the global fingerprint library contains fingerprint information for all data, ensuring its integrity and consistency. This step aims to restore the system's data management status after the extreme operating condition ends, ensuring the accuracy and effectiveness of subsequent data processing.

[0114] In this embodiment, load-associated fingerprints are generated by fusing data features with storage node load status, and the fingerprints are dynamically adjusted to match node load trends, thereby achieving intelligent data distribution and solving the problem of uneven data distribution. Redundant data is accurately identified through fingerprint matching, and storage nodes are selected in combination with node load status to improve storage efficiency and system performance. Regular optimization and adjustment and extreme working condition processing mechanisms are used to ensure stable system operation, thereby achieving the effect of efficiently eliminating redundancy and balancing loads in distributed industrial storage.

[0115] Based on the above embodiments, Figure 2 As shown, the present invention also provides a data redundancy elimination system for distributed industrial storage devices, which is used to support the data redundancy elimination method for distributed industrial storage devices in the above embodiment. The data redundancy elimination system for distributed industrial storage devices includes:

[0116] The data processing module 11 is used to receive external data blocks and split them into several data shards, and then apply a dynamic hierarchical fingerprint algorithm based on the data shards to generate load-related fingerprints. The load-related fingerprints integrate the characteristics of the data shards and the load status of the storage nodes.

[0117] The fingerprint management module 12 is used to maintain the global fingerprint library and the local fingerprint library of each storage node, support matching the load-associated fingerprint with the global fingerprint library, and update the global fingerprint library and the local fingerprint library of the target storage node after the data is stored;

[0118] The node selection and scheduling module 13 is used to determine whether the data shard is redundant data based on the matching result of the load-related fingerprint and the global fingerprint library, select the target storage node based on the load-related fingerprint, and regularly analyze the load status of each storage node and the global fingerprint library, and migrate the data shard according to the analysis results;

[0119] The load monitoring module 14 is used to monitor the load parameters of each storage node in real time and provide them to the data processing module to generate load-related fingerprints. It also provides a load status basis for node selection and data migration in the node selection and scheduling module.

[0120] The storage node 15 is used to store non-redundant data shards and maintain the local fingerprint library, respond to data read and write and migration requests, cooperate to implement the multi-level fingerprint degradation mechanism under extreme working conditions, and synchronize relevant fingerprint information after the system recovers.

[0121] In an optional embodiment, the data processing module 11 is also used to receive external data blocks in industrial scenarios; and divide the external data blocks into several data slices according to a preset fixed size, and the preset fixed size is set according to the industrial data characteristics and system processing requirements.

[0122] In an optional embodiment, the data processing module 11 is further configured to process the data fragments using the SHA-256 hash algorithm to generate a basic fingerprint. , introduce the timestamp hash fusion mechanism to generate timestamps for data Converted into a time signature code through an irreversible encryption algorithm , and the basic fingerprint Time signature Perform XOR operation to obtain enhanced basic fingerprint , the formula is: , where Represents an XOR operation; extracts the real-time load parameters of the storage node, which include CPU usage , memory usage , disk I / O rate , the real-time load parameters are standardized to obtain standardized load parameters , introduce reinforcement learning model to learn the weight coefficients of various real-time load parameters in different industrial scenarios , the load characteristic value is obtained by weighted summation , and adopt the fusion coefficient 、 The enhanced basic fingerprint and load characteristic value Fusion, generating load association fingerprints Then, the LSTM model is used to predict the load trend of the storage node within a preset time period in the future and set the predictive load threshold interval. , adjust the fusion coefficient according to the prediction results When the load is expected to exceed Increase , when the forecast load will be lower than Time decrease , to dynamically adjust the load association fingerprint .

[0123] In an optional embodiment, the fingerprint management module 12 is further configured to associate the load with a fingerprint. Compare the fingerprints with those in the global fingerprint library. When the global fingerprint library is updated, ensure cross-verification by verification nodes of at least three different physical racks. Support nodes to extract fingerprint feature vectors from the local fingerprint library and share features across nodes through the federated learning framework without leaking the original data.

[0124] In an optional embodiment, the node selection and scheduling module 13 is further configured to select and schedule non-redundant data slices according to the load association fingerprint. Reflects the load correlation degree, selects the load with lower load and the fingerprint associated with the load The storage node with the highest correlation is used as the target storage node. The correlation is calculated by comparing the storage node load parameter with the load characteristic value. The load status information of each storage node is analyzed at a preset time interval, and the preset time interval can be adjusted according to the system load characteristics. The data shard storage distribution is analyzed in combination with the global fingerprint library. For storage nodes with excessive load, redundant data shards or data shards with low correlation with the node load are migrated to storage nodes with lower load, and the global fingerprint library and the local fingerprint library of the storage nodes involved in the migration are updated synchronously.

[0125] In an optional embodiment, the load monitoring module 14 is further used to monitor the CPU usage of each storage node in real time. , memory usage , disk I / O rate The monitored load parameters are sent to the data processing module and the node selection and scheduling module in real time, providing real-time and accurate load status basis for the generation of load-related fingerprints, node selection, and data migration.

[0126] In an optional embodiment, the storage node 15 is also used to cooperate in executing a multi-level fingerprint degradation mechanism when extreme working conditions such as sudden power outages and network congestion occur at the industrial site. During the first level of degradation, only basic fingerprint layer processing is performed, a 128-bit simplified hash algorithm is used to generate fingerprints during the second level of degradation, and local cache fingerprints are enabled during the third level of degradation. After the system returns to normal, the local fingerprints generated during the degradation process are synchronized to the global fingerprint library to ensure the continuity and integrity of data processing.

[0127] In this embodiment, load-associated fingerprints are generated by fusing data features with storage node load status, and the fingerprints are dynamically adjusted to match node load trends, thereby achieving intelligent data distribution and solving the problem of uneven data distribution. Redundant data is accurately identified through fingerprint matching, and storage nodes are selected in combination with node load status to improve storage efficiency and system performance. Regular optimization and adjustment and extreme working condition processing mechanisms are used to ensure stable system operation, thereby achieving the effect of efficiently eliminating redundancy and balancing loads in distributed industrial storage.

[0128] Furthermore, the data redundancy elimination system of the distributed industrial storage device can run the data redundancy elimination method of the distributed industrial storage device. The specific implementation can be found in the method embodiment, which will not be described in detail here.

[0129] Based on the above embodiments, Figure 3 As shown, the present invention also provides an electronic device, the electronic device comprising:

[0130] At least one processor 22, at least one memory 21, a communication interface 23 and a communication bus 24, wherein the processor 22 is communicatively connected to the memory 21;

[0131] In this embodiment, the memory 21 can be implemented in any appropriate manner, for example, the memory 21 can be a read-only memory, a mechanical hard disk, a solid-state drive, or a USB flash drive, etc. The memory 21 is used to store at least one executable instruction executed by the processor;

[0132] In this embodiment, the processor 22 can be implemented in any appropriate manner. For example, the processor 22 can take the form of a microprocessor or a processor and a computer-readable medium storing computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, etc.; the processor is used to execute the executable instructions to implement the data redundancy elimination method of the distributed industrial storage device as described above.

[0133] Based on the above embodiments, the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the method for eliminating data redundancy in a distributed industrial storage device as described above is implemented.

[0134] Those skilled in the art will appreciate that the modules and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0135] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, equipment and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0136] In the several embodiments provided in this application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or units can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or equipment, which can be electrical, mechanical or other forms.

[0137] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0138] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0139] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program instructions, such as a USB flash drive, a mobile hard disk, a read-only storage server, a random access storage server, a magnetic disk, or an optical disk.

[0140] In addition, it should be noted that the combination of the various technical features in this case is not limited to the combination described in the claims of this case or the combination described in the specific embodiments. All technical features recorded in this case can be freely combined or combined in any way unless there is a contradiction between them.

[0141] It should be noted that the above examples are merely specific embodiments of the present invention. Obviously, the present invention is not limited to the above examples, and many similar variations are possible. All variations directly derived from or associating with the present invention by those skilled in the art are intended to fall within the scope of protection of the present invention.

[0142] The above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for eliminating data redundancy in a distributed industrial storage device, characterized in that: The following steps are involved: Receive external data blocks and perform segmentation processing to obtain several data fragments, providing standardized processing units for subsequent fingerprint generation; Based on the data sharding, a dynamic hierarchical fingerprint algorithm is applied to generate a load-related fingerprint. The load-related fingerprint integrates the data sharding characteristics and the storage node load status to provide a basis for data redundancy detection and node selection. Matching the load-associated fingerprint with a global fingerprint library, determining whether the data shard is redundant data based on the matching result, and selecting a target storage node based on the load-associated fingerprint; Storing the non-redundant data slices in the target storage node, and updating the global fingerprint library and the local fingerprint library of the target storage node to reflect the latest storage status of the data; Regularly analyze the load status of each storage node and the global fingerprint library, migrate data shards based on the analysis results, and synchronously update the relevant fingerprint library to achieve dynamic balancing of the system load; The method of applying a dynamic hierarchical fingerprint algorithm based on the data sharding to generate a load-associated fingerprint, wherein the load-associated fingerprint integrates the data sharding characteristics with the storage node load status to provide a basis for data redundancy detection and node selection, includes: Use SHA-256 hash algorithm to process the data fragments to generate basic fingerprints , introduce the timestamp hash fusion mechanism to generate timestamps from data Converted into a time signature code through an irreversible encryption algorithm , and the basic fingerprint Time signature Perform XOR operation to obtain enhanced basic fingerprint , the formula is: , where Represents exclusive OR operation; Extract the real-time load parameters of the storage node, including CPU usage , memory usage , disk I / O rate , the real-time load parameters are normalized to obtain the normalized load parameters , introduce reinforcement learning model to learn the weight coefficients of various real-time load parameters in different industrial scenarios , the load characteristic value is obtained by weighted summation , and adopt the fusion coefficient 、 The enhanced basic fingerprint and load characteristic value Fusion, generating load-related fingerprints , the formula is: ； Where, is the number of real-time load parameters, is the load characteristic value , ; Combined with the LSTM model to predict the load trend of the storage node within a preset time period in the future, and set the predictive load threshold interval , adjust the fusion coefficient according to the prediction results When the load is expected to exceed Increase , when the forecast load will be lower than Time decrease , to dynamically adjust the load association fingerprint .

2. The method for eliminating data redundancy in a distributed industrial storage device according to claim 1, characterized in that: The receiving of external data blocks and segmentation processing to obtain a number of data fragments provides a standardized processing unit for subsequent fingerprint generation, including: Receive external data blocks in industrial scenarios; The external data block is divided into a plurality of data slices according to a preset fixed size, and the preset fixed size is set according to industrial data characteristics and system processing requirements.

3. The method for eliminating data redundancy in distributed industrial storage devices according to claim 1, characterized in that: The matching of the load-associated fingerprint with a global fingerprint library, determining whether the data shard is redundant data according to the matching result, and selecting a target storage node based on the load-associated fingerprint includes: Comparing the load-associated fingerprint with the fingerprints in the global fingerprint library, if the same fingerprint exists, determining that the corresponding data slice is redundant data; if the same fingerprint does not exist, determining that the corresponding data slice is non-redundant data; For non-redundant data shards, based on the load correlation degree reflected by the load correlation fingerprint, a storage node with a lower load and a higher correlation degree with the load correlation fingerprint is selected as the target storage node. The correlation degree is the degree of matching between the storage node load state and the load characteristic part in the load correlation fingerprint. The matching degree is measured by the absolute value of the difference between the storage node load parameter and the load characteristic value. The smaller the absolute value of the difference, the higher the correlation degree.

4. The method for eliminating data redundancy in distributed industrial storage devices according to claim 1, characterized in that: The storing of the non-redundant data slices to the target storage node and updating the global fingerprint library and the local fingerprint library of the target storage node to reflect the latest storage status of the data includes: Storing the non-redundant data slices in the target storage node; The load-associated fingerprint of the non-redundant data shard and its corresponding target storage node information are updated to the global fingerprint library. At the same time, the target storage node records the load-associated fingerprint to the local fingerprint library. The update of the global fingerprint library needs to be cross-checked by verification nodes of at least 3 different physical racks.

5. The method for eliminating data redundancy in a distributed industrial storage device according to claim 1, wherein: The periodic analysis of the load status of each storage node and the global fingerprint library, migrating data shards according to the analysis results and synchronously updating the relevant fingerprint library to achieve dynamic system load balancing includes: Periodically collecting load status information of each storage node and analyzing data shard storage distribution in combination with the global fingerprint library, wherein the periodicity is a preset time interval, and the preset time interval can be adjusted according to system load characteristics; For storage nodes with excessive load, redundant data shards or data shards with low load correlation with the node are migrated to storage nodes with lower load based on the load correlation fingerprint of the data shards stored on them. The global fingerprint database and the local fingerprint databases of the storage nodes involved in the migration are updated synchronously.

6. The method for eliminating data redundancy in a distributed industrial storage device according to claim 1, characterized in that: Also includes: When an extreme working condition is detected, a multi-level fingerprint degradation mechanism is used to process data fragmentation. After the system returns to normal, the local fingerprints generated during the degradation process are synchronized to the global fingerprint library; The multi-level fingerprint degradation mechanism includes: When downgrading to the first level, the basic fingerprint layer processing is retained, and the load association layer and dynamic adjustment layer processing are suspended; A 128-bit simplified hash algorithm is used to generate fingerprints during the second-level downgrade; Enable local cache fingerprint during level 3 downgrade.

7. A data redundancy elimination system for distributed industrial storage devices, characterized in that: include: A data processing module is configured to receive an external data block and perform segmentation processing to obtain a plurality of data shards, and then generate a load-related fingerprint based on the data shards using a dynamic hierarchical fingerprint algorithm. The load-related fingerprint integrates the characteristics of the data shards and the load status of the storage node; The fingerprint management module is used to maintain the global fingerprint library and the local fingerprint library of each storage node, support matching the load-associated fingerprint with the global fingerprint library, and update the global fingerprint library and the local fingerprint library of the target storage node after data storage; A node selection and scheduling module is used to determine whether a data shard is redundant data based on the matching result of the load-related fingerprint and the global fingerprint library, select a target storage node based on the load-related fingerprint, and regularly analyze the load status of each storage node and the global fingerprint library, and migrate data shards based on the analysis results; The load monitoring module is used to monitor the load parameters of each storage node in real time and provide them to the data processing module to generate load-related fingerprints. It also provides a load status basis for node selection and data migration in the node selection and scheduling module. Storage nodes are used to store non-redundant data shards and maintain local fingerprint libraries, responding to data read, write, and migration requests; The data processing module is also used to: Use SHA-256 hash algorithm to process the data fragments to generate basic fingerprints , introduce the timestamp hash fusion mechanism to generate timestamps from data Converted into a time signature code through an irreversible encryption algorithm , and the basic fingerprint Time signature Perform XOR operation to obtain enhanced basic fingerprint , the formula is: , where Represents exclusive OR operation; Extract the real-time load parameters of the storage node, including CPU usage , memory usage , disk I / O rate , the real-time load parameters are normalized to obtain the normalized load parameters , introduce reinforcement learning model to learn the weight coefficients of various real-time load parameters in different industrial scenarios , the load characteristic value is obtained by weighted summation , and adopt the fusion coefficient 、 The enhanced basic fingerprint and load characteristic value Fusion, generating load-related fingerprints , the formula is: ； Where, is the number of real-time load parameters, is the load characteristic value , ; Combined with the LSTM model to predict the load trend of the storage node within a preset time period in the future, and set the predictive load threshold interval , adjust the fusion coefficient according to the prediction results When the load is expected to exceed Increase , when the forecast load will be lower than Time decrease , to dynamically adjust the load association fingerprint .

8. An electronic device, characterized in that: The electronic device comprises: a processor and a memory, wherein the memory is communicatively connected to the processor; The memory is used to store at least one executable instruction executed by the processor, and the processor is used to execute the executable instruction to implement the data redundancy elimination method of the distributed industrial storage device according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for eliminating data redundancy in a distributed industrial storage device according to any one of claims 1 to 6 is implemented.

Images