Industrial cold and hot data storage control method and system

Abstract

The application relates to the technical field of industrial data storage control, in particular to an industrial cold and hot data storage control method and system. The method comprises the following steps: obtaining the access features of to-be-allocated industrial data, and predicting the access heat of the industrial data according to the access features; generating a storage mode adjustment strategy according to the predicted access heat and cost-benefit evaluation; allocating compression algorithms and storage media for industrial data with different heat according to the storage mode adjustment strategy; realizing the dynamic migration among industrial data with different heat through an elastic layered storage architecture according to the allocated compression algorithms and storage media; the access features of to-be-allocated industrial data are obtained, the access heat is predicted, the storage mode adjustment strategy is generated in combination with cost-benefit evaluation, then appropriate compression algorithms and storage media are allocated for industrial data with different heat, and the dynamic migration of data is realized through the elastic layered storage architecture.

Description

Technical Field

[0001] This invention relates to the field of industrial data storage and control technology, and in particular to an industrial cold and hot data storage and control method and system. Background Technology

[0002] With the rapid development of information technology, big data has become a core asset in the industrial sector. Industrial big data has seen a surge in scale and diversified sources due to the widespread adoption of technologies such as the Internet of Things (IoT) and 5G. Industrial data is typically generated by tens of thousands of sensors from hundreds of devices, characterized by dense sampling cycles and strong correlations. Existing storage systems often employ static strategies, selecting a subset of data based on scenario experience and applying fixed compression algorithms (such as LZ4 and Zstandard) to save storage space. However, this approach has significant shortcomings, failing to fully consider data compressibility and access characteristics, resulting in an inability to effectively balance storage efficiency and system performance.

[0003] Existing technologies often overlook local characteristics of data distribution (such as data block similarity) and dynamic access patterns (such as access frequency and latency sensitivity) when manually selecting compression algorithms. For example, in international research, the decision tree-based adaptive compression method proposed by Stonebraker et al. did not address the locality of data distribution; Jin et al. used an LSTM model to predict data block features and compress similar blocks, but did not integrate global access dependencies; Devarajan's Ares framework uses a decision tree allocation algorithm, but requires predefined file access patterns and is difficult to adapt to dynamic changes. In domestic research, Wei Ling et al. improved the compression ratio through data type matching encoding, but the allocation accuracy was insufficient; Wang Haiyan et al. optimized HBase storage based on a Bayesian model, but did not adapt to changes in access frequency in real time; Guo Jing constructed compression selection as a multi-objective optimization problem, but did not incorporate access frequency prediction. These methods all fail to establish a dynamic mapping between data value (such as access frequency) and storage resources, resulting in the failure to distinguish between hot and cold data for storage, leading to high storage costs and performance degradation. In industrial scenarios, data access frequency changes over time, but static strategies cannot dynamically adjust storage methods (such as compression algorithms and storage media), resulting in low storage resource utilization. Furthermore, the lack of cost-benefit assessment during data migration means that migration overhead (such as time and performance loss) is not quantified, impacting business continuity and causing system performance bottlenecks.

[0004] To address the aforementioned issues, existing technologies urgently need improvement. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing an industrial cold and hot data storage control method and system.

[0006] In a first aspect, the present invention provides an industrial cold and hot data storage and control method, the method comprising the following steps: Obtain the access characteristics of the industrial data to be allocated, and predict the access popularity of the industrial data based on the access characteristics; Based on the predicted access volume and cost-effectiveness assessment, adjust the storage strategy accordingly; Based on the storage method, an adjustment strategy is implemented to allocate compression algorithms and storage media for industrial data with different levels of popularity. Based on the allocated compression algorithm and storage medium, a flexible tiered storage architecture enables dynamic migration between industrial data of varying importance.

[0007] Secondly, an industrial cold and hot data storage control system is provided, the system comprising: The access feature acquisition module is used to acquire the access features of the industrial data to be allocated, and predict the access popularity of the industrial data based on the access features. The strategy generation module is used to generate storage adjustment strategies based on predicted access volume and cost-effectiveness assessments. The allocation module is used to adjust the strategy according to the storage method and allocate compression algorithms and storage media to industrial data with different levels of popularity. The dynamic migration module is used to achieve dynamic migration between industrial data of different popularity based on the allocated compression algorithm and storage medium through a flexible tiered storage architecture.

[0008] Compared with the prior art, the present invention has the following beneficial effects: By acquiring the access characteristics of industrial data to be allocated and predicting its access frequency, and combining this with cost-benefit assessment to generate storage adjustment strategies, appropriate compression algorithms and storage media are assigned to industrial data with different access frequencies. Dynamic data migration is then achieved through a flexible tiered storage architecture. This method effectively solves the problem in existing industrial data storage systems that fail to fully consider data compressibility and access characteristics, leading to an ineffective trade-off between storage efficiency and system performance. By dynamically predicting data frequency and adjusting storage strategies accordingly, this application enables differentiated storage of hot and cold data, significantly reducing storage costs and improving system access performance. Furthermore, through the assessment of migration overhead and the dynamic migration mechanism, this application overcomes the shortcomings of existing technologies that lack cost-benefit assessments for data migration, impacting business continuity and causing system performance bottlenecks, thereby improving the utilization rate of storage resources and the overall operating efficiency of the system. Attached Figure Description

[0009] Figure 1 This is a flowchart of the method of the present invention.

[0010] Figure 2 This is a schematic diagram of the system structure of the present invention.

[0011] In the diagram: 201, Access Feature Acquisition Module; 202, Policy Generation Module; 203, Allocation Module; 204, Dynamic Migration Module. Detailed Implementation

[0012] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0013] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0014] The term "industrial data" as used in this application refers to all data generated during industrial production and operation, including but not limited to sensor data, equipment operation logs, production process parameters, and quality inspection data. This data typically features massive volume, high concurrency, and strong real-time requirements. "Access characteristics" refer to attributes describing the access patterns of industrial data, such as access frequency, access time, accessing users, and access type (read / write). "Access popularity" is a prediction of data activity based on access characteristics; high popularity data indicates frequent access, while low popularity data indicates infrequent access. "Compression algorithms" refer to technologies used to reduce data storage space, such as LZ4, Zstandard, and Gzip. "Storage media" refers to the physical devices used to store data, such as solid-state drives (SSDs), hard disk drives (HDDs), tape libraries, and cloud storage. "Elastic tiered storage architecture" refers to a storage system structure capable of dynamically migrating data between different storage media based on data popularity, access patterns, and storage strategies to optimize storage costs and access performance.

[0015] like Figure 1 The method for controlling industrial cold and hot data storage, as shown, includes the following steps: S101. Obtain the access characteristics of the industrial data to be allocated, and predict the access popularity of the industrial data based on the access characteristics. It should be noted that this step can be achieved by deploying a monitoring agent at the data acquisition or storage system level to collect access logs of industrial data in real time, including the timestamp of each access, access frequency, accessing user ID, data block ID, and other information. These raw access logs can be used as input for access characteristics. When predicting access popularity, statistical analysis methods based on historical access patterns can be used. For example, the average access frequency of a data block over a past period can be calculated and used as the access popularity of that data block. Alternatively, a simple threshold judgment can be used. For example, if a data block has been accessed more than 100 times in the past 24 hours, it is marked as "hot data," otherwise it is marked as "cold data."

[0016] S102. Based on the predicted access volume and cost-effectiveness assessment, generate a storage method adjustment strategy; It should be noted that the above step can be pre-set with a static cost-benefit evaluation model. This model calculates the overall cost-benefit score for each storage method combination based on parameters such as the unit storage cost and read / write performance of different storage media, as well as the compression ratio and decompression overhead of different compression algorithms. When a data block is predicted to be "hot data," the system will prioritize storage media with high read / write performance but potentially higher costs (such as SSDs) and compression algorithms with low decompression overhead (such as LZ4). When predicted to be "cold data," it tends to choose storage media with lower costs but average read / write performance (such as HDDs or magnetic tapes) and compression algorithms with high compression ratios (such as Gzip).

[0017] S103. Adjust the strategy according to the storage method and allocate compression algorithms and storage media for industrial data with different levels of popularity; It should be noted that in this step, for example, after receiving the storage method adjustment policy, the system will directly allocate the industrial data to the corresponding storage tier based on the compression algorithm and storage medium combination recommended for "hot data" and "cold data" in the policy. Specifically, if the policy indicates that "hot data" should be stored on an SSD and compressed using LZ4, then all industrial data predicted as "hot data" will be written to the SSD and compressed using LZ4. If the policy indicates that "cold data" should be stored on an HDD and compressed using Gzip, then all industrial data predicted as "cold data" will be written to the HDD and compressed using Gzip.

[0018] S104. Based on the allocated compression algorithm and storage medium, dynamic migration between industrial data of different popularity is achieved through a flexible tiered storage architecture.

[0019] It's important to note that this step, for example, when the access frequency of a data block changes from "hot" to "cold," triggers a data migration process based on preset migration rules. This process can be easily achieved by copying data from the source storage medium (such as an SSD) to the target storage medium (such as an HDD) and then deleting the source data after the copy is complete. During the migration, file system-level copy commands, such as `cp` or `rsync`, can be used to move data from one storage path to another. After the data migration is complete, the data access path needs to be updated to ensure that subsequent read and write requests correctly point to the new storage location.

[0020] This application's industrial hot and cold data storage control method achieves dynamic perception of data value by acquiring the access characteristics of industrial data to be allocated and predicting its access frequency. Based on this, combined with cost-benefit assessment, a refined storage method adjustment strategy is generated, allocating the most suitable compression algorithm and storage medium to industrial data with different access frequencies, thereby achieving an optimal balance between storage efficiency and access performance. Finally, through a flexible tiered storage architecture, dynamic migration between industrial data with different access frequencies is realized, ensuring optimized utilization of storage resources.

[0021] Compared with existing technologies, the core innovation of this application lies in its dynamic and adaptive storage control mechanism. Traditional storage systems often adopt static strategies, which cannot effectively cope with the dynamic changes in industrial data access patterns, resulting in low storage resource utilization and a lack of cost-effectiveness assessment during data migration, which can easily lead to performance bottlenecks.

[0022] This application, by introducing access characteristic prediction and cost-effectiveness assessment, can identify data popularity in real time and accurately, and dynamically adjust storage strategies accordingly. This mechanism enables the storage system to intelligently select the optimal storage method based on the actual value of the data and access needs, avoiding the waste of high-performance storage media for cold data and the access latency caused by storing hot data on low-performance media. Furthermore, dynamic data migration is achieved through a flexible tiered storage architecture, taking migration overhead into account to ensure business continuity and system performance. Therefore, this application significantly improves the efficiency, performance, and cost-effectiveness of industrial data storage, providing a more advanced and flexible solution for industrial big data management.

[0023] As one embodiment of the present invention, the steps of obtaining the access characteristics of the industrial data to be allocated and predicting the access popularity of the industrial data based on the access characteristics include: Real-time monitoring of access characteristics of industrial data streams containing industrial data to be assigned; It should be noted that the above step refers to the continuous collection and analysis of industrial data access behavior data within the storage system. These access characteristics may include, but are not limited to, access frequency, access timestamps, access patterns (e.g., sequential access, random access), data block size, read / write ratio, and the type of user or application accessing the data. Monitoring can be achieved by deploying probes or log analysis tools at the storage system layer, file system layer, or application layer.

[0024] Based on access characteristics, dynamically analyze the statistical distribution of access characteristics to determine whether the statistical distribution deviates from the preset stable range; It should be noted that the above step can be understood as the system periodically performing statistical analysis on the collected access characteristic data, such as calculating statistics like mean, variance, skewness, and kurtosis, or constructing a probability distribution model of access patterns. The preset stable range is a statistical characteristic interval of normal access patterns determined based on historical data or expert experience. When the current statistical distribution differs significantly from the preset stable range, it is judged as a deviation. For example, chi-square test, KL divergence, or threshold-based anomaly detection algorithms can be used to determine deviation.

[0025] When the statistical distribution deviates from the preset stable range, the key access features that cause the deviation are identified, and the judgment threshold or feature weight of the access popularity prediction is adjusted according to the degree of deviation of the key access features. It should be noted that in the above step, key access features refer to those features that have the greatest impact on the access popularity prediction results. For example, if the access frequency of a specific data block suddenly increases significantly, then "access frequency" is the key access feature. The degree of deviation can be quantified by indicators such as the magnitude of statistical changes and outlier scores. Based on the degree of deviation, the system can adaptively adjust the model parameters used to predict access popularity. For example, it can increase the weight of "access frequency" in the prediction model, or adjust the judgment threshold for classifying data into "hot," "warm," and "cold" to more accurately reflect the current actual access popularity.

[0026] Based on the adjusted judgment threshold or feature weights, predict the access popularity of industrial data.

[0027] It should be noted that the above step ensures that the prediction model can adapt to changes in industrial data access patterns, thereby providing more accurate and real-time heat prediction results.

[0028] This application's solution addresses the inaccuracy of traditional prediction methods when facing dynamically changing industrial data access patterns by introducing a real-time monitoring and dynamic analysis mechanism for industrial data stream access characteristics. Specifically, when the system detects that the statistical distribution of access characteristics deviates from a preset stable range, it indicates that the current access pattern may have changed. At this point, by identifying the key access characteristics causing the deviation and their degree of deviation, the system can adjust the judgment threshold or feature weights in the access popularity prediction model accordingly. For example, if a significant increase in the access frequency of a specific type of data is detected, the system will correspondingly increase the weight of access frequency in popularity prediction or adjust the threshold for popularity classification, enabling this type of data to be identified as "hot data" more promptly and accurately. This adaptive adjustment mechanism allows the prediction model to dynamically learn and adapt to new access patterns, thereby continuously outputting highly accurate access popularity prediction results.

[0029] Through the above technical solution, this application can significantly improve the accuracy and real-time performance of industrial data access popularity prediction. Compared with methods that use fixed parameters or rely on static historical data for prediction, the solution of this application can dynamically sense and respond to changes in industrial data access patterns, effectively avoiding improper allocation of storage resources due to inaccurate predictions. This ensures that "hot data" is promptly allocated to high-performance storage media, while "cold data" is effectively compressed and stored on lower-cost media, thereby optimizing the overall performance of the storage system, reducing operating costs, and improving data access efficiency.

[0030] As one embodiment of the present invention, the method further includes the following steps: Based on the actual processing effect of different popularity data in the storage process, evaluate the accuracy of access popularity prediction, and adjust the judgment threshold or feature weights based on the accuracy feedback.

[0031] Specifically, the processing effect of data with different access frequencies during actual storage can be understood as follows: after industrial data is allocated to the corresponding compression algorithms and storage media, the system continuously monitors indicators such as the actual access patterns, storage efficiency, migration frequency, and user perception of data access latency. For example, if data predicted as "cold data" is frequently accessed after actual storage, or data predicted as "hot data" is not accessed for a long time, this indicates that the prediction may be biased. Evaluating the accuracy of access frequency prediction means quantifying the accuracy of the prediction by comparing the predicted access frequency with the actual observed data access behavior. For example, indicators such as error rate, recall rate, precision rate, or F1 score can be calculated between the predicted and actual results. When there is a significant deviation between the actual processing effect and the predicted result, the prediction accuracy is considered low. Furthermore, adjusting the judgment threshold or feature weights based on accuracy feedback means that when the evaluation finds that the accuracy of the access frequency prediction does not meet the preset standard, the system will automatically or semi-automatically correct the judgment threshold or feature weights used for prediction according to the magnitude and direction of the deviation. For example, if the system tends to mispredict hot data as cold data, the threshold for judging hot data can be adjusted to make it easier to identify; if a certain access feature (such as access frequency) performs poorly in prediction, its weight can be reduced, or the weight of other more indicative features can be increased.

[0032] This application's solution effectively addresses the problem of prediction models failing due to changes in data patterns in dynamic industrial environments by introducing an evaluation and feedback adjustment mechanism for the accuracy of access popularity prediction. Specifically, by continuously monitoring the processing effects of data with different access volumes during actual storage, the system can obtain real feedback on the performance of the prediction model. When a deviation is found between the prediction result and the actual situation—for example, predicting hot data but having low actual access volume, or predicting cold data but having high actual access volume—this deviation is identified and used to evaluate the accuracy of the prediction. Based on this evaluation result, the system can adjust the judgment threshold or feature weights for access popularity prediction in a targeted manner. This adaptive adjustment process enables the prediction model to continuously learn and adapt to new data patterns, thereby improving its robustness and accuracy in complex and ever-changing industrial environments.

[0033] As one embodiment of the present invention, the step of generating a storage method adjustment strategy based on predicted access popularity and cost-effectiveness assessment includes: Real-time monitoring of storage system resource status; It should be noted that this step refers to continuously acquiring the current usage and availability status of various key resources in the storage system. For example, it can monitor information such as available space, read / write load, and network bandwidth for each storage medium. The purpose is to provide real-time system runtime context for subsequent policy generation, ensuring that policy formulation is based on the latest system conditions.

[0034] Based on the predicted access volume, cost-benefit assessment results, and the resource status of the storage system, the priorities of storage efficiency, access performance, and migration overhead are determined. It's important to note that this step can be understood as dynamically adjusting the emphasis on the three key metrics—storage efficiency, access performance, and migration overhead—during the storage strategy optimization process based on current business needs, data characteristics, and system load. For example, for high-priority, real-time-critical industrial data, access performance will be given significantly higher priority; while for archived data, storage efficiency may be given even higher priority. The goal is to ensure that the generated strategy better meets the specific needs of different scenarios.

[0035] Based on the priority of migration overhead, assess the impact of storage method adjustment actions on storage efficiency, access performance and migration overhead. Adjustment actions include combinations of different compression algorithms and storage media, as well as data migration operations. It should be noted that in practical applications, evaluating the impact of storage method adjustments on storage efficiency, access performance, and migration overhead based on migration cost priority refers to a predictive analysis of potential storage method adjustment schemes. Adjustments specifically include applying different compression algorithms to data, combining data storage across different types of storage media, and data migration between different storage media. The evaluation process can be based on pre-defined models, historical data analysis, or simulations to quantify the specific impact of each adjustment on storage efficiency improvements, access performance changes, and required migration overhead (such as time, computing resources, and network bandwidth). Its purpose is to provide a quantitative basis for strategy selection.

[0036] Based on the evaluation results, a storage method adjustment strategy is selected by balancing storage efficiency, access performance, and migration overhead, and a storage method adjustment strategy is generated.

[0037] It's important to note that this step refers to selecting a strategy that achieves the optimal trade-off between multiple objectives after comprehensively considering various evaluation metrics. This is not simply about maximizing a single metric, but rather finding an overall optimal solution while satisfying all constraints. For example, it might involve maximizing storage efficiency and minimizing migration overhead while ensuring access performance doesn't fall below a certain threshold. The goal is to ensure that the generated strategy meets performance requirements while also balancing resource utilization and cost control.

[0038] This application's solution introduces real-time monitoring of storage system resource status, enabling the strategy generation process to fully perceive the current system's operating environment and avoiding the limitations of decision-making based on static or outdated information. Because it can dynamically determine the priority of storage efficiency, access performance, and migration overhead, the system can flexibly adjust optimization objectives according to constantly changing business needs and system load, ensuring that the performance of critical services is prioritized. Simultaneously, by pre-assessing the impact of various storage method adjustments, the system can quantitatively analyze the potential effects and costs of different strategies, thus gaining a clear understanding of risks and benefits before making decisions. Finally, by combining the balance of various indicators to select a strategy, this solution can find a comprehensive optimal solution in the face of multiple conflicting objectives, effectively addressing the problems of traditional methods' inflexible strategy generation in dynamic industrial environments, which may lead to resource waste or performance bottlenecks.

[0039] Through the above technical solution, this application can generate a more intelligent, adaptive, and efficient industrial hot and cold data storage control strategy. This strategy not only enables tiered storage based on data access frequency, but more importantly, it can respond in real-time to changes in storage system resources and dynamically adjust the trade-off between storage efficiency, access performance, and migration overhead according to business priorities. This ensures that the data storage system always operates in optimal condition in complex and ever-changing industrial environments. Consequently, it significantly improves the overall efficiency and access performance of data storage, reduces operating costs, and enhances the system's robustness against sudden loads and resource constraints, providing a solid guarantee for the effective management of industrial data.

[0040] In some preferred embodiments, a specific example is given below. Suppose that on a large industrial production line, there are various types of data, including real-time sensor data, historical production logs, and equipment maintenance manuals. These data vary in access frequency, importance, and storage performance requirements.

[0041] First, the system monitors the storage system's resource status in real time. For example, it might detect that available space on high-speed solid-state drives (SSDs) is decreasing, while read / write loads on hard disk drives (HDDs) are low, and network bandwidth utilization is at a moderate level. Next, based on predicted access frequency (e.g., real-time sensor data is hot data, historical production logs are warm data, and equipment maintenance manuals are cold data), cost-benefit assessments, and the currently monitored resource status, the system prioritizes storage efficiency, access performance, and migration overhead. For example, due to limited SSD space, the system might prioritize storage efficiency to free up more high-speed storage resources; meanwhile, for real-time sensor data, access performance remains the highest priority. Then, the system evaluates the impact of various storage adjustment actions. For example, for historical production logs that are about to become warm data, the system might evaluate migrating them from SSDs to HDDs and applying a moderate compression algorithm. The evaluation might show that this action significantly improves storage efficiency, has an acceptable impact on access performance, and the migration overhead is manageable under current network bandwidth conditions. For cold data device maintenance manuals, the system may evaluate migrating them from HDDs to archive storage and applying high compression algorithms to maximize storage efficiency. Finally, based on these evaluations and a balance between storage efficiency, access performance, and migration overhead, the system will select the final storage adjustment strategy. For example, the system might decide to migrate some historical production logs from SSDs to HDDs and compress them to alleviate SSD pressure, while migrating infrequently used device maintenance manuals to archive storage, thereby optimizing overall storage resource utilization and cost-effectiveness while ensuring real-time data access performance.

[0042] As one embodiment of the present invention, the resource status includes the available space, read / write load, and network bandwidth of each storage medium.

[0043] The available space for each storage medium refers to the unused storage capacity on each storage medium (such as solid-state drives, hard disk drives, tape libraries, etc.) in the current storage system. This metric reflects the storage system's potential to accept new data or migrate data without data cleanup or expansion. Specifically, the size of the available space directly affects whether data can be successfully migrated from one storage medium to another, and whether a suitable storage location can be allocated for new data.

[0044] Read / write load refers to the intensity of data read and write operations performed on various storage media within a specific time period. This is typically measured by metrics such as Input / Output Operations Per Second (IOPS) and data throughput. High read / write load can lead to prolonged response times for storage media, impacting data access performance.

[0045] Network bandwidth refers to the maximum data transfer rate of a data transfer channel within a storage system or between the storage system and an external network. During data migration, network bandwidth is a critical factor determining the speed and efficiency of the migration process. Insufficient network bandwidth can become a bottleneck even if the storage medium itself is of good performance.

[0046] This application's solution explicitly defines the resource status of the storage system as the available space, read / write load, and network bandwidth of each storage medium, enabling the storage system to more comprehensively and accurately perceive its own operational status. When a storage adjustment strategy needs to be generated, this specific resource status information is monitored in real time and used as key input. For example, by monitoring available space, it can be determined which storage media have sufficient capacity to receive migrated data; by monitoring read / write load, the performance bottlenecks of the current storage media can be assessed, thereby avoiding migrating hot data to overloaded media; by monitoring network bandwidth, the concurrency and transmission rate of data migration can be rationally planned to avoid network congestion. It is precisely because of this precise understanding of these specific resource statuses that subsequent prioritization of storage efficiency, access performance, and migration overhead, as well as the assessment of the impact of storage adjustment actions, can be based on a more solid data foundation, thereby generating a more reasonable and effective storage adjustment strategy.

[0047] As one embodiment of the present invention, the step of allocating compression algorithms and storage media to industrial data of different popularity based on the storage method adjustment strategy includes: Obtain the non-hot characteristics of the industrial data to be allocated, including data integrity level, security sensitivity, and compliance requirements; It's important to note that "non-hotness" characteristics refer to inherent attributes of industrial data that are unrelated to the frequency of access but have a decisive impact on their storage and management. Data integrity level can be understood as the importance of maintaining the accuracy and consistency of data during storage, transmission, and processing. For example, high-integrity-level data may require technologies such as redundant storage, error check codes, or immutable storage to prevent data corruption or tampering. Security sensitivity refers to the degree of harm that unauthorized access, modification, or disclosure of data may cause. For example, highly security-sensitive data may require strong encryption algorithms, access control policies, and physically isolated storage media. Compliance requirements refer to the laws, regulations, industry standards, or corporate policies that data storage and processing must follow. For example, some data may need to meet requirements for long-term archiving, audit trails, or geographic location storage.

[0048] Based on the acquired non-thermal characteristics, a combination of compression algorithms and storage media that meets the non-thermal characteristic requirements is selected; It should be noted that, based on the acquired non-hotness characteristics, an initial screening of available compression algorithms and storage media combinations can be performed. For example, for data with high security sensitivity, compression algorithms and storage media that do not provide sufficient encryption strength or security isolation capabilities will be excluded; for data with high data integrity levels, storage media that support data redundancy, error detection, and correction functions will be prioritized. This ensures that the base set for subsequent allocation schemes can meet the basic non-hotness requirements of the data.

[0049] Based on the predicted access frequency and storage method adjustment strategy, an allocation scheme is selected from the selected combinations of compression algorithms and storage media; It should be noted that this step means that, after satisfying the non-hot data characteristics, the system will further consider the data access frequency and overall storage strategy (such as cost-effectiveness, performance goals, etc.) to optimize the allocation of storage resources while meeting basic requirements. For example, for cold data that meets high security requirements, a lower-cost but more secure archive storage medium and a high compression ratio encryption algorithm may be selected.

[0050] Based on the selected allocation scheme, compression algorithms and storage media are assigned to industrial data of different heat levels.

[0051] This application's solution addresses the potential risks of allocating compression algorithms and storage media solely based on access frequency by incorporating and prioritizing the non-hot characteristics of industrial data. Specifically, it first acquires non-hot characteristics such as data integrity level, security sensitivity, and compliance requirements, which directly determine the minimum standards and necessary conditions for data storage. Subsequently, based on these non-hot characteristics, all available compression algorithm and storage media combinations are rigorously screened to ensure that only solutions that meet the inherent data attribute requirements are considered. Building upon this, and combining predicted access frequency with storage method adjustment strategies, the optimal allocation scheme is selected from the screened qualified schemes. This hierarchical decision-making mechanism ensures that data access frequency and cost-effectiveness are optimized only after meeting its key non-hot requirements, thus avoiding situations where data security, integrity, or compliance are sacrificed for the sake of performance or cost.

[0052] Through the above technical solution, this application enables more comprehensive and refined control of industrial data storage. This solution not only allows for dynamic tiered storage based on data access frequency, but more importantly, it ensures that even industrial data with low access frequency but high value, high sensitivity, or stringent compliance requirements can be allocated to compression algorithms and storage media that meet its specific needs. This significantly improves the security, reliability, and compliance of industrial data storage, effectively reducing the risk of data loss, leakage, or non-compliance with regulations, and providing a solid guarantee for the stable operation of industrial production systems and the effective management of data assets.

[0053] In some preferred embodiments, a specific example is given below. Assume an industrial production system generates three types of data: The first category is real-time control parameters in the production process, which are frequently accessed but also have an extremely high level of data integrity (any loss or tampering could lead to a production accident).

[0054] The second category is employee operation logs, which have moderate access frequency but moderate security sensitivity (containing personal operation information) and strict compliance requirements (must be retained for five years for auditing).

[0055] The third category is historical equipment operation status data, which has low access frequency but generally has a low level of data integrity and low security sensitivity.

[0056] When allocating compression algorithms and storage media, the system first obtains the non-hot properties of these data.

[0057] For the first type of real-time control parameters, due to their high data integrity level, even with high access frequency, the system will prioritize the selection of storage media (e.g., RAID-configured SSD arrays) and lossless compression algorithms that support high redundancy, error checking, and fast recovery.

[0058] For the second type of employee operation logs, due to their security sensitivity and compliance requirements, the system will select storage media that support encrypted storage, access control, and long-term tamper-proof archiving (e.g., WORM storage or blockchain storage) and encrypted compression algorithms with a medium compression ratio.

[0059] For the third category of historical device operating status data, since its non-hotspot characteristics have lower requirements, the system will select a wider range of storage media (e.g., HDD arrays or tape libraries) and compression algorithms with high compression ratios.

[0060] After screening for non-hot data characteristics, the system combines predicted access frequency and cost-effectiveness assessments to select the final allocation scheme from the selected combinations. For example, the first type of data might be allocated to high-performance, high-reliability SSD storage with lossless compression; the second type of data might be allocated to archive storage with encryption and auditing capabilities, using encryption algorithms with medium compression ratios; and the third type of data might be allocated to cost-effective, high-capacity tape libraries or cloud archive storage, using compression algorithms with high compression ratios. This approach ensures that each type of data receives a storage solution that meets both its non-hot data characteristics requirements and its access frequency and cost-effectiveness.

[0061] In some of the embodiments described above in this application, a flexible tiered storage architecture is proposed to achieve dynamic migration between industrial data of varying popularity based on allocated compression algorithms and storage media. However, in real-world industrial applications, simple data migration operations may lead to problems such as data access interruption, excessive system resource consumption, or low migration efficiency. These potential risks are unacceptable, especially in industrial environments where data real-time performance, availability, and system stability are extremely critical.

[0062] In this regard, as one embodiment of the present invention, the steps for achieving dynamic migration between industrial data of different popularity levels through a flexible hierarchical storage architecture, based on the allocated compression algorithm and storage medium, include: Assess the access activity, system I / O load, and network bandwidth usage of the industrial data to be allocated, and adjust the concurrency and transmission rate of dynamic migration based on the assessment results; It's important to note that assessing the access activity, system I / O load, and network bandwidth usage of the industrial data to be allocated aims to understand the current system resource utilization and access patterns of the data to be migrated, thus providing a basis for subsequent migration strategy adjustments. Access activity can be understood as the frequency and intensity of data reading or writing; system I / O load refers to the amount of input / output operations processed by the storage system per unit of time; and network bandwidth usage reflects the busyness of the data transmission channel. Based on these assessment results, the concurrency and transmission rate of data migration can be dynamically adjusted. For example, when the system load is high, the concurrency can be reduced to avoid impacting existing business; when the system is idle, the concurrency can be increased to accelerate the migration process.

[0063] The industrial data to be allocated is divided into data blocks and the data blocks are migrated independently. During the data block migration process, the data on the source storage medium remains accessible. It's important to note that dividing the industrial data to be allocated into data blocks and migrating these blocks independently aims to achieve fine-grained data migration, improving its flexibility and efficiency. Data blocks are the basic unit of data migration, and their size can be configured based on the actual application scenario and storage media characteristics. During the data block migration process, the data on the source storage medium remains accessible. This means that the original data access path remains valid until the data blocks are completely migrated to the target storage medium, ensuring continuous data availability during the migration process and avoiding business interruptions.

[0064] Create a copy of the data block on the target storage medium and apply the assigned compression algorithm to the copy of the data block. After the copy of the data block is written and verified, update the data access path. It's important to note that creating copies of data blocks on the target storage medium and applying the assigned compression algorithm to these copies aims to ensure data integrity and optimize storage space. The choice of compression algorithm is based on the previously generated storage method adjustment strategy and data hotness allocation. After the data block copies are written and verified, the data access path is updated. This step ensures the correctness and consistency of the data on the target medium before data access can be safely redirected to the new storage location.

[0065] During the data access path update, a double buffering mechanism is used to direct write operations to the target storage medium and read operations to the source storage medium until the data access path update is completed. It should be noted that the purpose of the above step is to ensure the continuity and consistency of data read and write operations during the brief window of data access path switching. The double buffering mechanism, by simultaneously maintaining the old and new data paths, allows write operations to immediately apply to the target storage medium, while read operations can still retrieve data from the source storage medium, effectively avoiding data loss or inconsistency issues.

[0066] Monitor system I / O load and network bandwidth during data migration. When the system load exceeds a preset threshold, pause or reduce the concurrency and transmission rate of data migration.

[0067] It should be noted that the purpose of this step is to prevent the data migration process from negatively impacting the normal operation of the industrial production system. Through real-time monitoring, the system can respond promptly to resource bottlenecks, dynamically adjust migration strategies, and ensure that the resource needs of high-priority business operations are met.

[0068] This application's solution effectively addresses issues such as system resource contention, data access interruptions, and low migration efficiency that may arise during the dynamic migration of industrial data. By assessing system resource status and data access activity, migration parameters are intelligently adjusted to ensure a smooth migration process. Furthermore, the adoption of independent data block migration and a double-buffering mechanism significantly improves data availability and consistency during migration. In addition, real-time monitoring and dynamic adjustment mechanisms further guarantee the stable operation of the industrial system during data migration.

[0069] The above technical solution significantly improves the robustness and efficiency of industrial hot and cold data storage control methods. This solution not only ensures continuous availability of data during dynamic migration between different storage media, avoiding business interruptions caused by migration operations, but also effectively balances migration efficiency and system performance through intelligent resource management and dynamic adjustment, reducing the impact of migration on system I / O load and network bandwidth. This has significant practical application value for industrial environments with extremely high requirements for data real-time performance, reliability, and system stability, optimizing storage resource utilization, reducing operating costs, and improving overall data management levels.

[0070] As one embodiment of the present invention, the steps of adjusting the concurrency and transmission rate of dynamic migration include: Obtain the resource usage of high-priority real-time services in the industrial production system. The resource usage includes processor usage, storage I / O throughput, and network bandwidth usage. Based on the resource usage of high-priority real-time services, calculate the amount of remaining resources available for data migration in the system; Prioritize data migration tasks and set a resource allocation cap for each priority level; The concurrency and transmission rate of data migration are allocated based on the priority level of the data migration task and the amount of remaining system resources available for data migration. When changes in resource usage for high-priority real-time services cause the amount of remaining resources available for data migration to fall below the resource requirements of the data migration task, the concurrency and transmission rate of the data migration task should be reduced or suspended to free up resources for high-priority real-time services.

[0071] Specifically, acquiring resource usage data for high-priority real-time applications in an industrial production system refers to the system continuously monitoring and collecting data on the computing resources consumed by critical real-time applications running in the industrial production environment. This includes metrics such as CPU utilization, I / O throughput of storage devices, and network interface bandwidth usage. These high-priority real-time applications typically include tasks with extremely high latency and stability requirements, such as production line control, safety monitoring, and emergency response. The remaining resources available for data migration based on the resource usage of these high-priority applications can be understood as the total amount of resources the system can safely allocate to data migration tasks while ensuring the normal operation of these high-priority applications. This is typically obtained by subtracting the current resource consumption and reserved resources of high-priority applications from the total system resources. In practical applications, classifying data migration tasks into priority levels and setting a resource allocation cap for each priority level means dividing data migration tasks into different priorities based on their importance, timeliness, or business needs, such as high, medium, and low priorities. Each priority level is configured with a maximum available resource amount to prevent low-priority tasks from excessively consuming resources while ensuring that high-priority migration tasks receive the necessary resources. Furthermore, allocating concurrency and transmission rate for data migration based on the priority level of the data migration task and the amount of remaining system resources available for data migration means that the system dynamically adjusts the parallel processing capability (concurrency) and data transmission speed (transmission rate) of data migration based on the currently available remaining resources and the priority of the data tasks to be migrated. For example, high-priority migration tasks can achieve higher concurrency and transmission rates when resources are sufficient, while low-priority tasks will be limited. In addition, when changes in resource usage of high-priority real-time services cause the amount of remaining system resources available for data migration to fall below the resource requirements of the data migration tasks, the concurrency and transmission rate of the data migration tasks are reduced or suspended to release resources for high-priority real-time services. The purpose is to ensure the resource supply for high-priority real-time services. This means that the system continuously monitors changes in the resource requirements of high-priority services. Once it detects an increase in demand that may conflict with data migration tasks, it will immediately take measures, such as reducing the concurrency or transmission rate of ongoing data migration tasks, or even temporarily suspending some migration tasks, to prioritize the operation of critical services.

[0072] This application's solution addresses the issue of resource contention between high-priority real-time business resources in industrial production systems and critical business processes during traditional data migration by incorporating consideration of such resource usage. Specifically, by acquiring real-time processor usage, storage I / O throughput, and network bandwidth usage of high-priority business processes, the system can accurately calculate the amount of remaining resources available for data migration. Based on this, and combined with the prioritization of the data migration task itself and preset resource allocation limits, the system can intelligently allocate the concurrency and transmission rate of data migration, ensuring efficient data migration without impacting critical business processes. More importantly, when the resource demands of high-priority business processes change, resulting in insufficient remaining system resources to meet the current data migration task, the system can respond promptly, proactively reducing or suspending the data migration task. This releases necessary resources for high-priority business processes, fundamentally avoiding resource conflicts and ensuring the stable operation of the industrial production system.

[0073] As one embodiment of the present invention, the step of adjusting the concurrency and transmission rate of dynamic migration further includes: When changes in resource usage for high-priority real-time services cause the amount of remaining resources available for data migration to fall below the resource requirements of the data migration task, the concurrency and transmission rate of the data migration task should be reduced or suspended to free up resources for high-priority real-time services.

[0074] Specifically, changes in resource usage for high-priority real-time services can be understood as a sudden increase in the demand for processor usage, storage I / O throughput, or network bandwidth by critical control, monitoring, or safety systems in industrial production systems. For example, when an emergency occurs on the production line, requiring immediate processing of large amounts of sensor data or execution of emergency control commands, the resource consumption of these services will rise rapidly. When the remaining resources available for data migration are lower than the resource requirements of the data migration task, it means that after considering the current and predicted resource requirements of high-priority services, the remaining resources available for data migration are insufficient to support the current data migration task continuing to run at its existing concurrency and transmission rate. In this case, the system will immediately trigger a response mechanism to reduce the concurrency of the data migration task, such as reducing the number of simultaneous data block migrations, or directly suspending some or all data migration tasks, and reducing their transmission rate, to ensure that high-priority real-time services can obtain sufficient resources. The purpose is to prioritize the stable operation and performance of core industrial production services and avoid negative impacts on critical services due to excessive resource consumption during data migration activities.

[0075] The proposed solution ensures resource supply for high-priority real-time services by immediately reducing or suspending data migration tasks when increased resource demands for high-priority real-time services are detected and the system's remaining resources are insufficient to simultaneously meet the demands of data migration tasks. This mechanism enables the system to dynamically and preferentially respond to resource requests from critical services, avoiding resource contention caused by the continuous operation of data migration tasks, and effectively solving the problem of performance degradation of high-priority services under resource constraints. Through this proactive resource release strategy, the system can maintain the continuity and stability of industrial production, especially in emergencies, enabling rapid adjustments to resource allocation to ensure the normal operation of core services.

[0076] like Figure 2 The diagram illustrates an industrial cold and hot data storage control system, which includes: Access feature acquisition module 201 is used to acquire the access features of the industrial data to be allocated, and predict the access popularity of the industrial data based on the access features. The strategy generation module 202 is used to generate storage method adjustment strategies based on predicted access popularity and cost-effectiveness assessment. The allocation module 203 is used to adjust the strategy according to the storage method and allocate compression algorithms and storage media to industrial data with different levels of popularity. The dynamic migration module 204 is used to achieve dynamic migration between industrial data of different popularity through an elastic tiered storage architecture, based on the allocated compression algorithm and storage medium.

[0077] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. An industrial cold and hot data storage control method, characterized by, The method includes the following steps: Obtain the access characteristics of the industrial data to be allocated, and predict the access popularity of the industrial data based on the access characteristics; Based on the predicted access volume and cost-effectiveness assessment, adjust the storage strategy accordingly; Based on the storage method, an adjustment strategy is implemented to allocate compression algorithms and storage media for industrial data with different levels of popularity. Based on the allocated compression algorithm and storage medium, a flexible tiered storage architecture enables dynamic migration between industrial data of varying importance.

2. The industrial cold and hot data storage control method of claim 1, wherein, The step of obtaining the access characteristics of the industrial data to be allocated and predicting the access popularity of the industrial data based on the access characteristics includes: Real-time monitoring of access characteristics of industrial data streams containing industrial data to be assigned; Based on the access characteristics, the statistical distribution of the access characteristics is dynamically analyzed to determine whether the statistical distribution deviates from a preset stable range; When the statistical distribution deviates from the preset stable range, the key access features that cause the deviation are identified, and the judgment threshold or feature weight of the access popularity prediction is adjusted according to the degree of deviation of the key access features. Based on the adjusted judgment threshold or feature weights, the access popularity of the industrial data is predicted.

3. The industrial cold and hot data storage control method of claim 2, wherein, The method further includes the following steps: The accuracy of the access popularity prediction is evaluated based on the processing effect of different popularity data during actual storage, and the judgment threshold or feature weight is adjusted based on the accuracy feedback.

4. The industrial cold and hot data storage control method of claim 1, wherein, The step of generating a storage method adjustment strategy based on predicted access volume and cost-effectiveness assessment includes: Real-time monitoring of storage system resource status; Based on the predicted access volume, the cost-benefit assessment results, and the resource status of the storage system, the priorities of storage efficiency, access performance, and migration overhead are determined. Based on the priority of the migration overhead, evaluate the impact of storage method adjustment actions on storage efficiency, access performance and migration overhead, wherein the adjustment actions include combinations of different compression algorithms and storage media as well as data migration operations; Based on the evaluation results, a storage method adjustment strategy is selected by balancing the storage efficiency, access performance, and migration overhead, and the storage method adjustment strategy is generated.

5. The industrial cold and hot data storage control method of claim 4, wherein, The resource status includes the available space, read / write load, and network bandwidth of each storage medium.

6. The industrial cold and hot data storage and control method according to claim 1, characterized in that, The step of adjusting the strategy according to the storage method to allocate compression algorithms and storage media for industrial data with different levels of popularity includes: Obtain the non-hot characteristics of the industrial data to be allocated, including data integrity level, security sensitivity, and compliance requirements; Based on the acquired non-thermal characteristics, a combination of compression algorithms and storage media that meets the requirements of the non-thermal characteristics is selected; Based on the predicted access frequency and the storage method adjustment strategy, the allocation scheme is selected from the selected combinations of compression algorithms and storage media; Based on the selected allocation scheme, compression algorithms and storage media are assigned to industrial data of different heat levels.

7. The industrial cold and hot data storage control method according to claim 1, characterized in that, The steps for dynamically migrating industrial data of different popularity based on the allocated compression algorithm and storage medium through a flexible tiered storage architecture include: Assess the access activity, system I / O load, and network bandwidth usage of the industrial data to be allocated, and adjust the concurrency and transmission rate of dynamic migration based on the assessment results; The industrial data to be allocated is divided into data blocks and the data blocks are migrated independently, while the data on the source storage medium remains accessible during the data block migration process; A copy of the data block is created on the target storage medium, and the allocated compression algorithm is applied to the copy of the data block. After the copy of the data block is written and verified, the data access path is updated. During the data access path update, a double buffering mechanism is used to direct write operations to the target storage medium and read operations to the source storage medium until the data access path update is completed. Monitor system I / O load and network bandwidth during data migration. When the system load exceeds a preset threshold, pause or reduce the concurrency and transmission rate of the data migration.

8. The industrial cold and hot data storage control method according to claim 7, characterized in that, The steps for adjusting the concurrency and transmission rate of dynamic migration include: The resource usage of high-priority real-time services in an industrial production system is obtained, including processor usage, storage I / O throughput, and network bandwidth usage. Based on the resource usage of the high-priority real-time services, calculate the amount of remaining system resources available for data migration; Prioritize data migration tasks and set a resource allocation cap for each priority level; The concurrency and transmission rate of the data migration are allocated according to the priority level of the data migration task and the amount of remaining resources available for data migration in the system. When the resource usage of the high-priority real-time service changes, causing the amount of remaining resources available for data migration in the system to be lower than the resource requirements of the data migration task, the concurrency and transmission rate of the data migration task are reduced or suspended to release resources for the high-priority real-time service.

9. The industrial cold and hot data storage and control method according to claim 8, characterized in that, The steps for adjusting the concurrency and transmission rate of dynamic migration also include: When the resource usage of the high-priority real-time service changes, causing the amount of remaining resources available for data migration in the system to be lower than the resource requirements of the data migration task, the concurrency and transmission rate of the data migration task are reduced or suspended in order to release resources for the high-priority real-time service.

10. An industrial cold and hot data storage control system, used to execute an industrial cold and hot data storage control method as described in any one of claims 1-9, characterized in that, The system includes: The access feature acquisition module is used to acquire the access features of the industrial data to be allocated, and predict the access popularity of the industrial data based on the access features. The strategy generation module is used to generate storage adjustment strategies based on predicted access volume and cost-effectiveness assessments. The allocation module is used to adjust the strategy according to the storage method and allocate compression algorithms and storage media to industrial data with different levels of popularity. The dynamic migration module is used to achieve dynamic migration between industrial data of different popularity based on the allocated compression algorithm and storage medium through a flexible tiered storage architecture.

Images