Container storage management method and device, equipment and storage medium

Abstract

The invention relates to the field of data storage, and discloses a container storage management method, device and equipment and a storage medium, and the method is used for guaranteeing the high efficiency, stability and safety of data storage processing through data deduplication and compression, construction of a multi-layer storage architecture and monitoring early warning. The method comprises the following steps: acquiring container configuration parameters for container configuration; when a data storage instruction is received, performing deduplication processing on the data written into the container by adopting a distributed deduplication algorithm; compressing the deduplicated data by using a hybrid compression algorithm, and dynamically adjusting compression parameters according to the real-time compression effect of the deduplicated data to obtain compressed data; constructing a multi-layer storage framework in the container to obtain a plurality of storage layers, dividing the compressed data into hot data, temperature data and cold data, and respectively storing the hot data, the temperature data and the cold data in different storage layers; and monitoring the compressed data in different storage layers based on a preset monitoring index, and when an abnormal condition occurs, generating different early warning modes for different storage layers.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a container storage management method, device, equipment and storage medium. Background Art

[0002] With the continuous evolution of cloud computing technology, containerized applications have become a mainstream trend. With its rapid deployment capabilities, elastic expansion characteristics and efficient resource isolation mechanism, it has significantly improved the efficiency of application development and operation and maintenance. However, the traditional data storage method of containers has gradually exposed many problems when facing such large-scale and diverse data. On the one hand, there is a large amount of duplicate data in the data storage process, which not only occupies a lot of valuable storage space, but also increases the complexity and cost of data management. For example, in the backup systems of some enterprises, due to the lack of an effective deduplication mechanism, the same data may be stored multiple times, resulting in a huge waste of storage resources.

[0003] On the other hand, for different types of data, such as text, images, audio and video, using a single compression algorithm often cannot achieve the best compression effect. If a unified simple compression algorithm is used for all data, some data may not have a high compression ratio and cannot effectively save storage space; and for some data with high compression quality requirements, data quality may be lost due to improper compression algorithms.

[0004] In terms of storage architecture, the traditional single storage layer is difficult to meet the needs of different data access frequencies and importance. If hot data (data with high access frequency and high importance) is stored in low-speed storage devices, it will lead to slow data reading and writing speeds, seriously affecting the real-time nature of the business and user experience; while if cold data (data with low access frequency and relatively low importance) is stored in high-speed storage devices, it will cause a waste of storage resources and increase unnecessary costs.

[0005] In addition, traditional data storage monitoring methods also have shortcomings. Usually, they simply monitor some basic indicators, lack comprehensive and in-depth monitoring of stored data, and are difficult to detect potential abnormalities in advance. Once a problem occurs, it is often impossible to issue an early warning in a timely and accurate manner, resulting in serious consequences such as data loss and business interruption.

[0006] Therefore, the existing technology still needs to be improved and developed. Summary of the invention

[0007] The present invention provides a container storage management method, device, equipment and storage medium, which are used to ensure the efficiency, stability and security of data storage and processing through data deduplication and compression, construction of a multi-level storage architecture and monitoring and early warning.

[0008] A first aspect of the present invention provides a container storage management method, which includes: obtaining container configuration parameters, and configuring the container according to the container configuration parameters; when a data storage instruction is received, using a distributed deduplication algorithm to deduplicate data written to the container to obtain deduplicated data; using a hybrid compression algorithm to compress the deduplicated data, and through an adaptive compression algorithm, dynamically adjusting compression parameters according to the real-time compression effect of the deduplicated data to obtain compressed data; constructing a multi-level storage architecture in the container to obtain multiple storage layers, dividing the compressed data into hot data, warm data and cold data, and storing them in different storage layers respectively; monitoring the compressed data in different storage layers based on preset monitoring indicators, and when an abnormal situation occurs, generating different warning methods for different storage layers.

[0009] Optionally, in a first implementation manner of the first aspect of the present invention, obtaining container configuration parameters and configuring the container according to the container configuration parameters includes: obtaining container configuration parameters, the container configuration parameters including deployment and communication requirements, load balancing algorithm, network buffer size, and network timeout parameters; configuring the network topology of the container according to the deployment and communication requirements of the container; configuring the load balancing strategy of the container according to the load balancing algorithm; and configuring the network parameters of the container according to the network buffer size and the network timeout parameters.

[0010] Optionally, in a second implementation method of the first aspect of the present invention, when a data storage instruction is received, a distributed deduplication algorithm is used to deduplicate the data written to the container to obtain deduplicated data, including: when a data storage instruction is received, the data written to the container is divided into multiple smaller data blocks, and a hash value of each data block is calculated; the hash value of each data block is transmitted to each distributed node of the container, and if the hash value of the data block is found on a certain distributed node to match the hash value in the local database, the data block is determined to be duplicate data; otherwise, the data block is determined to be new data; and the new data is integrated to obtain deduplicated data.

[0011] Optionally, in a third implementation method of the first aspect of the present invention, the deduplicated data is compressed using a hybrid compression algorithm, and compression parameters are dynamically adjusted according to the real-time compression effect of the deduplicated data through an adaptive compression algorithm to obtain compressed data, including: performing feature analysis on the deduplicated data, and dividing the deduplicated data into multiple data segments according to the analysis results, and determining the compression algorithm corresponding to each data segment according to the type of each data segment; for each data segment, obtaining initial compression parameters of the compression algorithm, and based on the initial compression parameters, using the compression algorithm to compress the data segment; if the compression ratio of the compressed data segment is lower than a preset compression threshold, using an adaptive compression algorithm to dynamically adjust the compression parameters of the compression algorithm, and based on the adjusted compression parameters, using the compression algorithm to compress the data segment again to obtain optimized compressed data segments; integrating all optimized compressed data segments to obtain compressed data.

[0012] Optionally, in a fourth implementation method of the first aspect of the present invention, feature analysis is performed on the deduplicated data, and the deduplicated data is divided into multiple data segments according to the analysis results, and the compression algorithm corresponding to each data segment is determined according to the type of each data segment, including: feature analysis is performed on the deduplicated data, and the deduplicated data is divided into multiple data segments according to the analysis results, and the type of each data segment is determined; if the type of the data segment is text data, the LZW algorithm is used to compress the corresponding data segment; if the type of the data segment is image data, the JPEG algorithm is used to compress the corresponding data segment; if the type of the data segment is audio and video data, the MPEG-4 algorithm is used to compress the corresponding data segment.

[0013] Optionally, in a fifth implementation of the first aspect of the present invention, a multi-level storage architecture is constructed in the container to obtain multiple storage layers, the compressed data is divided into hot data, warm data and cold data, and the data are respectively stored in different storage layers, including: constructing a multi-level storage architecture in the container based on a preset number of levels to obtain multiple storage layers; dividing the compressed data into hot data, warm data and cold data according to the access frequency and importance of the compressed data and the operating status and business cycle of the container; and storing the hot data, the warm data and the cold data in different storage layers.

[0014] Optionally, in a sixth implementation method of the first aspect of the present invention, the compressed data in different storage layers is monitored based on preset monitoring indicators, and when an abnormal situation occurs, different warning methods are generated for different storage layers, including: collecting historical monitoring indicator data of different storage layers in a historical period, analyzing the collected historical monitoring indicator data using a time series analysis method, and establishing a normal range value for each monitoring indicator; collecting real-time monitoring data of compressed data in different storage layers based on preset monitoring indicators; comparing the collected real-time monitoring data with the normal range value of each monitoring indicator one by one, and if the real-time monitoring data exceeds the normal range value of any monitoring indicator, generating different warning methods for different storage layers.

[0015] The second aspect of the present invention provides a container storage management device, including: a configuration module, which is used to obtain container configuration parameters and configure the container according to the container configuration parameters; a deduplication module, which is used to use a distributed deduplication algorithm to deduplicate data written into the container when receiving a data storage instruction, so as to obtain deduplicated data; a compression module, which is used to compress the deduplicated data using a hybrid compression algorithm, and dynamically adjust the compression parameters according to the real-time compression effect of the deduplicated data through an adaptive compression algorithm to obtain compressed data; a storage module, which is used to construct a multi-level storage architecture in the container, obtain multiple storage layers, divide the compressed data into hot data, warm data and cold data, and store them in different storage layers respectively; a monitoring module, which is used to monitor the compressed data in different storage layers based on preset monitoring indicators, and when an abnormal situation occurs, generate different early warning methods for different storage layers.

[0016] Optionally, in a first implementation manner of the second aspect of the present invention, the configuration module includes: an acquisition unit, used to acquire container configuration parameters, the container configuration parameters including deployment and communication requirements, load balancing algorithm, network buffer size and network timeout parameters; a first configuration unit, used to configure the network topology of the container according to the deployment and communication requirements of the container; a second configuration unit, used to configure the load balancing strategy of the container according to the load balancing algorithm; and a third configuration unit, used to configure the network parameters of the container according to the size of the network buffer and the network timeout parameters.

[0017] Optionally, in a second implementation of the second aspect of the present invention, the deduplication module includes: a segmentation unit, which is used to segment the data written into the container into multiple smaller data blocks when a data storage instruction is received, and calculate the hash value of each data block; a deduplication unit, which is used to transmit the hash value of each data block to each distributed node of the container. If the hash value of the data block is found on a certain distributed node to match the hash value in the local database, the data block is determined to be duplicate data; otherwise, the data block is determined to be new data; a first integration unit, which is used to integrate the new data to obtain deduplicated data.

[0018] Optionally, in a third implementation of the second aspect of the present invention, the compression module includes: an analysis unit, used to perform feature analysis on the deduplicated data, and divide the deduplicated data into multiple data segments according to the analysis results, and determine the compression algorithm corresponding to each data segment according to the type of each data segment; a compression unit, used to obtain initial compression parameters of the compression algorithm for each data segment, and based on the initial compression parameters, use the compression algorithm to compress the data segment; if the compression ratio of the compressed data segment is lower than a preset compression threshold, use an adaptive compression algorithm to dynamically adjust the compression parameters of the compression algorithm, and based on the adjusted compression parameters, use the compression algorithm to compress the data segment again to obtain an optimized compressed data segment; a second integration unit, used to integrate all optimized compressed data segments to obtain compressed data.

[0019] Optionally, in a fourth implementation of the second aspect of the present invention, the storage module includes: a construction unit, used to construct a multi-level storage architecture in the container based on a preset number of levels to obtain multiple storage layers; a classification unit, used to divide the compressed data into hot data, warm data and cold data according to the access frequency and importance of the compressed data and the operating status and business cycle of the container; a storage unit, used to store the hot data, the warm data and the cold data in different storage layers, respectively.

[0020] Optionally, in a fifth implementation of the second aspect of the present invention, the monitoring module includes: a collection unit, used to collect historical monitoring indicator data of different storage layers in a historical period, use a time series analysis method to analyze the collected historical monitoring indicator data, and establish a normal range value for each monitoring indicator; a monitoring unit, used to collect real-time monitoring data of compressed data in different storage layers based on preset monitoring indicators; an early warning unit, used to compare the collected real-time monitoring data with the normal range value of each monitoring indicator one by one. If the real-time monitoring data exceeds the normal range value of any monitoring indicator, different early warning methods are generated for different storage layers.

[0021] A third aspect of the present invention provides a container storage management device, comprising: a memory and at least one processor, wherein the memory stores computer-readable instructions, and the memory and the at least one processor are interconnected via a line; and the at least one processor calls the computer-readable instructions in the memory so that the container storage management device executes each step of the container storage management method as described above.

[0022] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, which, when executed on a computer, enable the computer to execute the various steps of the container storage management method described above.

[0023] In the technical solution provided by the present invention, in terms of container configuration, by obtaining detailed parameters, it can accurately create a suitable operating environment, avoid unreasonable resource allocation, optimize communication capabilities, and lay the foundation for subsequent work; in terms of data storage, the distributed deduplication algorithm can process data deduplication in parallel, reduce storage volume and reduce costs; hybrid compression is combined with adaptive compression algorithms to improve compression rate and save storage and transmission time; in terms of storage architecture, the multi-level storage architecture stores data in layers according to data characteristics and business conditions, which not only meets the high-speed reading and writing requirements of hot data, but also balances cost and performance, and can adapt to business changes; in terms of monitoring and early warning, the blockchain-based mechanism ensures that monitoring indicators are true and reliable, and uses machine learning to predict anomalies by analyzing historical indicators, and generates early warnings according to severity levels to ensure stable operation of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A first flow chart of a container storage management method provided by an embodiment of the present invention;

[0025] Figure 2 A second flow chart of the container storage management method provided by an embodiment of the present invention;

[0026] Figure 3 A third flow chart of the container storage management method provided by an embodiment of the present invention;

[0027] Figure 4 A fourth flow chart of the container storage management method provided by an embodiment of the present invention;

[0028] Figure 5 A fifth flow chart of the container storage management method provided by an embodiment of the present invention;

[0029] Figure 6 A sixth flow chart of the container storage management method provided by an embodiment of the present invention;

[0030] Figure 7A schematic diagram of the structure of a container storage management device provided by an embodiment of the present invention;

[0031] Figure 8 A schematic diagram of the structure of a container storage management device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The embodiment of the present invention provides a container storage management method, device, equipment and storage medium, which is used to ensure the efficiency, stability and security of data storage and processing through data deduplication and compression, construction of a multi-level storage architecture and monitoring and early warning. The method includes: obtaining container configuration parameters, configuring the container according to the container configuration parameters; when receiving a data storage instruction, using a distributed deduplication algorithm to deduplicate the data written to the container to obtain deduplicated data; using a hybrid compression algorithm to compress the deduplicated data, and through an adaptive compression algorithm, dynamically adjusting the compression parameters according to the real-time compression effect of the deduplicated data to obtain compressed data; constructing a multi-level storage architecture in the container to obtain multiple storage layers, dividing the compressed data into hot data, warm data and cold data, and storing them in different storage layers respectively; monitoring the compressed data in different storage layers based on preset monitoring indicators, and generating different early warning methods for different storage layers when an abnormal situation occurs.

[0033] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 A first embodiment of a container storage management method in an embodiment of the present invention includes:

[0035] S101: Obtain container configuration parameters, and configure the container according to the container configuration parameters.

[0036] In this embodiment, the container configuration parameters include various information required for container operation, such as deployment and communication requirements, load balancing algorithms, network buffer size, and network timeout parameters. By accurately obtaining these parameters and configuring them, it is possible to ensure that the container runs in a suitable environment, providing a basic guarantee for subsequent operations such as data storage. Appropriate resource configuration can avoid performance problems caused by insufficient or over-allocated resources in the container, and the network configuration determines the communication capabilities between the container and external systems and other containers.

[0037] It is understandable that the execution subject of the present invention may be a container storage management device, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking a server as the execution subject as an example.

[0038] S102: When a data storage instruction is received, a distributed deduplication algorithm is used to deduplicate the data written into the container to obtain deduplicated data.

[0039] In this embodiment, the distributed deduplication algorithm is of great significance in large-scale data storage scenarios. In container storage, a large amount of data may have duplicate parts. Deduplication can effectively reduce the storage amount of data and save storage space. The distributed characteristics enable deduplication operations to be performed in parallel on multiple nodes, improving processing efficiency. For example, in a multi-node container cluster, each node can perform deduplication on the data it receives, and then integrate and store the deduplication data, which not only improves processing speed but also reduces storage costs.

[0040] S103, using a hybrid compression algorithm to compress the deduplicated data, and dynamically adjusting compression parameters according to the real-time compression effect of the deduplicated data through an adaptive compression algorithm to obtain compressed data.

[0041] In this embodiment, the hybrid compression algorithm combines the advantages of multiple compression algorithms. Different types of data (such as text, image, binary data, etc.) have different characteristics and are suitable for different compression algorithms. For example, text data is suitable for dictionary-based compression algorithms (such as Lempel-Ziv-Welch algorithm), while image data may be more suitable for transformation-based compression algorithms (such as JPEG algorithm). The hybrid compression algorithm selects the most suitable algorithm for combined compression according to different parts of the data, which can improve the overall compression rate. The adaptive compression algorithm further optimizes the compression process, and it can dynamically adjust the compression parameters according to the real-time compression effect of the data. For example, if it is found that the current data is not well compressed by a certain compression algorithm, the adaptive algorithm can automatically adjust the parameters or switch to other more suitable algorithms to ensure that a better compression effect can always be obtained, thereby further saving space during storage and reducing transmission time during data transmission.

[0042] S104: Build a multi-level storage architecture in the container to obtain multiple storage layers, divide the compressed data into hot data, warm data, and cold data, and store them in different storage layers respectively.

[0043] In this embodiment, a multi-level storage architecture is an effective way to optimize storage performance and cost. Hot data refers to data that is frequently accessed. This type of data has high access speed requirements, so storing it in a high-speed storage layer (such as a solid-state drive SSD) can ensure a quick response to read and write requests and improve the overall performance of the application. Warm data has a moderate access frequency and is stored in a storage layer with a relatively balanced performance and cost (such as an ordinary mechanical hard disk HDD). Cold data is data that is rarely accessed and can usually be stored in a low-cost, large-capacity storage layer (such as a tape library). Dividing data according to the operating status and business cycle of the container can better adapt to business changes. For example, during business peaks, more hot data may be generated, and it is necessary to ensure that the high-speed storage layer has sufficient space and performance to support it; during business troughs, data can be reorganized and migrated to optimize the utilization of storage resources.

[0044] S105. Monitor the compressed data in different storage layers based on preset monitoring indicators, and when an abnormal situation occurs, generate different warning methods for different storage layers.

[0045] In this embodiment, the high-speed storage layer focuses on setting indicators such as data read and write speed and storage capacity utilization. Since the high-speed storage layer mainly processes hot data and has extremely high requirements for read and write performance, the normal range of data read and write speed should be accurately set according to the real-time requirements of the business. Storage capacity utilization also needs to be closely monitored to ensure that data read and write requests can be quickly responded to under high load.

[0046] In addition to data read and write speed and storage capacity utilization, data error rate is a key indicator of the medium-speed storage layer. The normal range of data read and write speed can be set according to the daily business access frequency and data volume.

[0047] The low-speed, large-capacity storage layer mainly sets indicators such as storage capacity, device temperature, and disk I / O queue length. Storage capacity needs to be monitored in real time to ensure that its remaining space can meet the long-term storage needs of cold data. The device temperature should be kept within the normal operating temperature range, such as [G]℃-[H]℃, and the disk I / O queue length should not be too long to avoid excessive data read and write delays.

[0048] This embodiment provides a container storage management method, which can accurately create a suitable operating environment by obtaining detailed parameters in container configuration, avoid unreasonable resource allocation, optimize communication capabilities, and lay the foundation for subsequent work; in terms of data storage, the distributed deduplication algorithm can process data deduplication in parallel, reduce storage volume and reduce costs; hybrid compression is combined with adaptive compression algorithms to improve compression rate and save storage and transmission time; in terms of storage architecture, the multi-level storage architecture stores data in layers according to data characteristics and business conditions, which not only meets the high-speed reading and writing requirements of hot data, but also balances cost and performance and can adapt to business changes; in terms of monitoring and early warning, the blockchain-based mechanism ensures that monitoring indicators are true and reliable, and uses machine learning to predict anomalies by analyzing historical indicators, and generates early warnings according to severity levels to ensure stable operation of the system.

[0049] See also Figure 2 , a second embodiment of the container storage management method in the embodiment of the present invention includes:

[0050] S201. Obtain container configuration parameters, which include deployment and communication requirements, load balancing algorithm, network buffer size, and network timeout parameters.

[0051] In this embodiment, the deployment and communication requirements determine the position of the container in the entire system architecture and the way it interacts with other components. Understanding these requirements helps determine key information such as the type of network connection and data transmission volume required by the container, and is the basis for subsequent network topology configuration. The load balancing algorithm is related to how to reasonably distribute requests to each container instance to ensure the high availability and performance of the system. Different load balancing algorithms are suitable for different scenarios. Obtaining this parameter can select the optimal allocation strategy according to actual conditions. The network buffer size affects the temporary storage capacity of the container during data transmission. The appropriate buffer size can prevent data loss while avoiding excessive system resources due to being too large. The network timeout parameter specifies the maximum waiting time for network operations (such as request sending, response receiving, etc.). It is crucial to promptly handle network anomalies and avoid invalid waiting, and directly affects the response speed and stability of the system.

[0052] S202: Configure the network topology of the container according to the deployment and communication requirements of the container.

[0053] In this embodiment, if the deployment and communication requirements of the containers indicate that frequent and efficient communication is required between the containers, a star topology is configured as the network topology of the container. In a star topology, all containers are connected to a central node, which is responsible for forwarding and managing data. This structure facilitates centralized control and management, and has high communication efficiency, because data is transferred through the central node, reducing confusion and conflicts in data transmission. If the communication between containers is more decentralized and peer-to-peer, a mesh topology is configured as the network topology of the container. In a mesh topology, each container is directly connected to multiple other containers, and data can be transmitted directly between containers without passing through intermediate nodes. The advantage of this structure is high reliability, and even if some connections fail, data can still be transmitted through other paths.

[0054] S203: Configure the load balancing strategy of the container according to the load balancing algorithm.

[0055] In this embodiment, the load balancing algorithm uses a weighted polling algorithm to make up for this deficiency. It assigns different weights to each container according to its performance (such as CPU processing power, memory size, etc.). Containers with high performance have high weights and will receive more requests; containers with low performance have low weights and receive relatively few requests. In this way, the load can be distributed more reasonably, the resources of each container can be fully utilized, and the overall performance of the system can be improved. By accurately configuring the load balancing strategy and parameters, it is possible to avoid overloading some containers and ensure the stability and response speed of the system.

[0056] S204: Configure the network parameters of the container according to the size of the network buffer and the network timeout parameters.

[0057] In this embodiment, the size of the network buffer is crucial to the stability of data transmission. If the buffer is too small, when the amount of data transmission suddenly increases, the data may be lost because it cannot be processed in time. For example, during the peak period of the network, a large amount of data arrives at the container at the same time. If the buffer cannot accommodate this data, packet loss will occur, affecting the integrity of the data and the normal operation of the system. On the contrary, if the buffer is too large, although data loss can be avoided, it will occupy too many system memory resources, resulting in a reduction in available resources for other processes or services, affecting the performance of the entire system. Appropriate network timeout parameters can handle network anomalies in a timely manner. When the network fails or the delay is too high, if the timeout is set too long, the system may wait for an invalid response for a long time, reducing the user experience; if the timeout is set too short, normal network delays may be misjudged as timeout errors, resulting in unnecessary retries or error handling, affecting the stability of the system.

[0058] In this embodiment, by obtaining rich container configuration parameters covering deployment and communication requirements, load balancing algorithms, network buffer size, and network timeout parameters, the foundation is laid for subsequent precise configuration; moreover, by configuring the network topology according to the deployment and communication requirements, the communication path between containers can be optimized, and the load balancing strategy can be configured according to the load balancing algorithm. For example, the weighted polling algorithm can reasonably distribute the load according to the container performance, which can avoid overloading of some containers and improve the overall performance of the system; in addition, by configuring the network parameters based on the network buffer size and network timeout parameters, the data transmission can be ensured to be stable, data loss can be prevented, network anomalies can be handled in a timely manner, misjudgment can be avoided, and the stability and efficiency of the container in network communication can be guaranteed.

[0059] See also Figure 3 A third embodiment of a container storage management method in an embodiment of the present invention includes:

[0060] S301. When a data storage instruction is received, the data written into the container is divided into multiple smaller data blocks, and a hash value of each data block is calculated.

[0061] In this embodiment, for example, for a large file, the block operation can be performed according to a preset fixed size (such as 1 MB), so that each block becomes an independent data block, which is convenient for subsequent processing.

[0062] After the data is sharded, the hash value of each data block is calculated. The function of the hash function is to convert data of any length into a hash value of fixed length. Different data will normally produce different hash values ​​(although there is a very small probability of hash collision). In practical applications, common hash algorithms such as MD5 and SHA-256 are suitable for this step. The hash value is equivalent to the unique "fingerprint" of the data. The hash value is then compared to determine whether the data is repeated. Compared with directly comparing the data content, it greatly reduces the computing overhead and improves the judgment efficiency.

[0063] S302, transmitting the hash value of each data block to each distributed node of the container. If the hash value of the data block is found to match the hash value in the local database on a distributed node, the data block is determined to be duplicate data; otherwise, the data block is determined to be new data.

[0064] In this embodiment, each distributed node maintains a local hash value database (or cache), which is used to store hash values ​​of previously processed data.

[0065] When a new hash value (i.e., the hash value of a data block) is transmitted to a distributed node, the node will immediately compare the new hash value with the hash values ​​in the local database one by one. If the new hash value is found to completely match a hash value in the local database during the comparison process, the corresponding data block is determined to be duplicate data; conversely, if the new hash value does not match any hash value in the local database, the corresponding data block is determined to be new data.

[0066] S303: Integrate new data to obtain deduplicated data.

[0067] In this embodiment, in data processing, the data is quickly segmented after receiving the storage instruction, and the efficiency is significantly improved by leveraging the parallel processing capabilities of the distributed system. For example, when storing massive files, multi-node parallel processing can shorten the time. In terms of computing resource utilization, hash value comparison is used instead of direct comparison of data content to greatly reduce computing overhead and improve resource utilization. In addition, the distributed architecture avoids the bottleneck of centralized data processing, and the local hash value database can quickly respond to deduplication requests to enhance real-time performance.

[0068] See also Figure 4 A fourth embodiment of a container storage management method in an embodiment of the present invention includes:

[0069] S401, performing feature analysis on the deduplicated data, dividing the deduplicated data into multiple data segments according to the analysis results, and determining a compression algorithm corresponding to each data segment according to the type of each data segment.

[0070] In this embodiment, feature analysis is performed on the deduplicated data, and the deduplicated data is divided into multiple data segments according to the analysis results, and the compression algorithm corresponding to each data segment is determined according to the type of each data segment, specifically including: feature analysis is performed on the deduplicated data, and the deduplicated data is divided into multiple data segments according to the analysis results, and the type of each data segment is determined; if the type of the data segment is text data, the LZW algorithm is used to compress the corresponding data segment; if the type of the data segment is image data, the JPEG algorithm is used to compress the corresponding data segment; if the type of the data segment is audio and video data, the MPEG-4 algorithm is used to compress the corresponding data segment.

[0071] S402. For each data segment, obtain initial compression parameters of the compression algorithm, and based on the initial compression parameters, use the compression algorithm to compress the data segment. If the compression ratio of the compressed data segment is lower than a preset compression threshold, use an adaptive compression algorithm to dynamically adjust the compression parameters of the compression algorithm. Based on the adjusted compression parameters, use the compression algorithm to compress the data segment again to obtain an optimized compressed data segment.

[0072] In this embodiment, for each divided data segment, the initial compression parameters of the corresponding compression algorithm are obtained. These parameters play a key role in the compression effect and speed. Taking the JPEG algorithm as an example, the initial quality factor setting will directly affect the compression ratio and fidelity of the image. The higher the quality factor, the lower the image compression ratio, because more image details are retained, so the fidelity is higher; conversely, the lower the quality factor, the higher the compression ratio, but the image may have more distortion, such as blurring, color blocks, etc. Based on these initial compression parameters, each data segment is initially compressed using the corresponding compression algorithm, and different data segments are processed by their corresponding algorithms.

[0073] After the initial compression is completed, the compression effect of each compressed data segment must be strictly evaluated. The main evaluation indicator is to calculate the compression ratio, that is, the ratio of the size of the compressed data segment to the size of the original data segment, and compare it with the preset compression threshold. At the same time, for image, audio and other data, it is also necessary to check the integrity and accuracy of the data, evaluate the quality of the decompressed data, and check whether there is obvious distortion. If the compression ratio of a compressed data segment is lower than the preset compression threshold, it means that the current compression effect does not meet expectations. At this time, the adaptive compression algorithm is used to dynamically adjust the compression parameters of the compression algorithm corresponding to the data segment. For example, if the image data segment is severely distorted after compression, the quality factor of the JPEG algorithm should be appropriately increased; if the compression ratio is too low, try to reduce the quality factor or adjust other related parameters, and optimize the compression effect by continuously adjusting the parameters.

[0074] In this embodiment, the process of real-time compression effect evaluation, parameter adjustment, and re-compression is repeated until a satisfactory compression effect is achieved for each data segment.

[0075] S403: Integrate all optimized compressed data segments to obtain compressed data.

[0076] In this embodiment, all optimized compressed data segments are integrated, and the obtained compressed data is the result of collaborative processing by the hybrid compression algorithm and the adaptive compression algorithm.

[0077] In this embodiment, by performing accurate feature analysis and segmentation on the deduplicated data, the most appropriate compression algorithm (LZW, JPEG, MPEG-4, etc.) can be matched according to the characteristics of different types of data (such as text, images, audio and video) to achieve efficient compression; in the compression process, the initial parameters are reasonably set for the first compression, and then the parameters are dynamically adjusted according to strict compression effect evaluation and adaptive algorithm to perform compression again, and the compression effect is continuously optimized to ensure that while ensuring data integrity and quality, the storage space is minimized and the data transmission and storage efficiency are improved. This combination of hybrid compression and adaptive adjustment gives full play to the advantages of different algorithms, and can be flexibly optimized according to actual conditions to adapt to a variety of data types and complex application scenarios.

[0078] See also Figure 5 A fifth embodiment of a container storage management method in an embodiment of the present invention includes:

[0079] S501: construct a multi-level storage architecture in a container based on a preset number of levels to obtain multiple storage layers.

[0080] In this embodiment, in actual applications, the preset number of layers needs to comprehensively consider multiple factors, such as business scale, data volume, budget, etc. The multiple storage layers include a high-speed storage layer, a medium-speed storage layer, and a low-speed large-capacity storage layer. The high-speed storage layer is usually composed of a solid-state drive (SSD), which has the characteristics of fast reading and writing speed and short response time, and can meet the data access requirements with extremely high real-time requirements. The medium-speed storage layer is generally composed of an ordinary mechanical hard disk (HDD), which is suitable for storing data with medium access frequency and importance. The low-speed large-capacity storage layer often uses equipment such as a tape library, which has a slow reading and writing speed and is suitable for storing data with low access frequency and relatively low importance.

[0081] S502: According to the access frequency and importance of the compressed data and the operating status and business cycle of the container, the compressed data is divided into hot data, warm data and cold data.

[0082] In this embodiment, multiple key factors need to be considered for classifying compressed data. The first is the access frequency of the data. Through log records, data analysis tools and other means, the number of times each data block is accessed within a period of time is counted. For example, in a real-time business system, such as an online trading system, transaction record data will be frequently read and written, and its access frequency is high; while historical data backup and long-term archived data are rarely called in daily business and have a low access frequency.

[0083] The importance of data is also a key consideration, and its importance can be assessed from the impact of data on the business and the value of the data itself. Core business data such as customer information and financial data, once lost or damaged, will have a significant impact on business operations, so such data is of high importance; while some auxiliary data, such as temporarily generated cache data, is relatively less important.

[0084] At the same time, the analysis should be combined with the operating status and business cycle of the container. When the container load is high, business activities are frequent, and more hot data will be generated, because the read and write operations on the data are more frequent at this time; when the container is in a low-load state, the frequency of data access is reduced. Different businesses have different activity intensities in different time periods. Taking the e-commerce platform as an example, during promotional activities, order data and user browsing data are generated in large quantities and frequently accessed. These data are hot data; during non-promotional activities, the frequency of data access decreases. Taking these factors into consideration, compressed data is divided into hot data, warm data, and cold data. Hot data is data with high access frequency and high importance, warm data has medium access frequency and importance, and cold data is data with low access frequency and relatively low importance.

[0085] S503: Store hot data, warm data, and cold data in different storage layers respectively.

[0086] In this embodiment, the hot data is stored in the high-speed storage layer according to the compressed data classification result. Due to the fast read and write characteristics of the high-speed storage layer, the read and write requests of the hot data can be responded to quickly to meet the real-time requirements. For example, during the promotion period of the e-commerce platform, the order data is stored on the solid-state drive to ensure that the user's order placement, order query and other operations can be responded to quickly, thereby improving the user experience.

[0087] Warm data is stored in the medium-speed storage layer, which has a balanced read and write speed, cost, and capacity, enabling it to meet the access needs of warm data. The company's recent business report data is stored on ordinary mechanical hard disks, which can meet the needs of daily data analysis without incurring excessive storage costs.

[0088] Cold data is stored in the low-speed, high-capacity storage layer, taking advantage of its low cost and large capacity to store infrequently accessed data. The company's historical financial data from many years ago is stored in the tape library, which can be retrieved and restored when needed, but it does not occupy high-speed and medium-speed storage resources at ordinary times, effectively saving storage costs.

[0089] In this embodiment, data classification is performed by comprehensively integrating the access frequency and importance of compressed data, as well as factors such as the container operation status and business cycle, so that data classification is more scientific and reasonable, and different types of data (hot data, warm data, cold data) are accurately stored in storage layers that match them, giving full play to the advantages of each storage layer, which can not only meet the high real-time requirements of hot data, but also balance the cost and performance through the medium-speed storage layer, and can also use the low-speed large-capacity storage layer to reduce the storage cost of cold data.

[0090] See also Figure 6 A sixth embodiment of a container storage management method in an embodiment of the present invention includes:

[0091] S601. Collect historical monitoring indicator data of different storage layers in historical periods, analyze the collected historical monitoring indicator data using a time series analysis method, and establish a normal range value for each monitoring indicator.

[0092] In this embodiment, historical monitoring indicator data of different storage layers are widely collected over a period of time. For the high-speed storage layer, the focus is on collecting indicator data such as data read and write speed and storage capacity utilization. Because the high-speed storage layer usually processes hot data and has extremely high requirements for read and write speed, its storage capacity utilization will also change rapidly due to the frequent reading and writing of hot data. For example, during an e-commerce promotion, the data read and write speed and storage capacity utilization of the high-speed storage layer will fluctuate significantly.

[0093] The medium-speed storage layer needs to collect indicators such as data read and write speed, storage capacity utilization, and data error rate. The medium-speed storage layer stores a large amount of data with a moderate access frequency, and changes in the data error rate may gradually affect the accuracy of the business. If the company's daily business data is stored in the medium-speed storage layer, an increase in the data error rate may cause deviations in business reports.

[0094] The low-speed, high-capacity storage layer mainly collects indicators such as storage capacity, device temperature, and disk I / O queue length. This storage layer is used to store cold data, and focuses on whether its storage capacity is sufficient and whether the device is operating normally. For example, if the company's historical archive data is stored in the low-speed, high-capacity storage layer, if the storage capacity is close to saturation, it may need to be cleaned up or expanded in a timely manner.

[0095] Use time series analysis methods to conduct in-depth analysis of the collected historical monitoring indicator data. Through trend analysis, you can clearly see whether the storage capacity utilization is increasing, decreasing, or stable. For example, through trend analysis of the storage capacity utilization of the high-speed storage layer in the past year, it is found that it has gradually increased with the development of business. Through seasonal analysis, you can understand the fluctuation pattern of certain indicators in a specific time period. For example, during the peak business period every day, the data read and write speed of each storage layer will change significantly.

[0096] Based on the results of time series analysis, combined with actual business needs and storage system performance characteristics, the normal range of each monitoring indicator is established. For the data read and write speed of the high-speed storage layer, it is determined through analysis that under normal business conditions, it should be maintained in the rate range of [X]MB-[X+Y]MB per second, which is the normal range of this indicator. For the data error rate of the medium-speed storage layer, statistical analysis shows that under normal circumstances, it should not exceed [Z]%, which is its normal range.

[0097] S602: Collect real-time monitoring data of compressed data in different storage layers based on preset monitoring indicators.

[0098] In this embodiment, based on the preset monitoring indicators, a special real-time monitoring system is started to collect real-time monitoring data of compressed data in different storage layers. Using professional monitoring software and hardware equipment, data is collected from each storage layer at a set time interval (such as every 5 minutes).

[0099] For the high-speed storage layer, the storage capacity utilization data can be obtained in real time by connecting to the interface of the storage management system. At the same time, the network monitoring tool can be used to accurately monitor the data reading and writing speed, and the data transmission rate can be recorded in detail. Taking the logistics company as an example, during the peak period of logistics distribution, the high-speed storage layer stores a large amount of real-time order data, vehicle location information and other key data. These data are frequently read and written, and the system response speed is extremely high. Through the above data collection method, the reading and writing of data can be grasped in time to ensure the efficient operation of business links such as order processing and vehicle scheduling. For example, when the reading and writing speed of order data is found to be slow, it may mean that the system load is too high, and timely resource allocation or troubleshooting is required.

[0100] For the medium-speed storage layer, in addition to obtaining storage capacity utilization and data read and write speed, it is also necessary to use special data verification mechanisms and error detection tools to collect data error rates in real time. Daily business data of logistics companies, such as customer information and historical order records, are usually stored in the medium-speed storage layer. The accuracy of this data is crucial for business analysis, customer service, etc. Through data verification mechanisms and error detection tools, data errors can be discovered in a timely manner to avoid business decision errors or customer complaints caused by inaccurate data. For example, if the contact information in the customer information is incorrect, it may affect the delivery notification of the goods. By collecting data error rates in real time and processing them in a timely manner, the accuracy of the data can be effectively guaranteed.

[0101] For the low-speed, large-capacity storage layer, the built-in sensors of the storage device can read the operating status data such as device temperature and disk I / O queue length in real time, and obtain the real-time usage of storage capacity in combination with the storage management system. The historical logistics data and backup files of logistics companies are generally stored in the low-speed, large-capacity storage layer. By monitoring the device temperature and disk I / O queue length, you can understand the operating health of the device and prevent device failures. At the same time, understanding the real-time usage of storage capacity helps to plan storage resources in advance. For example, when it is found that the storage capacity utilization rate is close to 80%, you can clean up expired data in advance or consider adding storage devices to ensure the stable operation of the storage system and avoid data loss or business interruption due to insufficient storage capacity.

[0102] S603: Compare the collected real-time monitoring data with the normal range value of each monitoring indicator one by one. If the real-time monitoring data exceeds the normal range value of any monitoring indicator, generate different warning methods for different storage layers.

[0103] In this embodiment, for the high-speed storage layer, if the data read and write speed drops suddenly, it may cause the online transaction system to freeze, affecting the real-time performance of the business. At this time, a variety of emergency notification methods can be used, such as sending SMS and email notifications to administrators immediately, and displaying eye-catching red warning information on the monitoring system interface, accompanied by sound alarms, so that administrators can quickly take measures, such as troubleshooting network failures or increasing storage resources.

[0104] For the medium-speed storage layer, if the data error rate increases, although it may not have a significant impact on the business for the time being, it may cause data loss or business errors over time. At this time, relevant technical personnel can be notified through message reminders and pop-up prompts within the monitoring system to let them pay attention and conduct data error troubleshooting and repair in a timely manner.

[0105] For the low-speed, large-capacity storage layer, if the storage capacity utilization reaches 80%, although there is still some storage space, its growth trend needs to be paid attention to. Log records and regular summary reports can be used to facilitate subsequent continuous tracking and analysis of storage capacity changes and make storage plans in advance.

[0106] In this embodiment, by comprehensively collecting historical monitoring indicator data of different storage layers and using time series analysis to establish normal range values, a scientific basis is provided for abnormal judgment. Real-time monitoring data is collected based on preset indicators to ensure the pertinence and effectiveness of the data. When an abnormal situation occurs, different early warning methods are generated for different storage layers. This differentiated processing strategy can accurately notify relevant personnel and prompt them to take appropriate measures based on the characteristics of each storage layer and the degree of impact of the abnormal situation. This not only improves the accuracy and timeliness of monitoring, but also effectively ensures the stable storage of compressed data in different storage layers and the normal operation of the business, and minimizes the negative impact of abnormal situations on the storage system and business.

[0107] The container storage management method in the embodiment of the present invention is described above. The device in the embodiment of the present invention is described below. Figure 7 , the implementation of the container storage management device in the embodiment of the present invention includes:

[0108] Configuration module 701, used to obtain container configuration parameters and configure the container according to the container configuration parameters;

[0109] A deduplication module 702 is used to, when receiving a data storage instruction, use a distributed deduplication algorithm to perform deduplication processing on the data written into the container to obtain deduplicated data;

[0110] A compression module 703 is used to compress the deduplicated data using a hybrid compression algorithm, and dynamically adjust compression parameters according to the real-time compression effect of the deduplicated data through an adaptive compression algorithm to obtain compressed data;

[0111] The storage module 704 is used to construct a multi-level storage architecture in the container to obtain multiple storage layers, divide the compressed data into hot data, warm data and cold data, and store them in different storage layers respectively;

[0112] The monitoring module 705 is used to monitor the compressed data in different storage layers based on preset monitoring indicators, and generate different early warning methods for different storage layers when an abnormal situation occurs.

[0113] In this embodiment, the configuration module 701 includes: an acquisition unit 7011, used to acquire container configuration parameters, wherein the container configuration parameters include deployment and communication requirements, load balancing algorithm, network buffer size and network timeout parameters; a first configuration unit 7012, used to configure the network topology of the container according to the deployment and communication requirements of the container; a second configuration unit 7013, used to configure the load balancing strategy of the container according to the load balancing algorithm; a third configuration unit 7014, used to configure the network parameters of the container according to the size of the network buffer and the network timeout parameters.

[0114] In this embodiment, the deduplication module 702 includes: a segmentation unit 7021, which is used to segment the data written into the container into multiple smaller data blocks and calculate the hash value of each data block when a data storage instruction is received; a deduplication unit 7022, which is used to transmit the hash value of each data block to each distributed node of the container. If the hash value of the data block is found on a certain distributed node to match the hash value in the local database, the data block is determined to be duplicate data; otherwise, the data block is determined to be new data; a first integration unit 7023, which is used to integrate new data to obtain deduplicated data.

[0115] In this embodiment, the compression module 703 includes: an analysis unit 7031, which is used to perform feature analysis on the deduplicated data, and divide the deduplicated data into multiple data segments according to the analysis results, and determine the compression algorithm corresponding to each data segment according to the type of each data segment; a compression unit 7032, which is used to obtain the initial compression parameters of the compression algorithm for each data segment, and based on the initial compression parameters, use the compression algorithm to compress the data segment; if the compression ratio of the compressed data segment is lower than the preset compression threshold, use the adaptive compression algorithm to dynamically adjust the compression parameters of the compression algorithm, and based on the adjusted compression parameters, use the compression algorithm to compress the data segment again to obtain an optimized compressed data segment; a second integration unit 7033, which is used to integrate all optimized compressed data segments to obtain compressed data.

[0116] In this embodiment, the storage module 704 includes: a construction unit 7041, which is used to construct a multi-level storage architecture in the container based on a preset number of levels to obtain multiple storage layers; a classification unit 7042, which is used to divide the compressed data into hot data, warm data and cold data according to the access frequency and importance of the compressed data and the operating status and business cycle of the container; a storage unit 7043, which is used to store the hot data, the warm data and the cold data in different storage layers respectively.

[0117] In this embodiment, the monitoring module 705 includes: a collection unit 7051, which is used to collect historical monitoring indicator data of different storage layers in historical time periods, analyze the collected historical monitoring indicator data using a time series analysis method, and establish a normal range value for each monitoring indicator; a monitoring unit 7052, which is used to collect real-time monitoring data of compressed data in different storage layers based on preset monitoring indicators; an early warning unit 7053, which is used to compare the collected real-time monitoring data with the normal range value of each monitoring indicator one by one. If the real-time monitoring data exceeds the normal range value of any monitoring indicator, different early warning methods are generated for different storage layers.

[0118] In this embodiment, in terms of container configuration, by obtaining detailed parameters, it can accurately create a suitable operating environment, avoid irrational resource allocation, optimize communication capabilities, and lay the foundation for subsequent work; in terms of data storage, the distributed deduplication algorithm can process data deduplication in parallel, reduce storage volume and reduce costs; hybrid compression is combined with adaptive compression algorithms to improve compression rate and save storage and transmission time; in terms of storage architecture, the multi-level storage architecture stores data in layers based on data characteristics and business conditions, which not only meets the high-speed reading and writing requirements of hot data, but also balances cost and performance, and can adapt to business changes; in terms of monitoring and early warning, the blockchain-based mechanism ensures that monitoring indicators are true and reliable, and uses machine learning to predict anomalies by analyzing historical indicators, and generates early warnings according to severity levels to ensure stable system operation.

[0119] Figure 7 The structure of the container storage management device shown does not constitute a limitation on the container storage management device, and can implement the steps of the container storage management method provided by the above-mentioned method embodiments.

[0120] above Figure 7 The container storage management apparatus in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The container storage management device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0121] Figure 8 800 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 810 (for example, one or more processors) and memory 820, and one or more storage media 830 (for example, one or more mass storage devices) storing application programs 833 or data 832. Among them, the memory 820 and the storage medium 830 may be short-term storage or persistent storage. The program stored in the storage medium 830 may include one or more modules (not shown), and each module may include a series of instruction operations on the device 800. Furthermore, the processor 810 may be configured to communicate with the storage medium 830 to execute a series of instruction operations in the storage medium on the device 800.

[0122] The device 800 may also include one or more power supplies 840, one or more wired or wireless network interfaces 850, one or more input and output interfaces 860, and / or one or more operating systems 831, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc.

[0123] An embodiment of the present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the container storage management method.

[0124] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device, or unit can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0125] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0126] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A container storage management method, characterized in that: The container storage management method comprises: Obtaining container configuration parameters, and configuring the container according to the container configuration parameters; When a data storage instruction is received, a distributed deduplication algorithm is used to deduplicate the data written into the container to obtain deduplicated data; The deduplicated data is compressed using a hybrid compression algorithm, and compression parameters are dynamically adjusted according to the real-time compression effect of the deduplicated data using an adaptive compression algorithm to obtain compressed data; Constructing a multi-level storage architecture in the container to obtain multiple storage layers, dividing the compressed data into hot data, warm data, and cold data, and storing them in different storage layers respectively; The compressed data in different storage layers are monitored based on preset monitoring indicators. When an abnormal situation occurs, different warning methods are generated for different storage layers.

2. The container storage management method according to claim 1, characterized in that: The obtaining of container configuration parameters and configuring the container according to the container configuration parameters includes: Obtaining container configuration parameters, including deployment and communication requirements, load balancing algorithms, network buffer sizes, and network timeout parameters; Configure the network topology of the container based on the deployment and communication requirements of the container; Configure the container's load balancing strategy based on the load balancing algorithm; Configure the container's network parameters based on the network buffer size and network timeout parameters.

3. The container storage management method according to claim 1, characterized in that: When receiving the data storage instruction, the distributed deduplication algorithm is used to perform deduplication processing on the data written into the container to obtain deduplication data, including: When receiving a data storage instruction, the data written into the container is divided into multiple smaller data blocks, and a hash value of each data block is calculated; The hash value of each data block is transmitted to each distributed node of the container. If the hash value of the data block is found on a distributed node to match the hash value in the local database, the data block is determined to be duplicate data; otherwise, the data block is determined to be new data; Integrate new data to obtain deduplicated data.

4. The container storage management method according to claim 1, characterized in that: The method of compressing the deduplicated data using a hybrid compression algorithm and dynamically adjusting compression parameters according to the real-time compression effect of the deduplicated data using an adaptive compression algorithm to obtain compressed data includes: Performing feature analysis on the deduplicated data, dividing the deduplicated data into a plurality of data segments according to the analysis result, and determining a compression algorithm corresponding to each data segment according to the type of each data segment; For each data segment, the initial compression parameters of the compression algorithm are obtained, and based on the initial compression parameters, the data segment is compressed using the compression algorithm. If the compression ratio of the compressed data segment is lower than a preset compression threshold, the compression parameters of the compression algorithm are dynamically adjusted using an adaptive compression algorithm, and based on the adjusted compression parameters, the data segment is compressed again using the compression algorithm to obtain an optimized compressed data segment. All optimized compressed data segments are integrated to obtain compressed data.

5. The container storage management method according to claim 4, characterized in that: The performing feature analysis on the deduplicated data, dividing the deduplicated data into a plurality of data segments according to the analysis result, and determining a compression algorithm corresponding to each data segment according to the type of each data segment, includes: Performing feature analysis on the deduplicated data, dividing the deduplicated data into a plurality of data segments according to the analysis result, and determining the type of each data segment; If the data segment type is text data, the LZW algorithm is used to compress the corresponding data segment; If the type of the data segment is image data, the JPEG algorithm is used to compress the corresponding data segment; If the data segment type is audio or video data, the MPEG-4 algorithm is used to compress the corresponding data segment.

6. The container storage management method according to claim 1, characterized in that: The multi-level storage architecture is constructed in the container to obtain multiple storage layers, the compressed data is divided into hot data, warm data and cold data, and the data are stored in different storage layers respectively, including: A multi-level storage architecture is constructed in the container based on a preset number of levels to obtain multiple storage layers; According to the access frequency and importance of the compressed data and the operation status and business cycle of the container, the compressed data is divided into hot data, warm data and cold data; The hot data, the warm data and the cold data are stored in different storage layers respectively.

7. The container storage management method according to claim 1, characterized in that: The compressed data in different storage layers are monitored based on preset monitoring indicators. When an abnormal situation occurs, different early warning methods are generated for different storage layers, including: Collect historical monitoring indicator data of different storage layers in historical periods, analyze the collected historical monitoring indicator data using time series analysis methods, and establish the normal range value of each monitoring indicator; Collect real-time monitoring data of compressed data in different storage layers based on preset monitoring indicators; The collected real-time monitoring data is compared with the normal range value of each monitoring indicator one by one. If the real-time monitoring data exceeds the normal range value of any monitoring indicator, different warning methods are generated for different storage layers.

8. A container storage management device, characterized in that: include: A configuration module, used to obtain container configuration parameters and configure the container according to the container configuration parameters; A deduplication module is used to, when receiving a data storage instruction, use a distributed deduplication algorithm to perform deduplication processing on the data written into the container to obtain deduplication data; A compression module, used to compress the deduplicated data using a hybrid compression algorithm, and dynamically adjust compression parameters according to the real-time compression effect of the deduplicated data through an adaptive compression algorithm to obtain compressed data; A storage module is used to construct a multi-level storage architecture in the container to obtain multiple storage layers, divide the compressed data into hot data, warm data and cold data, and store them in different storage layers respectively; The monitoring module is used to monitor the compressed data in different storage layers based on preset monitoring indicators. When an abnormal situation occurs, different early warning methods are generated for different storage layers.

9. A container storage management device, characterized in that: comprising a memory and at least one processor, wherein the memory has computer-readable instructions stored therein; The at least one processor calls the computer-readable instructions in the memory to execute the steps of the container storage management method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer-readable instructions stored thereon, characterized in that: When the computer-readable instructions are executed by a processor, the steps of the container storage management method according to any one of claims 1 to 7 are implemented.

Images