Data compression storage method and device and electronic equipment

Abstract

The invention provides a data compression storage method and device and electronic equipment, and relates to the technical field of distributed storage and data compression.The method comprises the steps that a target data stream is obtained, feature vectors and data fingerprints of data blocks in the target data stream are extracted, and description information of the data blocks is generated; determining a first compression strategy corresponding to the data block according to a preset mapping relation between the cluster center vector and the compression strategy and the similarity between the feature vector and the cluster center vector; compressing the data block by adopting the first compression strategy; and storing the compressed data blocks, and constructing an index relationship of the data blocks, the index relationship comprising a first compression strategy corresponding to the data blocks. And determining a first compression strategy corresponding to the data block according to a pre-obtained mapping relationship between the cluster center vector and the compression strategy. The first compression strategy is a strategy corresponding to the characteristics of the data block, so that the compression efficiency can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed storage and data compression, and particularly relates to a data compression storage method and device and electronic equipment. BACKGROUND

[0002] With the rapid development of Internet of Things technology, the amount of data is growing exponentially. Distributed architecture, which has the advantages of strong scalability and high fault tolerance, has become the mainstream choice for mass data storage. In order to cope with the contradiction between the expansion of data size and the limitation of storage resources, data compression has become a key technology to reduce storage costs and improve transmission efficiency.

[0003] In the current common compression strategy, a single dimension is usually used to determine the compression strategy, for example, fixed-length data is classified and compressed. This method is prone to poor data adaptation to compression, resulting in low compression efficiency. SUMMARY

[0004] The embodiments of the present application provide a data compression storage method, device and electronic equipment to solve the problem of poor data adaptation to compression, resulting in low compression efficiency.

[0005] To solve the above technical problems, the present application is implemented as follows:

[0006] In a first aspect, the embodiments of the present application provide a data compression storage method, which comprises:

[0007] Obtaining a target data stream, extracting a feature vector and a data fingerprint of a data block in the target data stream, and generating description information of the data block, the description information comprising the feature vector and the data fingerprint;

[0008] According to a preset mapping relationship between a cluster center vector and a compression strategy, and a similarity between the feature vector and the cluster center vector, a first compression strategy corresponding to the data block is determined;

[0009] The data block is compressed by using the first compression strategy;

[0010] The compressed data block is stored, and an index relationship of the data block is constructed, the index relationship comprising the first compression strategy corresponding to the data block.

[0011] In a second aspect, the embodiments of the present application provide a data compression storage device, which comprises:

[0012] A first obtaining module is configured to obtain a target data stream, extract a feature vector and a data fingerprint of a data block in the target data stream, and generate description information of the data block, the description information comprising the feature vector and the data fingerprint;

[0013] The first determining module is configured to determine the first compression strategy corresponding to the data block according to a preset mapping relationship between the cluster center vector and the compression strategy, and a similarity between the feature vector and the cluster center vector.

[0014] The compression module is configured to compress the data block by using the first compression strategy.

[0015] The storage module is configured to store the compressed data block, and construct an index relationship of the data block, wherein the index relationship includes the first compression strategy corresponding to the data block.

[0016] In a third aspect, an electronic device is provided, which includes a processor, a memory, and a program stored in the memory and executable on the processor, and when the program is executed by the processor, the steps of the data compression storage method in the first aspect are implemented.

[0017] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data compression storage method in the first aspect are implemented.

[0018] In a fifth aspect, a computer program product is provided, which includes computer instructions, and when the computer instructions are executed by a processor, the steps of the data compression storage method in the first aspect are implemented.

[0019] In the embodiments of the present application, the feature vector and the data fingerprint of the data block are obtained, and the first compression strategy corresponding to the data block is determined according to the mapping relationship between the cluster center vector and the compression strategy obtained in advance, and the similarity between the feature vector of the data block and the cluster center vector. Since the first compression strategy is the strategy corresponding to the feature of the data block, the compression efficiency can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0021] Figure 1 is one of the flowcharts of the data compression storage method provided by the embodiments of the present application;

[0022] Figure 2 is a feature analysis schematic diagram of a data block provided by the embodiments of the present application;

[0023] Figure 3 is a data index schematic diagram provided by an embodiment of the application;

[0024] Figure 4 is a data clustering flowchart provided by an embodiment of the application;

[0025] Figure 5 is a data compression storage method flowchart two provided by an embodiment of the application;

[0026] Figure 6 is a data compression storage device structural schematic diagram provided by an embodiment of the application;

[0027] Figure 7 is an electronic device structural schematic diagram provided by an embodiment of the application. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0029] At present, when data is compressed, a single parameter such as data length is usually used as the core basis for selecting a compression algorithm by adopting a fixed compression strategy. The target data length is determined by dividing a fixed threshold interval, without considering key internal characteristics such as data repetition mode and entropy distribution, so that data with large differences in characteristics but the same length are compressed by using the same compression algorithm, which cannot achieve precise adaptation and is prone to problems such as low compression ratio or poor efficiency. Since the preset threshold interval and algorithm rules are relied on, there is a lack of dynamic adjustment capability. When data characteristics change, the threshold and algorithm rules need to be manually adjusted, which cannot respond to data dynamic changes online and is difficult to adapt to real-time processing requirements of massive heterogeneous data streams in a distributed environment.

[0030] The embodiments of the present application provide a data compression storage method, device and electronic equipment to solve the problem of poor adaptability of data to compression, thereby causing low compression efficiency.

[0031] Referring to Figure 1 , Figure 1 is a data compression storage method flowchart provided by an embodiment of the application, as shown in Figure 1 , the method comprises the following steps:

[0032] Step 101: Obtain the target data stream, extract the feature vectors and data fingerprints of the data blocks in the target data stream, and generate description information of the data blocks, wherein the description information includes the feature vectors and the data fingerprints;

[0033] Step 102: Determine the first compression strategy corresponding to the data block based on the preset mapping relationship between the cluster center vector of the data feature cluster and the compression strategy, and the similarity between the feature vector and the cluster center vector;

[0034] Step 103: Compress the data block using the first compression strategy;

[0035] Step 104: Store the compressed data blocks and construct an index relationship for the data blocks, the index relationship including the first compression strategy corresponding to the data blocks.

[0036] The target data stream can be received from other devices or stored locally. For example, in a distributed storage system (including compression nodes, access nodes, a global policy optimization center, and a central routing decision-maker), the access nodes receive massive raw data streams from different applications or data sources and perform real-time and lightweight preliminary parsing on the incoming raw data blocks to extract low-level multidimensional feature vectors that can directly reflect the inherent structure and statistical regularities of the data, i.e., the feature vectors of the data blocks.

[0037] The feature vector may include, but is not limited to, the entropy distribution characteristics of the data block itself, the frequency and length distribution characteristics of repetitive patterns in the byte or symbol sequence within the data block, the dynamic range and numerical distribution statistical moment characteristics of the numerical data within the data block, and the locality and correlation characteristics exhibited by the data block sequence in the time dimension.

[0038] After obtaining the feature vectors of the data blocks, a globally unique feature fingerprint identifier is generated for each data block. This feature fingerprint identifier is then bound to the corresponding feature vector to form a concise data feature description file, i.e., description information. The above process can be found in [reference needed]. Figure 2 As shown.

[0039] Specifically, let the original feature vector extracted from the data block be... The first data block Each component represents a low-order statistic such as local entropy or repetition pattern frequency. The access node maintains a moving average for each type of statistic throughout the batch statistical history. and sliding standard deviation For the current data block, the first The original feature components are adaptively standardized to obtain normalized feature components. The normalization formula is wherein is a very small positive number for numerical stability, and The update of adopts exponential moving average to balance the historical stability and response speed.

[0040] wherein, the update rule is implemented by accumulating statistics online within a time window, such normalization has two purposes: one is to eliminate the dimension and range difference of different statistics, which is convenient for real similarity comparison between vectors; the other is to control the response smoothness when the burst distribution changes through the sliding parameter. and the sliding parameter controls the response smoothness when the burst distribution changes. The normalized vector is mapped to a compact and persistent feature fingerprint identifier FID, so as to realize the routing exchange and deduplication judgment in the low bandwidth scene.

[0041] The data fingerprint generation can adopt the combination of projection quantization and irreversible hash to balance the similarity maintenance and uniqueness, and the data fingerprint generation formula is:

[0042]

[0043] In the above formula, is a projection matrix, and its row vector represents a set of locally sensitive directions learned, is a bias vector, which is used to eliminate the projection bias, is a quantization operation that projects real values into small integers to reduce the expression length and enhance the tolerance to approximate similarity, round is the operation of rounding to an integer, concat is to splice the quantized components into a binary string in a fixed order for hash function processing, SHA256 is an irreversible hash, which is used to generate a fixed-length fingerprint string.

[0044] Through the data fingerprint, fast hash-level deduplication detection and compact key as clustering index can be realized. The parameters in the two formulas are pre-trained to obtain the initial and at system startup, and are updated regularly by the central strategy update center at runtime to avoid offset accumulation.

[0045] The access node writes the original data block and its data feature description file (i.e. description information) (containing the normalized vector and the data fingerprint FID) into the local temporary buffer queue at the same time, and records the timestamp and source identifier, so as to realize the time sequence related locality judgment by the central router.

[0046] In this way, by reducing the expression dimension and quantizing the hash, the efficiency of the routing decision is improved, and the network transmission load is reduced while maintaining the distinguishability. The access node will wait for the central routing decision maker to perform similarity matching and routing instruction issuing based on the global cluster head mapping table, and then enter the next step.

[0047] According to the global cluster head mapping table, a preset mapping relationship between the cluster head vector and the compression strategy can be obtained.

[0048] In some embodiments, the above mapping table can record the cluster head vectors of all active data feature clusters in the current system and the compression strategy identifiers corresponding to each data feature cluster.

[0049] The central routing decision maker obtains the feature vector in the description information of the data block, and performs similarity calculation and matching between the feature vector and each cluster head vector in the global data feature cluster head mapping table. According to a predetermined similarity threshold (i.e., a preset threshold), the data block is determined to belong to a high-similarity feature cluster in the mapping table, or to belong to a new feature cluster, thereby requiring the creation of a new cluster head. The correspondence between the cluster head vector in the mapping table and the compression strategy can be determined according to historical compression feedback results, that is, when a corresponding compression strategy is used for compression for a cluster head vector, the compression efficiency is the highest and the compression index is the best.

[0050] In some embodiments, for the data block determined to be similar to the existing feature cluster according to the similarity, the central routing decision maker generates an intelligent routing instruction containing the target compression node address and the specified compression algorithm instruction according to the compression strategy identifier mapped by the feature cluster; for the data block determined to be dissimilar to the existing feature cluster, i.e., belonging to a new feature cluster, the central routing decision maker will start a compression strategy exploration process, randomly specify a candidate compression algorithm, and also generate a corresponding routing instruction. Finally, the central routing decision maker sends the generated routing instruction back to the corresponding access node.

[0051] According to the matched cluster head vector or the created cluster head vector, a corresponding first compression strategy is determined, and the target compression node uses the first compression strategy to compress the data block. The data block can be compressed according to the compression strategy corresponding to the cluster head vector, thereby improving the efficiency of data block compression.

[0052] The compressed data block is stored, and the corresponding compression strategy of the data block is constructed, so as to facilitate the rapid determination of the compression strategy according to the data block.

[0053] The data block compressed by the target compression node is distributed to multiple physical storage nodes in the distributed storage cluster for persistent storage. A fault-tolerant mechanism suitable for compressed data is used during storage to improve the reliability of the data.

[0054] In addition, a central metadata management service creates a metadata index for each stored compressed data block, which contains at least the following key information: the feature fingerprint identifier of the data block, the data feature cluster identifier it belongs to, the compression strategy and parameter version used, the physical location pointer of the compressed data in the distributed storage system, and the logical ownership information of the data block. The above process can be seen from Figure 3 as shown.

[0055] This set of metadata index system enables subsequent data query and read requests to be quickly routed to the correct storage location, and after reading the data, it can perform accurate reverse decompression operation according to the compression algorithm information recorded in the metadata to completely restore the original data content.

[0056] It should be noted that the execution subject of the above method can be a device corresponding to the distributed storage system. Among them, the compression node, the access node, the global strategy optimization center, and the central routing decision maker all belong to different functional components in the device corresponding to the distributed storage system. Among them:

[0057] The access node is the front-end data entrance, responsible for data reception, multi-dimensional feature analysis, feature fingerprint generation, and belongs to the "data preprocessing and forwarding node";

[0058] The central routing decision maker is the global scheduling core, responsible for dynamic clustering, similarity matching, and generating routing instructions, and belongs to the "decision and routing node";

[0059] The target compression node is a computing node, responsible for performing parallel compression, local parameter fine-tuning, and generating compression feedback, and belongs to the "data compression and local learning node";

[0060] The global strategy optimization center is the strategy training and updating core, responsible for building an effect knowledge base, training reinforcement learning models, and updating optimal compression strategies, and belongs to the "global optimization and strategy management node".

[0061] In addition, the above compression node, access node, global strategy optimization center, and central routing decision maker can also be multiple different devices, each device being used to perform a corresponding function.

[0062] In the embodiment of the application, the feature vector and the data fingerprint of the data block are obtained, and the first compression strategy corresponding to the data block is determined according to the mapping relationship between the cluster center vector and the compression strategy obtained in advance and the similarity between the feature vector of the data block and the cluster center vector. Since the first compression strategy is a strategy corresponding to the features of the data block, the compression efficiency can be improved.

[0063] Optionally, the first compression strategy corresponding to the data block is determined according to a preset mapping relationship between the cluster center vector and the compression strategy and a similarity between the feature vector and the cluster center vector.

[0064] In the pre-acquired mapping table, a preset mapping relationship between a cluster center vector of a data feature cluster and a compression strategy is acquired;

[0065] A similarity between the feature vector of the data block and the cluster center vector is determined.

[0066] The first compression strategy corresponding to the data block is determined based on the similarity.

[0067] In a case where the similarity between the feature vector and the first cluster center vector is higher than or equal to a preset threshold, the first compression strategy is a compression strategy corresponding to the first cluster center vector, and the first cluster center vector is any cluster center vector in the cluster center vector.

[0068] In a case where the similarity between the feature vector and the cluster center vector is lower than the preset threshold, the first compression strategy is any compression strategy in candidate compression strategies.

[0069] A central routing decision maker independent of data storage nodes can continuously collect data feature description files from temporary buffer queues of various access nodes, maintain and dynamically update a global data feature cluster center mapping table, and the mapping table includes a preset mapping relationship between a cluster center vector and a compression strategy.

[0070] The feature vector of the data block is compared with each cluster center vector in the mapping table to determine a similarity.

[0071] If the similarity between the feature vector of the data block and the first cluster center vector is higher than or equal to a preset threshold, it indicates that the feature vector is similar to the first cluster center vector, and the same compression strategy can be used. Therefore, the first compression strategy corresponding to the first cluster center vector is taken as the compression strategy of the data block.

[0072] If the similarity between the feature vector of the data block and any cluster center vector in the mapping table is lower than the preset threshold, a new feature cluster is created for the data block, a new cluster center is created, and a first compression strategy is randomly or specifically selected from candidate compression strategies. The above process can be referred to as shown in FIG. 2. Figure 4

[0073] Specifically, in some embodiments, the central routing decision maker performs dynamic clustering and routing decision on the data feature description files transmitted by the access nodes. First, a normalized feature vector (i.e., a feature vector of a data block) is compared with a cluster center vector set of global feature clusters. ​​Similarity is calculated to determine the most matching feature cluster. Each cluster center represents the weighted mean vector of all current members of the cluster, and the similarity is calculated using the improved cosine similarity index:

[0074]

[0075] where is the vector dot product measure of directional similarity, and is the Euclidean norm for normalization, measures the absolute deviation between vectors, is a balance factor to control the weight of direction and amplitude.

[0076] The above formula can not only capture the statistical feature consistency in the direction of the vector, but also consider the influence of amplitude deviation on the compression strategy selection, so that the high similarity determination is not only based on the consistency of the trend, but also considers the proximity of the absolute statistical value, thereby improving the consistency of the compression strategy of the data block in the feature cluster.

[0077] After the calculation is completed, the central routing decision maker will belong to the similarity the cluster with the largest similarity, and trigger the cluster center update of the cluster.

[0078] When is lower than the set threshold , trigger the new cluster creation mechanism, initialize as the new cluster center , and also complete the cluster center update, and the update method is to adjust the cluster center vector using a dynamic learning rate η. At the same time, the initial compression strategy is randomly specified to explore optimization.

[0079] After determining the cluster belonging, the central routing decision maker generates intelligent routing instructions according to the compression strategy identifier p_{k^} corresponding to the cluster c{k^} in the cluster center mapping table. The routing instruction includes the identification of the target compression node , the type of compression algorithm and the optional parameter fine-tuning range , and the routing algorithm realizes efficient transmission and processing of data blocks by issuing instructions to access nodes and carrying metadata index association table.

[0080] To further improve the accuracy of strategy selection, the central routing decision maker adjusts the cluster center vector update formula at each cluster center update using a dynamic learning rate , where is the cluster Current number of members. Through cluster center vector update, the feature information of newly arrived data blocks can be smoothly introduced to avoid the cluster center from being sharply deviated due to sudden abnormal data, while keeping the representativeness of the cluster center to historical features. Among them, The setting allows the control of the influence weight of new data on the cluster center, so as to realize online clustering convergence and stability in dynamic data flow. After the cluster center matching and updating are completed, the access node receives the routing instruction and forwards the data blocks according to the allocated target compression node and algorithm type.

[0081] According to the similarity between the feature vector and the cluster center vector, the first compression vector is determined, which can determine the matching compression strategy according to the characteristics of the data block, or try a new compression strategy, which can improve the compression efficiency, and explore a more suitable compression strategy.

[0082] By using an improved cosine similarity formula to calculate the similarity between the feature vector of new data and each cluster center vector, the formula considers the vector direction similarity and amplitude deviation through the balance factor λ, which not only determines the feature trend consistency, but also considers the absolute statistical value proximity; when the similarity is lower than the threshold, a new cluster is created and an initial compression strategy is randomly specified, and when the similarity is higher than the threshold, it is classified into the corresponding existing cluster, and the cluster center vector is updated with a dynamic learning rate to avoid the cluster center from being sharply deviated due to abnormal data, and to realize efficient clustering of dynamic data flow.

[0083] Optionally, the compression of the data block using the first compression strategy comprises:

[0084] According to the first compression strategy, a parameter vector is constructed, and the parameter vector comprises at least one of a dictionary construction rule, a probability model weight or a predictor coefficient;

[0085] According to the association between the parameter vector and the loss function, the parameter vector is updated, and the loss function is used to measure the weighted deviation of compression ratio and compression speed;

[0086] Based on the updated parameter vector, the data block is compressed, and a feedback report is generated, the feedback report comprising at least one of a compression efficiency index and a feature response vector, the feature response vector being used to describe the trend of the distribution characteristics of the data block.

[0087] According to the first compression strategy, a parameter vector is constructed, and based on the association between the parameter vector and the loss function, the parameter vector is updated, so that the parameter vector is suitable for the characteristics of the current data block, and the compression quality index is improved.

[0088] In some embodiments, the access node, upon receiving the routing instruction from the central routing decider, routes the data blocks in the temporary buffer queue to the designated target compression nodes according to the instruction. Among them, the target compression node is a physical or logical node responsible for performing data compression tasks in the distributed cluster, and each target compression node is pre-installed with a plurality of different basic compression algorithm libraries, which cover different types of classic compression technologies such as dictionary encoding, entropy encoding, and prediction encoding. According to the compression algorithm instruction specified in the routing instruction, the target compression node calls the corresponding basic compression algorithm to perform compression calculation on the allocated data blocks.

[0089] In the above process, the target compression node will simultaneously start a local fine-tuning learning process, that is, for the current batch of data blocks with similar characteristics being processed, the parameter vector of the called compression algorithm is finely adjusted, for example, the construction strategy of the dictionary is adjusted, the probability model is updated, or the coefficients of the predictor are optimized, so that the algorithm can better adapt to the specific statistical characteristics of the current small batch of data, thereby obtaining additional compression efficiency improvement based on the general algorithm.

[0090] Specifically, after receiving the data blocks forwarded by the access node, the target compression node performs a strategy-driven parallel compression process. Each target compression node performs local feature learning while compressing according to the specified compression algorithm, and updates the parameter vector to adapt to the data statistical structure of the current feature cluster.

[0091] First, data blocks from the same feature cluster are grouped into the current batch The target compression node constructs the corresponding parameter vector according to the compression strategy specified in the routing instruction The parameter vector includes dictionary construction rules, probability model weights, or predictor coefficients. By quickly analyzing the statistical rules of the batch , the vector parameters are updated to make the compression behavior closer to the distribution characteristics of the current feature cluster.

[0092] To realize online updating, the target compression node uses an adaptive fine-tuning algorithm to perform gradient-based updating of the parameter vector based on compression feedback. Define the loss function of the current batch compression performance The loss function is used to measure the weighted deviation value of compression ratio and compression speed. According to the correlation between the parameter vector and the loss function, the local parameter update rule is constructed:

[0093]

[0094] Among them, is the local learning rate, is the parameter gradient calculated according to the compression feedback of the current batch.

[0095] This update rule enables the parameter vector to quickly adjust according to the statistical structure of the current batch of data, and in the case of low compression cost, to improve the compression ratio and keep the speed stable.

[0096] After updating the parameter vector, the target compression node performs an actual compression operation on the data block and records the compression ratio and the compression speed , and generates a feedback report. The feedback report includes the compressed data block and can also include compression efficiency indicators such as the compression ratio and the compression speed, compression efficiency, etc. It can also include the local feature response vector based on the current batch , which is used to describe the local distribution trend of the current batch of data, including the size of the updated dictionary, the entropy order change of the probability model, or the statistical moments of the prediction error.

[0097] The final target compression node encapsulates the feedback report of this compression task as a triple and returns it to the global strategy optimization center, which provides the basis for state transition evidence for strategy evolution and provides the basis for strategy optimization.

[0098] In this embodiment, the access node distributes the data block to the target compression node according to the routing instructions, and the node is built-in with a dictionary encoding, entropy encoding, prediction encoding, and other types of basic compression algorithm library; while executing the specified compression algorithm, it adjusts the algorithm internal parameters through an adaptive fine-tuning algorithm (gradient-based update based on compression performance loss function) for the similar feature data of the current batch, so that the algorithm adapts to the statistical characteristics of the data, improves the compression ratio while ensuring the compression speed, and records the fine-tuned parameters and compression indicators to form a local effect feedback report.

[0099] Optionally, the method further comprises:

[0100] using a pre-trained evaluation network model to evaluate the first compression strategy according to the feedback report, to obtain an evaluation result;

[0101] in a case where it is determined according to the evaluation result that a second compression strategy is superior to the first compression strategy, updating the first compression strategy in the mapping relationship to the second compression strategy.

[0102] By continuously collecting state, action, and reward sequence data of the data block, an evaluation network model for evaluating the compression strategy is trained. Based on the trained evaluation network model, it can more and more accurately predict which compression strategy can achieve the maximum expected return for any given data feature cluster. That is, when the data feature cluster is input, the recommended compression strategy can be output. In some embodiments, multiple compression strategies can be output in order of return size, and the corresponding scores or recommendation degrees can be output.

[0103] When the recommendation degree of the second compression strategy of the feature cluster is significantly higher than the first compression strategy already in the current mapping table according to the evaluation network model, that is, the difference between the recommendation degree of the second compression strategy and the recommendation degree of the first compression strategy is greater than a preset threshold, the global strategy optimization center will issue an update instruction to the central routing decision maker, requiring it to update the first compression strategy identifier corresponding to the feature cluster in the global data feature cluster center mapping table to the identifier of the second compression strategy, thereby realizing the online, dynamic and autonomous evolution of the compression strategy. The matching degree of the compression strategy corresponding to the cluster center vector in the mapping table can be improved, thereby improving the compression effect.

[0104] Optionally, the method further comprises:

[0105] The data feature cluster is taken as state data, the compression strategy and the parameter vector are taken as action data, and the score of the compression efficiency index is taken as reward data. The state data, the action data and the reward data are input into the evaluation network model constructed in advance, the evaluation network model is trained, and the trained evaluation network model is obtained.

[0106] After completing the compression task, the target compression node uniformly sends the generated feedback report to the global strategy optimization center. The global strategy optimization center collects effect data of different feature cluster data using different compression strategies (also referred to as compression algorithms) and their fine-tuning from all compression nodes, and constructs a large compression strategy effect knowledge base.

[0107] The global strategy optimization center adopts a reinforcement learning framework, takes each data feature cluster as a state data, takes the selected compression strategy and its fine-tuning parameters as action data, and takes the comprehensive score of the compression efficiency index (such as compression ratio and compression speed) as reward data. Through the continuously collected state, action and reward sequence data, and inputting into the evaluation network model for training.

[0108] In specific implementation, the sequence data composed of the state of each data feature cluster in the distributed storage system (that is, the normalized multi-dimensional feature vector corresponding to the data feature cluster, the cluster stability index and other information reflecting the statistical characteristics of the cluster data), the compression algorithm and fine-tuning parameters (action) selected for the cluster, and the comprehensive score of the compression efficiency corresponding to the compression strategy (reward, integrating the compression ratio and compression speed index) are input. The network model outputs the recommended compression strategy prediction result, specifically including the compression algorithm type suitable for the cluster and the optimal fine-tuning parameter suggestion.

[0109] In this embodiment, the data feature cluster is regarded as a state, the compression algorithm and the fine-tuning parameter are regarded as an action, and the compression efficiency comprehensive score is regarded as a reward. The strategy evaluation network is trained. When the recommended strategy is significantly better than the current strategy, an update instruction is issued to the central routing decision maker to dynamically update the optimal compression strategy identifier in the global cluster center mapping table, so as to realize online autonomous evolution of the compression strategy.

[0110] Optionally, after the data block is compressed by using the first compression strategy, the method further comprises:

[0111] According to the first feature of the data block before compression and the second feature of the data block after compression, a compression quality deviation of the data block is determined, and the compression quality deviation is used to represent the deviation degree of the second feature relative to the first feature;

[0112] Based on the compression quality deviation and a historical compression quality deviation, a feedback update factor is generated;

[0113] According to the update factor, the compression strategy in the mapping table is updated.

[0114] The first feature of the data block before compression and the second feature of the data block after compression are obtained, and a compression quality deviation is obtained according to the first feature and the second feature. And according to the compression quality deviation and a historical compression quality deviation, a feedback update factor is generated. The historical compression quality deviation can be a historical average deviation or a deviation feature value, such as a deviation median, a maximum deviation, a minimum deviation, etc.

[0115] Specifically, a measurable compression quality feedback closed loop is constructed, so that each compression output can be quantified as a positive or negative signal to the strategy system. These signals are written back to the strategy update module in real time, and the effectiveness of the feedback directly depends on The accuracy of the judgment of the current cluster stability, and the quality index after compression will in turn affect the cluster center update and the subsequent compression strategy adjustment.

[0116] In order to make the feedback have interpretability and convergence, a compression quality deviation degree is introduced to reflect the deviation degree of the second feature after compression and the first feature before compression in the key dimension. The definition is as follows:

[0117]

[0118] Wherein, is the original feature of the data block before compression (i.e. the first feature), is the reconstructed feature (i.e. the second feature) obtained by applying the compression strategy and then decoding after compression; is a compression rate adjustment function, which reflects the joint mapping of cluster stability and feature similarity, is directly related to

[0119] The compression quality deviation metric formula can realize adaptive normalization, when is higher, it represents that the system considers that the cluster structure is stable and the current feature deviates from the cluster center less, at this time, the denominator increases to make decrease, thereby avoiding “over punishment”; when is lower, it represents that the system is in an unstable state, the denominator decreases to make increase, so as to strengthen the punishment of the poor compression strategy. In this way, the feedback is consistent with the system confidence, and the coherence of the evaluation logic is realized.

[0120] In order to enable the feedback to directly affect the strategy trajectory, a feedback update factor is introduced to drive the fine adjustment of the strategy number, so that the strategy sequence has directional convergence. The definition is as follows:

[0121]

[0122] wherein, is an update step factor, is the average quality reference of the current cluster in the historical window, which is stored in the strategy mapping table. Since accumulates the value from the history, the update rule forms a time-dependent self-balancing structure.

[0123] When exceeds , the system triggers the strategy adjustment; when is lower than , the strategy remains or is fine-tuned towards a more aggressive compression direction. In combination with , the central router updates the compression strategy number to , so that the strategy direction moves towards the area with lower , forming a closed-loop learning mechanism.

[0124] The updated strategy number is written back to the compression strategy mapping table in reverse, and affects the interpretation of next time, so that the system enters the global performance optimization stage.

[0125] In the embodiment, the compression quality deviation is defined by the Euclidean distance between the original feature and the reconstructed feature after compression decoding, and adaptive normalization is realized by combining the compression rate adjustment function, so that the feedback punishment degree is consistent with the system confidence in the cluster structure; the feedback update factor is introduced to drive the fine adjustment of the strategy number, and the updated strategy is written back to the strategy mapping table in reverse.

[0126] ​​Optionally, the method further comprises:

[0127] obtaining a cluster stability index corresponding to the cluster center vector in the mapping table, the cluster stability index being used to represent the concentration degree of the feature set contained in the cluster around the cluster center vector;

[0128] constructing a compression rate adjustment function, the compression rate adjustment function being used to represent the association between the cluster stability index and the compression rate adjustment coefficient;

[0129] The determination of the compression quality deviation of the data block according to the first feature before compression and the second feature after compression comprises:

[0130] According to the first feature before compression, the second feature after compression, and the compression rate adjustment function, the compression quality deviation of the data block is determined.

[0131] obtaining a cluster center of a data feature cluster in a mapping table , input features , similarity , and the cluster updated in the above steps . On the basis of these pre-variables, a quantifiable and iteratively convergent control model is established, so that the compression strategy not only depends on the stability of the cluster structure, but also reflects the dynamic fluctuations of the feature distribution.

[0132] In order to make the compression strategy have acute response to the internal dispersion degree of the current cluster, a cluster stability index is introduced, which is used to measure the concentration degree of the features in the cluster around the cluster center . The stability is defined as the reciprocal of the average value of the weighted distance in the cluster, so that the cluster is more concentrated and the stability is larger. The stability function is defined as:

[0133]

[0134] wherein, represents all feature sets contained in the cluster . The function realizes smoothness through the denominator structure, so that it maintains high sensitivity when the cluster internal fluctuation is small, and does not produce excessive amplification effect on noise samples, is the Euclidean distance, the more stable the position of the cluster center converges, the lower the average value of the distance in the cluster, and the higher the stability, so as to promote the adoption of a higher compression rate strategy, so that the overall compression is more robust.

[0135] Based on , an adaptive compression rate adjustment function is constructed The target is to map the cluster stability index to the corresponding compression rate adjustment coefficient, so that the compression rate tends to increase when the cluster is stable, and automatically decreases when the cluster is unstable, so as to reduce information loss. The adaptive compression rate adjustment function is defined as follows:

[0136]

[0137] wherein, is a similarity quantification function, reflecting the matching degree between the new input feature and the cluster center ; reflecting the convergence of the cluster as a whole; and is a system adjustment coefficient, used to maintain the balance between the global stability state of the cluster and the local similarity of the input. The parameter describes the local geometric structure through the combination of the numerator and the denominator , so that the local matching and the global stability are coupled, so that the compression strategy has multi-scale discrimination ability.

[0138] The determination method of the compression quality deviation metric may be determined according to the formula in the above embodiment.

[0139] The central routing decision maker continuously adjusts the corresponding compression strategy of the cluster according to the output result of . When rises, the compression rate increasing mechanism is triggered to make the compression more aggressive; when falls, the conservative compression mechanism is triggered to preferentially maintain information integrity. Inside the decision maker, the selection logic directly maps to the strategy number , and maintains the compression strategy trajectory of different clusters in the historical time dimension through the strategy update table, so that the strategy update is interpretable and traceable.

[0140] After the adjustment is completed, the data transmission quality feedback can be performed to provide an input basis for performance evaluation.

[0141] In this embodiment, the reciprocal of the weighted distance average of the feature vectors in the cluster and the cluster center is defined, the sensitivity and noise immunity are balanced by smoothing the denominator structure, and the degree of concentration in the feature set in the cluster is accurately described; based on the cluster stability and the similarity , the adaptive compression rate adjustment function is constructed to realize the dynamic regulation of increasing the compression rate when the cluster is stable and decreasing the compression rate when the cluster is unstable, and to preferentially guarantee information integrity.

[0142] The flow of the embodiment of the present application can be seen fromFigure 5 The embodiment can improve data management efficiency, solve the problems of low data retrieval and recovery efficiency after data storage, poor platform performance stability due to limited storage resources caused by data scale expansion, and the like. The embodiment can be widely adapted to massive data storage scenarios of communication operators, Internet platforms, and Internet of Things enterprises.

[0143] The embodiment of the application can build a hierarchical adaptation mechanism by multi-dimensional feature analysis, dynamic clustering based on improved cosine similarity, taking into account feature direction and amplitude deviation, dynamically updating cluster centers, and local parameter fine-tuning of target compression nodes, without relying on fixed training data distribution, so as to match the optimal compression strategy for different feature cluster data, effectively reduce the compression performance instability problem caused by dynamic heterogeneous data, and further improve the compression efficiency through parallel compression and strategy optimization. Specifically, the embodiment includes the following aspects:

[0144] First, a reinforcement learning framework is built based on a global strategy optimization center. In the framework, the system defines each data feature cluster as a state, defines the selected compression algorithm and the corresponding fine-tuning parameters as an action, and defines the comprehensive score of compression ratio and compression speed as a reward. The strategy evaluation network is trained based on the continuously collected state-action-reward sequence data, and finally the online dynamic update of the compression strategy is realized.

[0145] Second, an iterative loop is built in combination with compression quality deviation feedback. The system quantitatively calculates the deviation between the original feature and the reconstructed feature after compression decoding, and simultaneously adjusts the feedback punishment degree in association with the cluster stability index - when the cluster stability is high, the deviation punishment is reduced; when the cluster stability is low, the deviation punishment is enhanced. Through this dynamic feedback mechanism, the compression strategy is continuously evolved.

[0146] Third, the data storage and access performance is improved through central metadata index management. The central metadata management service creates a complete index record for each compressed data block, which includes the feature fingerprint identifier of the data block, the feature cluster identifier to which the data block belongs, the compression algorithm and parameter version used, the physical location pointer in the distributed storage cluster, and the like. Based on this index system, the compressed data can be quickly routed to the target storage location for retrieval, and the precise reverse decompression can be realized according to the compression algorithm information in the index, so as to finally improve the data storage reliability, and realize the dual optimization of long-term compression effect and data access efficiency.

[0147] Reference Figure 6 , Figure 6 is a structural schematic diagram of a data compression storage device provided by the embodiment of the application, as Figure 6 shown, the data compression storage device 600 includes:

[0148] The first acquisition module 601 is configured to acquire a target data stream, extract a feature vector and a data fingerprint of a data block in the target data stream, and generate description information of the data block, wherein the description information comprises the feature vector and the data fingerprint.

[0149] The first determination module 602 is configured to determine a first compression strategy corresponding to the data block according to a preset mapping relationship between a cluster center vector and a compression strategy and a similarity between the feature vector and the cluster center vector.

[0150] The compression module 603 is configured to compress the data block by using the first compression strategy.

[0151] The storage module 604 is configured to store the compressed data block and construct an index relationship of the data block, wherein the index relationship comprises the first compression strategy corresponding to the data block.

[0152] Optionally, the determination module is specifically configured to:

[0153] acquire, in a pre-acquired mapping table, a preset mapping relationship between a cluster center vector of a data feature cluster and a compression strategy;

[0154] determine a similarity between the feature vector of the data block and the cluster center vector;

[0155] determine the first compression strategy corresponding to the data block based on the similarity;

[0156] wherein, in a case where the similarity between the feature vector and a first cluster center vector is higher than or equal to a preset threshold, the first compression strategy is a compression strategy corresponding to the first cluster center vector, and the first cluster center vector is any cluster center vector in the cluster center vector;

[0157] in a case where the similarity between the feature vector and the cluster center vector is lower than the preset threshold, the first compression strategy is any compression strategy in candidate compression strategies.

[0158] Optionally, the compression module is specifically configured to:

[0159] construct a parameter vector according to the first compression strategy, wherein the parameter vector comprises at least one of a dictionary construction rule, a probability model weight or a predictor coefficient;

[0160] update the parameter vector according to an association relationship between the parameter vector and a loss function, wherein the loss function is used to measure a weighted deviation of a compression ratio and a compression speed;

[0161] compress the data block based on the updated parameter vector, and generate a feedback report, the feedback report comprising at least one of a compression efficiency indicator and a feature response vector used to describe a trend of distribution characteristics of the data block.

[0162] Optionally, the apparatus further comprises:

[0163] an evaluation module configured to evaluate the first compression strategy according to the feedback report by using a pre-trained evaluation network model, to obtain an evaluation result;

[0164] a first updating module configured to update the first compression strategy in the mapping relationship to a second compression strategy in a case where it is determined that the second compression strategy is superior to the first compression strategy according to the evaluation result.

[0165] Optionally, the apparatus further comprises:

[0166] a training module configured to input data feature clusters as state data, compression strategies and parameter vectors as action data, and scores of compression efficiency indicators as reward data into the pre-constructed evaluation network model, train the evaluation network model, and obtain the trained evaluation network model.

[0167] Optionally, the apparatus further comprises:

[0168] a second determining module configured to determine a compression quality deviation of the data block according to first features before compression and second features after compression of the data block, the compression quality deviation being used to represent a degree of deviation of the second features relative to the first features;

[0169] a generating module configured to generate a feedback update factor based on the compression quality deviation and a historical compression quality deviation;

[0170] a second updating module configured to update a compression strategy in the mapping table according to the update factor.

[0171] Optionally, the apparatus further comprises:

[0172] a second obtaining module configured to obtain a cluster stability indicator corresponding to a cluster center vector in the mapping table, the cluster stability indicator being used to represent a degree of concentration of a feature set contained in a cluster around the cluster center vector;

[0173] a constructing module configured to construct a compression rate adjustment function, the compression rate adjustment function being used to represent an association between the cluster stability indicator and a compression rate adjustment coefficient;

[0174] The second determining module is specifically configured to:

[0175] determining the compression quality deviation of the data block according to the first feature of the data block before compression, the second feature of the data block after compression, and the compression rate adjustment function.

[0176] The data compression storage apparatus can implement Figure 1 the various processes implemented in the method embodiments and achieve the same technical effects. To avoid repetition, the various processes will not be described again here.

[0177] As Figure 7 shown, the embodiments of the present application also provide an electronic device 500, which includes a processor 501, a memory 502, and a program stored in the memory 502 and executable on the processor 501. The program, when executed by the processor 501, implements the various processes of the above-described data compression storage method embodiments and achieves the same technical effects. To avoid repetition, the various processes will not be described again here.

[0178] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The computer program, when executed by a processor, implements the various processes of the above-described data compression storage method embodiments and achieves the same technical effects. To avoid repetition, the various processes will not be described again here. The computer readable storage medium may, for example, be a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk.

[0179] The embodiments of the present application also provide a computer program product, which includes computer instructions. The computer instructions, when executed by a processor, implement the various processes of the above-described method embodiments and achieve the same technical effects. To avoid repetition, the various processes will not be described again here. Figure 1

[0180] It should be noted that, in this document, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles, or apparatuses that include a series of elements not only include those elements, but also include other elements that are not explicitly listed, or other elements inherent to such processes, methods, articles, or apparatuses. Without more limitations, the element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0181] ​Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned example methods can be realized by means of software and a necessary general hardware platform, and of course, can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product in essence or in the form of a part that contributes to the prior art. The computer software product is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk) and includes a plurality of instructions for causing a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in the various embodiments of the present application.

[0182] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-mentioned specific embodiments, and the above-mentioned specific embodiments are only illustrative and not restrictive. Those skilled in the art can make many forms without departing from the scope of the present application under the inspiration of the present application, and all of them belong to the protection of the present application.

Claims

1. A data compression and storage method, characterized in that, include: Acquire a target data stream, extract feature vectors and data fingerprints of data blocks in the target data stream, and generate description information of the data blocks, wherein the description information includes the feature vectors and the data fingerprints; Based on the preset mapping relationship between the cluster center vector and the compression strategy, and the similarity between the feature vector and the cluster center vector, the first compression strategy corresponding to the data block is determined; The data block is compressed using the first compression strategy; The compressed data blocks are stored, and an index relationship of the data blocks is constructed, the index relationship including the first compression strategy corresponding to the data blocks.

2. The method according to claim 1, characterized in that, The step of determining the first compression strategy corresponding to the data block based on the preset mapping relationship between the cluster center vector and the compression strategy, and the similarity between the feature vector and the cluster center vector, includes: In the pre-acquired mapping table, obtain the preset mapping relationship between the cluster center vector of the data feature cluster and the compression strategy; Determine the similarity between the feature vector of the data block and the cluster center vector; Based on the similarity, a first compression strategy corresponding to the data block is determined; Wherein, when the similarity between the feature vector and the first cluster center vector is higher than or equal to a preset threshold, the first compression strategy is the compression strategy corresponding to the first cluster center vector, and the first cluster center vector is any one of the cluster center vectors; If the similarity between the feature vector and the cluster center vector is lower than the preset threshold, the first compression strategy is any compression strategy among the candidate compression strategies.

3. The method according to claim 1 or 2, characterized in that, The step of compressing the data block using the first compression strategy includes: A parameter vector is constructed according to the first compression strategy, and the parameter vector includes at least one of dictionary construction rules, probability model weights, or predictor coefficients. The parameter vector is updated based on the correlation between the parameter vector and the loss function, wherein the loss function is used to measure the weighted deviation between the compression ratio and the compression speed. The data block is compressed based on the updated parameter vector, and a feedback report is generated. The feedback report includes at least one of a compression efficiency index and a feature response vector, which is used to describe the trend of the distribution characteristics of the data block.

4. The method according to claim 3, characterized in that, The method further includes: A pre-trained evaluation network model is used to evaluate the first compression strategy based on the feedback report, and the evaluation result is obtained. If, based on the evaluation results, it is determined that the second compression strategy is superior to the first compression strategy, the first compression strategy in the mapping relationship is updated to the second compression strategy.

5. The method according to claim 4, characterized in that, The method further includes: The data feature clusters are used as state data, the compression strategy and parameter vectors are used as action data, and the compression efficiency index score is used as reward data. The state data, the action data and the reward data are input into the pre-built evaluation network model, and the evaluation network model is trained to obtain the trained evaluation network model.

6. The method according to claim 2, characterized in that, After compressing the data block using the first compression strategy, the method further includes: Based on the first feature of the data block before compression and the second feature after compression, the compression quality deviation of the data block is determined, and the compression quality deviation is used to characterize the degree of offset of the second feature relative to the first feature; Based on the compression quality deviation and historical compression quality deviation, a feedback update factor is generated; The compression strategy in the mapping table is updated according to the update factor.

7. The method according to claim 6, characterized in that, The method further includes: Obtain the cluster stability index corresponding to the cluster center vector in the mapping table. The cluster stability index is used to characterize the degree of concentration of the feature set contained in the cluster around the cluster center vector. A compression ratio adjustment function is constructed, which is used to characterize the correlation between the cluster stability index and the compression ratio adjustment coefficient; The step of determining the compression quality deviation of the data block based on the first characteristic of the data block before compression and the second characteristic after compression includes: The compression quality deviation of the data block is determined based on the first characteristic of the data block before compression, the second characteristic after compression, and the compression ratio adjustment function.

8. A data compression and storage device, characterized in that, include: The first acquisition module is used to acquire a target data stream, extract feature vectors and data fingerprints of data blocks in the target data stream, and generate description information of the data blocks, wherein the description information includes the feature vectors and the data fingerprints; The first determining module is used to determine the first compression strategy corresponding to the data block based on the preset mapping relationship between the cluster center vector and the compression strategy, and the similarity between the feature vector and the cluster center vector. A compression module is used to compress the data block using the first compression strategy; A storage module is used to store the compressed data blocks and construct an index relationship of the data blocks, the index relationship including a first compression strategy corresponding to the data blocks.

9. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the data compression storage method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data compression storage method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the data compression and storage method as described in any one of claims 1 to 7.

Images