A streaming data storage method incorporating adaptive compression

Abstract

The application discloses a kind of streaming data storage methods combined with adaptive compression, specifically related to data storage field, including receiving real-time data and executing first-order analysis task, first-order analysis task is based on the feature vector of data determines the segmentation mode and compression algorithm of each data segment;Data stream continues to input, and executes second-order analysis task, analyzes data stream in real-time processing pipeline, and the distribution characteristics of current stored data segment is evaluated by B+ tree index;If the evaluation of second-order analysis task meets the expectation, the reconstruction of data segment is completed, otherwise, return first-order analysis task to reevaluate.Through the feature vector of data Dynamic selection suitable compression algorithm, make the compression ratio of data and storage space form optimal matching, improve storage efficiency and reduce storage cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage, more particularly, the present application relates to a streaming data storage method combined with adaptive compression. BACKGROUND

[0002] Adaptive compression is a method of dynamically adjusting compression algorithms and strategies, selecting the most appropriate compression method in real time according to the characteristics of the data and storage requirements, automatically adjusting the compression ratio and algorithm by analyzing the redundancy, entropy, access frequency and other characteristics of the data, to optimize data storage efficiency and access speed; adaptive compression can automatically adapt to changes in data in different operating environments, improve storage efficiency and reduce storage costs.

[0003] In the adaptive compression process, segmented storage is usually performed, which relies on local decompression to improve access efficiency, but local decompression can cause fragmentation of storage space. When different data segments use different compression algorithms, or some parts of the data stream are not effectively allocated compression, it can cause uneven distribution of storage space, resulting in waste of storage space. SUMMARY

[0004] In order to overcome the above-mentioned defects of the prior art, embodiments of the present application provide a streaming data storage method combined with adaptive compression, which solves the problems raised in the above background technology by fragmentation evaluation and management and processing of data feature vectors.

[0005] To achieve the above object, the present application provides the following technical scheme: a streaming data storage method combined with adaptive compression, comprising:

[0006] Receiving real-time data and performing a first-order analysis task, the first-order analysis task determines the segmentation method and compression algorithm of each data segment based on the feature vector of the data;

[0007] Continuously inputting the data stream and performing a second-order analysis task, analyzing the data stream in the real-time processing pipeline, and evaluating the distribution characteristics of the currently stored data segment through B+ tree indexing; if the evaluation of the second-order analysis task meets the expectation, the reconstruction of the data segment is completed, otherwise the first-order analysis task is returned for reevaluation;

[0008] Evaluating the delay access degree of the reconstructed data segment, if the delay access degree exceeds the expectation, adjusting the storage priority in the preset sliding time window based on the access frequency and timeliness of the data segment through hash indexing and queue management to obtain the timeliness data segment;

[0009] Performing fragmentation evaluation on the timeliness data segment, if the fragmentation evaluation shows that there is storage fragmentation, performing fragmentation management task, otherwise the timeliness data segment is stored according to the original allocation and continues to be processed;

[0010] According to the result of the fragmentation evaluation, the data segments are divided into optimized data segments and merged data segments using hash tables and B+ tree indexes, and the data segments that are not divided are still retained as time-sensitive data segments;

[0011] The optimized data segments, merged data segments, and time-sensitive data segments are given different weights in a stream data priority calculation model, and a stream data segment adaptive mapping operation is performed.

[0012] In a preferred embodiment, the compression potential of a data segment is defined by a feature vector of the data, which includes redundancy, entropy value, and access frequency in a first-order analysis task; and the data segment is defined as a first data segment, a second data segment, and a third data segment based on the compression potential;

[0013] The selection of the first data segment includes: if the redundancy of the data segment is greater than a preset redundancy threshold, a first type of compression ratio algorithm is used to perform compression, and the first type of compression ratio algorithm includes LZ77 or Huffman encoding;

[0014] The selection of the second data segment includes: if the entropy value of the data segment is greater than a preset entropy threshold, and the storage requirement of the data segment exceeds a preset target storage requirement, a second type of compression ratio algorithm is used to perform compression; the second type of compression ratio algorithm includes Zlib; and the storage requirement of the data segment includes the number of bytes of the data and the complexity of the data;

[0015] The selection of the third data segment includes: if the data segment is not assigned to the first data segment or the second data segment within a preset time window, or the access frequency of the data segment is greater than a preset access frequency, then no compression or a third type of compression ratio algorithm is selected to perform compression; the third type of compression ratio algorithm includes Run-Length Encoding.

[0016] In a preferred embodiment, a distribution feature model is established based on the distribution characteristics of the data segment in a second-order analysis task, and the distribution feature model is constructed based on data redundancy, data access frequency, data segment storage density, and data segment location information as input variables; a distribution feature value is calculated based on the distribution feature model, and if the distribution feature value is greater than a preset distribution feature threshold, it is determined that the evaluation of the second-order analysis task meets the expectation;

[0017] The target of data redundancy in the distribution feature model is to quantify the repetitiveness of information in the data segment, and by calculating the frequency of each field in the data segment and comparing it with the frequency peak and average frequency, the redundancy of the data is measured;

[0018] The target of data access frequency in the distribution feature model is to measure the index of the access activity of the data in the storage process, and based on the hot attribute reflecting the data segment, the time series data pair is evaluated to predict whether the data segment is frequently accessed hot data;

[0019] Data segment storage density in distribution characteristic model aims to measure the space utilization of data in the storage area; by comparing the storage occupancy of each field in the data segment with the storage space of the entire data segment, the utilization efficiency of the storage space is measured to determine whether there is waste of storage resources in the data segment;

[0020] Data segment location information in the distribution characteristic model aims to evaluate the location of the data segment in the storage system and optimize the storage layout; by calculating the distance between each data segment and other data segments in the system, the efficiency of the storage location is evaluated.

[0021] In a preferred embodiment, when obtaining the time-sensitive data segment, each data segment is first evaluated, and its access frequency and time sensitivity are calculated, so that the required data segment is preferentially entered into the cache;

[0022] In the evaluation of delay access degree, each data segment is evaluated, and its access frequency and time sensitivity are calculated, so that the required data segment is preferentially entered into the cache, and then the priority of each data segment is calculated according to the product of and;

[0023] A unique storage location identifier is assigned to each data segment by hashing index, and the storage location of the data segment is established based on the storage location identifier;

[0024] According to the priority obtained by the access frequency and the time sensitivity, the storage priority of the data segment is dynamically adjusted, and the queue is constructed according to the change of the real-time data stream, and the position of the data segment in the queue is adjusted in real time;

[0025] In the sliding time window, the access of each data segment is tracked, and the new priority of the data segment is calculated and generated based on the initial priority and the time sensitivity decay factor, and the time sensitivity decay factor makes the priority of the data segment weaken over time;

[0026] Finally, the time-sensitive data segment is obtained.

[0027] In a preferred embodiment, fragmentation evaluation is performed based on the fragmentation degree and storage efficiency of the time-sensitive data segment; a fragmentation evaluation model is constructed by fragmentation evaluation; the time-sensitive data segment is divided into multiple storage blocks, and each storage block is assigned a storage state, including idle, occupied or fragmented;

[0028] Fragmentation degree is used to measure the effective utilization of storage space, which is based on the ratio of idle space to actual storage space of the data segment to represent the proportion of unused space to used space in the storage block;

[0029] Storage efficiency is used to measure the storage space occupancy rate of the storage system;

[0030] The fragmentation degree and storage efficiency of the time-sensitive data segment are taken as input variables to calculate each storage block, the fragmentation score of the entire time-sensitive data segment is calculated based on the fragmentation evaluation model, the fragmentation score is used to reflect the fragmentation degree of the time-sensitive data segment in the storage process, and the time-sensitive data segment is divided into an optimized data segment and a merged data segment based on the fragmentation score; if the fragmentation score is less than the lower limit of the preset fragmentation score threshold, it is divided into an optimized data segment; if the fragmentation score is greater than the upper limit of the preset fragmentation score threshold, it is divided into a merged data segment; and the data segment that is not divided is still retained as a time-sensitive data segment.

[0031] In a preferred embodiment, the goal of the fragmentation management task is to reduce fragmentation and improve the utilization efficiency of storage space by merging and reorganizing storage blocks, and a reorganization cost function is constructed based on this, and the resource consumption and effect of the fragmentation management task are quantified by the reorganization cost function;

[0032] In the execution of the fragmentation management task, the size and weight factor of each storage block are calculated, and the fragmentation score and storage compactness are combined to calculate the reorganization cost of each storage block; then, based on the density and fragmentation degree of the storage block, a merging strategy is selected, and according to the calculated reorganization cost, it is judged whether the storage block needs to be reconstructed, if the reorganization cost of the reconstructed storage block is lower than the preset reorganization cost threshold, the reconstruction is executed, otherwise the original storage allocation is continued.

[0033] In a preferred embodiment, it also includes a streaming data segment adaptive mapping operation:

[0034] The input variables of each data segment are calculated: a first-order analysis task is performed according to the characteristics of the streaming data segment, and the effectiveness characteristics of each data segment are supplemented and collected in the first-order analysis task, the effectiveness characteristics include access frequency, compression rate, and storage space, the effectiveness characteristics are taken as input variables, and a weighted calculation is performed to obtain the final priority of each data segment;

[0035] The priority of the data segment is calculated: the priority of the optimized data segment, the merged data segment and the time-sensitive data segment is calculated, each data segment is assigned an independent priority value, and the independent priority value affects the subsequent storage and access scheduling decision;

[0036] Priority scheduling: the priority of the optimized data segment, the merged data segment and the time-sensitive data segment is comprehensively evaluated according to the global weight coefficient, and the comprehensive priority of each data segment is obtained, the comprehensive priority determines the mapping order of the data segment in the streaming data storage, that is, the priority of the data segment in storage and access;

[0037] Performing data segment adaptive mapping operation: according to the comprehensive priority of each data segment, performing adaptive mapping operation to rearrange the storage location or scheduling access priority of the data segment in the streaming storage system.

[0038] Technical effects and advantages of the present application:

[0039] 1. By dynamically selecting the appropriate compression algorithm according to the feature vector of the data, the optimal matching between the compression ratio of the data and the storage space is formed, the storage efficiency is improved and the storage cost is reduced.

[0040] 2. Through real-time data analysis and reconstruction, the distribution characteristics of the data segment are evaluated based on B+ tree index, and the storage structure of the data segment is dynamically adjusted to ensure the flexibility in data stream processing.

[0041] 3. By combining the access frequency, timeliness and fragmentation of the data, and adjusting the priority of the data segment according to the hash index and queue management, the access speed and storage performance of the data are optimized, and the timeliness data segment is processed preferentially.

[0042] 4. Through fragmentation evaluation and management, it is beneficial to identify storage fragments and optimize timeliness data segments, avoid waste of storage space, and improve the utilization rate of storage resources.

[0043] 5. Based on the comprehensive priority of each data segment, the adaptive mapping operation is performed to dynamically adjust the storage location and access priority of the data segment, so as to ensure relatively efficient access operation in streaming data storage. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 Flow of the present application Figure 1 .

[0045] Figure 2 Flow of the present application Figure 2 . DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0047] Referring to the drawings in the specification Figures 1-2 , an adaptive compression combined streaming data storage method according to an embodiment of the present application comprises:

[0048] Receiving real-time data and performing a first-order analysis task, the first-order analysis task determines the segmentation manner and compression algorithm of each data segment based on the feature vector of the data;

[0049] Continuously inputting the data stream and performing a second-order analysis task, analyzing the data stream in the real-time processing pipeline, and evaluating the distribution characteristics of the currently stored data segment through B+ tree indexing; if the evaluation of the second-order analysis task meets the expectation, the reconstruction of the data segment is completed, otherwise, the first-order analysis task is returned for reevaluation;

[0050] Evaluating the delay access degree of the reconstructed data segment, if the delay access degree exceeds the expectation, adjusting the storage priority in the preset sliding time window through hash indexing and queue management based on the access frequency and timeliness of the data segment, and obtaining the timeliness data segment;

[0051] Performing fragmentation evaluation on the timeliness data segment, if the fragmentation evaluation shows that there is storage fragmentation, performing fragmentation management task, otherwise, the timeliness data segment is stored according to the original allocation and continues to be processed;

[0052] According to the result of fragmentation evaluation, using hash table and B+ tree index to divide it into optimized data segment and merged data segment, and the data segment not divided is still retained as timeliness data segment;

[0053] Assigning different weights to the optimized data segment, the merged data segment, and the timeliness data segment in the stream data priority calculation model, and performing stream data segment adaptive mapping operation.

[0054] Defining the compression potential of the data segment by the feature vector of the data, the feature vector of the data including redundancy, entropy value, and access frequency in the first-order analysis task; defining the data segment as first data segment, second data segment, and third data segment based on the compression potential;

[0055] The selection of the first data segment includes: if the redundancy of the data segment is greater than the preset redundancy threshold, using a type of compression ratio algorithm to perform compression, the type of compression ratio algorithm including LZ77 or Huffman encoding; the type of compression ratio algorithm refers to high compression ratio algorithm;

[0056] The selection of the second data segment includes: if the entropy value of the data segment is greater than the preset entropy threshold, and the storage requirement of the data segment exceeds the preset target storage requirement, then using a type of compression ratio algorithm to perform compression; the type of compression ratio algorithm includes Zlib, and the type of compression ratio algorithm is also a medium compression ratio algorithm; the storage requirement of the data segment includes the number of data bytes and the complexity of the data, and the complexity of the data includes but is not limited to the number of data fields and attribute types;

[0057] The selection of the third data segment includes: if the data segment is not allocated to the first data segment or the second data segment within a preset time window of the data segment, or the access frequency of the data segment is greater than a preset access frequency, then selecting a non-compression or a three-type compression ratio algorithm to perform compression; the three-type compression ratio algorithm includes Run-Length Encoding, and the three-type compression ratio algorithm is a low compression ratio algorithm.

[0058] A distribution feature model is established based on the distribution characteristics of the data segments in the second-order analysis task, and the distribution feature model is constructed based on the data redundancy R d , data access frequency A f , data segment storage density D s , and data segment position information P l as input variables; the distribution feature value is calculated based on the distribution feature model, and if the distribution feature value is greater than a preset distribution feature threshold, it is determined that the evaluation of the second-order analysis task meets the expectation;

[0059] The data redundancy R d in the distribution feature model aims to quantify the repetition of information in the data segment, and by calculating the frequency of each field in the data segment and comparing it with the frequency peak and average frequency, the redundancy of the data is measured, wherein the data segment with high redundancy represents high compression potential because it contains a large amount of repeated information data;

[0060] The data access frequency A f in the distribution feature model aims to measure the index of the access activity of the data in the storage process, and the data segment with high access frequency should be optimized and stored efficiently, and A f reflects the hot attribute of the data segment, and A f is evaluated through time series data to predict whether the data segment is frequently accessed hot data;

[0061] The data segment storage density D s in the distribution feature model aims to measure the space utilization of the data in the storage area, wherein the data segment with high storage density means that the data segment occupies less space when stored, and therefore is more efficient when stored; by comparing the storage occupation of each field in the data segment with the storage space of the entire data segment, the utilization efficiency of the storage space is measured to determine whether the data segment has storage resource waste;

[0062] The data segment position information P l in the distribution feature model aims to evaluate the position of the data segment in the storage system and optimize the storage layout; it evaluates the efficiency of the storage position by calculating the distance between each data segment and other data segments in the system; the closer the storage position is to the frequently accessed data segment, the higher the data access efficiency is, and based on this, the storage strategy of the data is optimized to reduce unnecessary access delay;

[0063] Based on data redundancy R d , data access frequency A f , data segment storage density D s , data segment location information P l When constructing the distribution feature model, it can be expressed as:

[0064]

[0065] Where f i is the frequency of occurrence of the i-th field in the data segment, f i is used to reflect the degree of repetition of each field in the data segment, f i is higher, indicating that the redundancy of the field is stronger; f max is the maximum frequency of occurrence of the field in the data segment, which is used to normalize the frequency of occurrence of the field. The purpose of this is to make the redundancy independent of the size and content of the data segment; f avg is the average frequency of the field in the data segment, indicating the median level of the frequency distribution of different fields within the data segment; n is the number of fields in the data segment; wherein By calculating the logarithmic difference between the field frequency and the average frequency, the redundancy of high-frequency fields is further emphasized. A larger deviation indicates that the field frequency is much higher than the average level, showing that the field has strong redundancy and is suitable for compression;

[0066] Where a t is the number of accesses to the data segment at time point t, indicating the access frequency of the data segment within a specific time window. The more the access times, the more frequently the data segment is accessed; A max is the maximum value of the number of accesses to the data segment within the time window; A avg is the average value of the number of accesses to the data segment within the time window, reflecting the overall level of data access, which is used to calculate the average value of the number of accesses at all time points; T represents the length of the evaluation period; wherein By calculating the logarithmic difference between the access frequency and the average frequency, the fluctuation degree of the access frequency is measured. A larger deviation indicates that the access frequency is much higher than the average level, suggesting that the data segment is hot data;

[0067] Where s i is the storage space occupied by the i-th field in the data segment, which is used to measure the size of each field in storage. The larger the storage space occupied by the field, the lower the storage density of the field; S total is the total storage space of the data segment; a higher data segment storage density D s means that the storage space is more fully utilized and the data storage efficiency is higher;

[0068] Where di Let be the storage distance of the data segment at the i-th location in the storage area, and represent the physical location of the data segment within the storage area. The smaller d... i The value indicates that the data segment is located in a nearby storage location, resulting in faster access speed; D total D represents the total storage distance of all data segment locations in the storage system, indicating the overall layout of all data segment storage locations; avg D is the average storage distance of data segment locations in the storage system. avg This reflects the average layout level of the storage area; smaller d i This means that the data segment is stored in a higher priority location, which means it can respond to access requests more quickly;

[0069] Based on the above formula, the calculated data redundancy R d Data access frequency A f Data segment storage density D s Data segment location information P l The values ​​are calculated and weighted to obtain the distribution characteristic model M of the data segment. d ;

[0070] M d =w1R d +w2A f +w3D s +w4P l

[0071] Where w1, w2, w3, and w4 correspond to the data redundancy R, respectively. d Data access frequency A f Data segment storage density D s Data segment location information P l The weights can be adjusted according to the actual application scenario.

[0072] When acquiring time-sensitive data segments, first evaluate each data segment D. i Calculate its access frequency Freq(D) i ) and timeliness St(D) i This allows the required data segments to be cached first.

[0073] In evaluating delayed access, each data segment D is evaluated. i Calculate its access frequency Freq(D) i ) and timeliness St(D) i This allows the required data segments to be cached first, and then, according to Freq(D)... i ) and St(D i The product of ) is used to calculate the priority P(D) of each data segment. i ); P(Di ) = Freq(D i ) x St(D i ) ;

[0074] A unique storage location identifier is assigned to each data segment D i by the hash index H(D i ), and the storage location of the data segment is established based on the storage location identifier; H(D i ) is expressed as: H(D i ) = hash(ID(D i )) ;

[0075] The storage priority of the data segment D i is dynamically adjusted according to P(D i ) obtained from the access frequency Freq(D i ) and the timeliness St(D i ), and the queue is constructed according to the change of the real-time data stream, and the position of the data segment D i in the queue is adjusted in real time;

[0076] Within the sliding time window, the access of each data segment D i is tracked, and the new priority New Priority(D i ) of the data segment D i is calculated and generated based on the initial priority Queue(D i ) and the timeliness decay factor a; the timeliness decay factor a causes the priority of the data segment D i to weaken over time; T now is the current time, T last (D i ) is the last access time of the data segment D i , and based on T now -T last (D i ), the time difference from the last access to the current time is expressed, and the expression of New Priority(D i ) is:

[0077]

[0078] The timeliness data segment is finally obtained;

[0079] The access frequency Freq(D i ) refers to the number of times a data segment D i is accessed within a given time window, which is used to evaluate the activity level of the data segment D i in the streaming data; for each data segment D i , the number of accesses within the sliding time window is recorded, and the time window is set as Tw Then the access frequency Freq(D) i ) is represented as:

[0080]

[0081] Among them access(D) i ,t) represents data segment D i The number of times it is accessed at time t;

[0082] Timeliness St(D) i ) used to measure data segment D i The degree to which it remains valid or useful in the current streaming data system; timeliness is directly related to the expiration level of the data, access patterns, and storage requirements; St(D i In the function expression, by considering the data segment D i T last (D i ) and when T now A threshold θ is set to define its validity period; timeliness St(D) i ) is represented as:

[0083]

[0084] Where α is the time-dependent decay factor, T last (D i ) is data segment D i Last access time, T now It represents the current time and indicates the expiration level of the data segment; a smaller α value will decay more slowly.

[0085] Fragmentation assessment is performed based on the fragmentation degree and storage efficiency of time-sensitive data segments; a fragmentation assessment model is constructed based on the fragmentation assessment; and time-sensitive data segments are split into multiple storage blocks B1, B2, ..., B n Each storage block B i Assign a storage state, which may be free, occupied, or fragmented; the size of the storage block |B i The decision is made by the storage system based on the capacity of the storage units and the data compression algorithm.

[0086] Fragmentation level F is used to measure the effective utilization of storage space. It is based on the ratio between free space and the actual storage space of data segments to represent storage block B. i The ratio of unused space to used space; where the larger the fragmentation value F, the more serious the waste of data storage space and the lower the system's storage efficiency. The formula for fragmentation F is:

[0087]

[0088] wherein S unused,i represents the unused storage space in the i-th storage block B i ; S total,i represents the total storage space of the i-th storage block B i ; R i represents the density factor of the storage block B i , the density factor is used to measure the tightness of the storage block usage;

[0089] The storage efficiency is used to measure the indicator of the storage system on the storage space occupancy rate, the storage efficiency is represented by the storage tightness p, the higher the storage tightness p, the higher the storage space utilization rate, and the storage efficiency is represented as:

[0090]

[0091] wherein S used,i represents the used storage space in the storage block B i ; E i represents the weighting factor used to consider the storage timeliness and access frequency of the data segment; the larger the value of the storage tightness p, the more efficient the storage;

[0092] The fragmentation degree of the timeliness data segment and the storage efficiency are taken as the input variables to calculate each storage block B i , the fragmentation score F score of the entire timeliness data segment is calculated based on the fragmentation evaluation model, the fragmentation score F score is used to reflect the fragmentation degree of the timeliness data segment in the storage process, and the timeliness data segment is divided into an optimized data segment and a merged data segment based on the fragmentation score F score ; if the fragmentation score is less than the preset lower limit of the fragmentation score threshold, it is divided into an optimized data segment, because the optimized data segment has efficiently utilized the storage space, and is suitable for further optimizing the storage layout; if the fragmentation score is greater than the preset upper limit of the fragmentation score threshold, it is divided into a merged data segment, and the merged data segment needs to be merged to reduce fragmentation and improve storage efficiency; the data segment that is not divided is still kept as a timeliness data segment, and continues to maintain a high priority for storage and access; the fragmentation score F score is represented as:

[0093]

[0094] wherein F i represents the fragmentation degree of the i-th storage block; a i is the weight factor of the fragmented block; p j is the storage tightness of the j-th data segment; b j is the weight factor of the storage tightness; c is the number of storage blocks; v is the number of data segments.

[0095] The goal of the fragmentation management task is to reduce fragmentation and improve the utilization efficiency of storage space by merging and reorganizing storage blocks, and a reorganization cost function is constructed based on this to quantify the resource consumption and effect of the fragmentation management task;

[0096] In the execution of the fragmentation management task, the size and the weight factor W i of each storage block B i are calculated, and the reorganization cost C reorg of each storage block is calculated by combining the fragmentation score F score and the storage compactness ρ j ; then a merging strategy is selected based on the density and fragmentation of the storage blocks, and the storage blocks with lower density and higher fragmentation are preferentially merged to reduce fragmentation and improve storage density, according to the calculated reorganization cost C reorg , it is judged whether the storage block needs to be reconstructed, if the reorganization cost C reorg of the reconstructed storage block is lower than the preset reorganization cost threshold, the reconstruction is executed, otherwise the original storage allocation is continued and the processing is continued;

[0097]

[0098] The reorganization cost C reorg is used to represent the computing resources required to execute the fragmentation management task; is the size of the storage block B i , representing the physical space of the storage block; W i is the weight factor of each storage block B i ; F score is the fragmentation score, indicating the severity of fragmentation, if the fragmentation score increases, the demand for merging and reconstruction increases, so its influence in the reorganization cost will be increased; λ is the weight coefficient, λ is used to adjust the influence of the fragmentation score on the reorganization cost, the larger λ indicates the more serious the fragmentation problem, the higher the reorganization cost of the system; ζ is the influence coefficient of the storage compactness on the reorganization cost; the storage compactness ρ j is used to measure the occupancy efficiency of data in storage, when ρ j is higher, it means that the storage space is better utilized, and the reorganization cost is lower.

[0099] Also includes a streaming data segment adaptive mapping operation:

[0100] The input variables of each data segment are calculated: a first-order analysis task is performed according to the characteristics of the streaming data segment, and the validity characteristics of each data segment are supplemented in the first-order analysis task, including access frequency, compression rate, and storage space, which are used as input variables to perform weighted calculation to obtain the final priority of each data segment;

[0101] The priority of the data segment is calculated: the priority of the optimized data segment, the merged data segment and the time-sensitive data segment is calculated, and each data segment is assigned an independent priority value, which affects the subsequent storage and access scheduling decisions based on the independent priority value;

[0102] Priority scheduling: the priority of the optimized data segment, the merged data segment and the time-sensitive data segment is evaluated according to the global weight coefficient, and the comprehensive priority of each data segment is obtained, which determines the mapping order of the data segment in the streaming data storage, that is, the priority of the data segment in storage and access;

[0103] Performing adaptive mapping operation of data segment: according to the comprehensive priority of each data segment, the adaptive mapping operation is performed to rearrange the storage location or schedule the access priority of the data segment in the streaming storage system, so as to ensure the efficiency and real-time performance of data access;

[0104] Further description is needed for the above embodiments:

[0105] The priority P of the optimized data segment opt is the data segment after fragmentation management and reconstruction, which has low delay access and high access frequency; according to the effectiveness of access frequency, compression rate and storage space, the priority is calculated;

[0106] P opt =α·F opt +β·C opt -γ·S opt

[0107] Wherein F opt is the access frequency of the optimized data segment; C opt is the compression rate of the optimized data segment, the higher the compression rate, the greater the value, which means the more advantages in the embodiment; S opt is the storage space usage of the optimized data segment; α, β, γ are weight coefficients, representing the influence of access frequency, compression rate and storage space on priority respectively;

[0108] The priority P of the merged data segment merge is a data segment merged by multiple smaller data segments, and its priority is determined by multiple factors, which is calculated by the historical access frequency, delay access degree and time span of the merged data segment;

[0109] P merge = δ · H merge + λ · D merge - θ · T merge

[0110] wherein H merge is the historical access frequency of the merged data segment; D merge is the delay access degree of the merged data segment; T merge is the time span of the merged data segment; δ, λ, θ are weight coefficients respectively representing the influence of the historical access frequency, the delay access degree and the time span on the priority of the merged data segment;

[0111] The priority P timeliness of the time-sensitive data segment refers to a data segment with time-sensitive requirements, which usually needs a higher access priority when stored. The priority P timeliness of the time-sensitive data segment is calculated by combining the time-sensitive requirements of storage and the access frequency of real-time data stream;

[0112] P timeliness = μ · F timeliness + ν · A timeliness - ξ · S timeliness

[0113] wherein F timeliness is the real-time access frequency of the time-sensitive data segment; A timeliness is the time-sensitive requirement of the time-sensitive data segment; S timeliness is the storage resource occupation of the time-sensitive data segment; μ, ν, ξ are weight coefficients representing the influence of the access frequency, the time-sensitive requirement and the storage resource occupation on the priority of the time-sensitive data segment;

[0114] The generation target of the priority calculation model of the stream data segment is to calculate the priority of each data segment according to its characteristics, so as to determine how to adaptively map the storage and access order of the data segment;

[0115] P total = α1 · P opt + α2 · P merge + α3 · P timeliness

[0116] wherein P total is the comprehensive priority of each data segment; P opt is the priority of the optimized data segment; P merge is the priority of the merged data segment; P timeliness is the priority of the time-sensitive data segment; α1, α2, α3 are global weight coefficients for adjusting the influence of the optimized data segment, the merged data segment and the time-sensitive data segment in the final priority calculation.

[0117] The above merely provides the preferred embodiments of the application, and is not intended to limit the application. Any modification, equivalent replacement, improvement, and the like made within the spirit and principle of the application shall fall within the protection scope of the application.

Claims

1. A streaming data storage method combining adaptive compression, characterized in that, include: It receives real-time data and performs a first-order analysis task, which determines the segmentation method and compression algorithm for each data segment based on the feature vector of the data. The system continuously inputs data streams and performs second-order analysis tasks, analyzing the data streams in the real-time processing pipeline and evaluating the distribution characteristics of currently stored data segments using B+ tree indexes. If the evaluation of the second-order analysis task meets expectations, the data segment reconstruction is completed; otherwise, the first-order analysis task is returned for re-evaluation. The latency of the reconstructed data segment is evaluated. If the latency exceeds the expectation, the storage priority is adjusted within a preset sliding time window based on the access frequency and timeliness of the data segment, using hash index and queue management, to obtain a time-sensitive data segment. Perform a fragmentation assessment on time-sensitive data segments. If the fragmentation assessment shows that storage fragmentation exists, perform a fragmentation management task; otherwise, store the time-sensitive data segments according to the original allocation and continue processing. Based on the fragmentation assessment results, hash tables and B+ tree indexes are used to divide the data into optimized data segments and merged data segments. Data segments that are not divided are retained as time-sensitive data segments. Fragmentation assessment is performed based on the fragmentation level and storage efficiency of time-sensitive data segments; A fragmentation assessment model is constructed through fragmentation assessment; time-sensitive data segments are split into multiple storage blocks, and each storage block is assigned a storage state, which includes idle, occupied, or fragmented. Fragmentation is used to measure the effective utilization of storage space. It is based on the ratio between free space and the actual storage space of data segments to represent the proportion of unused space to used space in a storage block. Storage efficiency is a metric used to measure the utilization of storage space by a storage system; Each storage block is calculated using the fragmentation degree and storage efficiency of the time-sensitive data segment as input variables. Based on the fragmentation evaluation model, the fragmentation score of the entire time-sensitive data segment is calculated. The fragmentation score is used to reflect the degree of fragmentation generated by the time-sensitive data segment during storage. Based on the fragmentation score, the time-sensitive data segment is divided into optimized data segment and merged data segment. If the fragmentation score is less than the preset lower limit of the fragmentation score threshold, it is divided into optimized data segment. If the fragmentation score exceeds the preset fragmentation score threshold, it will be divided into merged data segments; data segments that are not divided will remain as time-sensitive data segments. Different weights are assigned to optimized data segments, merged data segments, and time-sensitive data segments in the streaming data priority calculation model, and an adaptive mapping operation for streaming data segments is performed.

2. The streaming data storage method combining adaptive compression according to claim 1, characterized in that: The compression potential of a data segment is defined by the feature vector of the data. In a first-order analysis task, the feature vector of the data includes redundancy, entropy, and access frequency. Based on the compression potential, the data segments are defined as the first data segment, the second data segment, and the third data segment. The selection of the first data segment includes: if the redundancy of the data segment is greater than the preset redundancy threshold, then a compression ratio algorithm is used to perform compression, and the compression ratio algorithm includes LZ77 or Huffman coding. The selection of the second data segment includes: if the entropy value of the data segment is greater than the preset entropy threshold, and the storage requirement of the data segment exceeds the preset target storage requirement, then a type II compression ratio algorithm is used to perform compression; the type II compression ratio algorithm includes Zlib; the storage requirement of the data segment includes the number of bytes of data and the complexity of the data; The selection of the third data segment includes: if the data segment is not allocated to the first or second data segment within the preset time window, or if the access frequency of the data segment is greater than the preset access frequency, then either no compression or compression is performed using one of the three compression ratio algorithms; the three compression ratio algorithms include Run-Length Encoding.

3. The streaming data storage method combining adaptive compression according to claim 2, characterized in that: A distribution feature model is established based on the data segment distribution characteristics in the second-order analysis task. The distribution feature model is constructed based on data redundancy, data access frequency, data segment storage density, and data segment location information as input variables. The distribution feature value is calculated based on the distribution feature model. If the distribution feature value is greater than the preset distribution feature threshold, it is determined that the evaluation of the second-order analysis task meets the expectations. In distribution feature models, the goal of data redundancy is to quantify the repetition of information in a data segment. This is achieved by calculating the frequency of each field in the data segment and comparing it with the peak and average frequencies to measure the degree of data redundancy. The goal of the data access frequency distribution feature model is to measure the access activity of data during storage. Based on the hot spot attributes that reflect the data segment, it is evaluated through time series data to predict whether the data segment is frequently accessed hot data. The goal of data segment storage density in the distribution feature model is to measure the space utilization of data in the storage area; by comparing the storage occupancy of each field in the data segment with the storage space of the entire data segment, the efficiency of storage space utilization is measured, and it is determined whether there is any waste of storage resources in the data segment; The goal of data segment location information in the distribution feature model is to evaluate the location of data segments in the storage system and optimize the storage layout; It evaluates the efficiency of storage locations by calculating the distance between each data segment and other data segments in the system.

4. The streaming data storage method combining adaptive compression according to claim 3, characterized in that: When retrieving time-sensitive data segments, each data segment is first evaluated, its access frequency and timeliness are calculated, and the required data segments are prioritized for cache entry. In evaluating delayed access, each data segment is evaluated, its access frequency and timeliness are calculated, and the required data segments are prioritized to enter the cache. Then, the priority of each data segment is calculated based on the product of access frequency and timeliness. Each data segment is assigned a unique storage location identifier using a hash index, and the storage location of the data segment is established based on the storage location identifier. Based on the priority obtained from access frequency and timeliness, the storage priority of data segments is dynamically adjusted, and a queue is built according to the changes in real-time data stream, and the position of data segments in the queue is adjusted in real time. Within the sliding time window, access to each data segment is tracked, and a new priority for the data segment is calculated and generated based on the initial priority and the time-related decay factor. The time-related decay factor weakens the priority of the data segment over time. The final result is a time-sensitive data segment.

5. A streaming data storage method combining adaptive compression according to claim 4, characterized in that: The goal of fragmentation management is to reduce fragmentation and improve storage space utilization efficiency by merging and reorganizing storage blocks. Based on this, a reorganization cost function is constructed to quantify the resource consumption and effectiveness of fragmentation management. In the fragmentation management task, the reorganization cost of each storage block is calculated by calculating the size and weight factor of each storage block, and combining the fragmentation score and storage compactness. Then, a merging strategy is selected based on the density and fragmentation of the storage blocks. Based on the calculated reorganization cost, it is determined whether the storage block needs to be reorganized. If the reorganization cost of reorganizing the storage block is lower than the preset reorganization cost threshold, the reorganization is performed; otherwise, the storage is allocated as originally and processing continues.

6. A streaming data storage method combining adaptive compression according to claim 5, characterized in that: It also includes adaptive mapping operations for streaming data segments: The input variables for each data segment are calculated: a first-order analysis task is performed based on the characteristics of the streaming data segment. In the first-order analysis task, the validity characteristics of each data segment are collected. The validity characteristics include access frequency, compression ratio, and storage space. The validity characteristics are used as input variables, and a weighted calculation is performed to obtain the final priority of each data segment. Calculate the priority of data segments: Calculate the priority of optimized data segments, merged data segments, and time-sensitive data segments, assign an independent priority value to each data segment, and influence subsequent storage and access scheduling decisions based on the independent priority value; Priority scheduling: The priority of data segments, data segments to be merged, and time-sensitive data segments are comprehensively evaluated based on global weight coefficients to obtain the comprehensive priority of each data segment. The comprehensive priority determines the mapping order of data segments in streaming data storage, that is, the priority of data segments being stored and accessed. Perform adaptive mapping operation on data segments: Based on the overall priority of each data segment, perform adaptive mapping operation to rearrange the storage location or schedule access priority of the data segments in the streaming storage system.

Images