Predicting compression ratio of data with compressible decision

Abstract

The data compression analyzer can quickly make a binary decision to compress or not compress an input data block, or it can use a relatively slow neural network to predict the compression ratio of the data block through a regression model. The concentration (CV) and the number of zero-value (NZ) symbols are calculated based on the sum of squared frequencies from an unsorted symbol frequency table. A compression decision is made quickly when their product, CV*NZ, exceeds a level threshold THH. During training, CV*NZ is plotted as a function of the compression ratio C% for multiple training data blocks. Different THH test values ​​are applied to this plot to determine the true positive rate and false positive rate, and are plotted as a receiver operating characteristic (ROC) curve. The point on the ROC curve with the highest Yoden index is selected as the optimal THH for future binary decisions.

Description

Technical Field

[0001] This invention relates to neural network accelerators, and more particularly to compressed prediction and decision-making using neural networks. Background Technology

[0002] Neural networks use weights on the nodes within the network. The network's topology and connections can be defined by feature maps. For deep neural networks and convolutional neural networks, these weights and feature maps can be quite large.

[0003] The weights and feature maps of a neural network can be compressed before storage or transmission. Figure 1 shows a neural network whose weights and feature maps have been compressed before storage or transmission. The neural network 104 can be a graphics processing unit (GPU) or a dedicated neural network processor. The neural network 104 may have been trained, and its weights and feature maps have been tuned to optimize for a specific problem or dataset. These weights and feature maps can be backed up or stored in memory 102.

[0004] However, the memory 102 can be located remotely from the neural network 104, for example, when the neural network 104 is on a portable device and the memory 102 is cloud storage. The connection between the neural network 104 and the memory 102 may be bandwidth-limited. The compressor 100 can use lossless compression to compress the weights and feature maps from the neural network 104 and send the compressed data to the memory 102. This can reduce the amount of memory required for storage in the memory 102 and also reduce the bandwidth consumed by the transmission.

[0005] The compressed weights and feature maps stored in memory 102 can be transmitted via a network link to a local device including compressor 100, which decompresses the weights and feature maps and loads them into neural network 104 to configure neural network 104 for a specific processing task.

[0006] Some data may be so random or unstructured that compression is ineffective. It's best to avoid compressing such data, as the compressed result may be larger than the original. Entropy and other methods can be used to predict the compression ratio of the dataset. The predicted compression ratio can be used to reserve storage space in memory 102 and to determine when to compress or not compress the data.

[0007] However, entropy-based compression ratio prediction can be difficult to implement in hardware because entropy calculations typically require logarithmic operations. The predicted compression ratio can also be highly data-dependent.

[0008] In some applications, memory 102 may not reserve storage space based on the predicted compression ratio. In this case, the actual compression ratio is not the true measure; only the decision of whether to compress or not compress the data is truly important. The predicted compression ratio can be compared to a threshold to determine when to compress the data, but apart from the comparison to the threshold, the value of the predicted compression ratio is not truly important.

[0009] There is a desire for a simpler method to make compression / non-compression decisions without calculating the predicted compression ratio. There is a desire to use neural networks to make compression / non-compression decisions without predicting the compression ratio. In other applications that use predicted compression ratios, there is a desire to use trained neural networks to calculate the predicted compression ratio. There is a desire for a neural network that can both predict the compression ratio and generate compression decisions faster. There is a desire for a compression predictor that does not use entropy calculations or logarithmic operations. Attached Figure Description

[0010] Figure 1 shows a neural network whose weights and feature maps are compressed before storage or transmission.

[0011] Figure 2 This is a block diagram of a compressed predictive neural network.

[0012] Figure 3 The regression model is shown in more detail.

[0013] Figure 4 The classification model is shown in more detail.

[0014] Figure 5 It is a graph in which the classification model uses the Receiver Operating Characteristic (ROC) curve to set a compressible decision threshold.

[0015] Figure 6 Display the confusion matrix.

[0016] Figure 7 This is a graph of the training data for prediction based on the number of zeros (NZ).

[0017] Figure 8 This is a graph of the training data used for prediction based on Concentration Value (CV).

[0018] Figure 9 It is a graph of the training data for prediction based on concentration (CV) multiplied by the number of zero values ​​(NZ).

[0019] Figure 10 yes Figure 9 An enlarged view of the curve graph.

[0020] Figure 11A-11B This is a flowchart of the operation of a neural network used for compressed prediction and decision-making.

[0021] Figure 12 This is a flowchart of optimizing the horizontal threshold THH using the receiver operating characteristic (ROC) curve.

[0022] Figure 13 This is a graph of several receiver operating characteristic (ROC) curves.

[0023] Figures 14A-14B The generation of the ROC curve is highlighted, and the best THH with the highest Yorkden index is selected. Detailed Implementation

[0024] This invention relates to improvements to compressed predictive neural networks. The following description is intended to enable those skilled in the art to make and use the invention in the context of specific applications and requirements. Various modifications to the preferred embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. Therefore, the invention is not intended to be limited to the specific embodiments shown and described, but is to be given the broadest scope consistent with the principles and novel features disclosed herein.

[0025] Figure 2 This is a block diagram of a compression prediction neural network. Input data 12 can be one or more blocks of data to be transmitted over a network with limited bandwidth. The goal is to compress input data 12. When it is necessary to estimate the size of the compressed data, for example, to conserve bandwidth or storage space, input data 12 is fed into a regression model 20 implemented on the neural network. Regression model 20 outputs a compression ratio prediction 14. The compression ratio can be expressed as a percentage of the original file size, where 100% represents an incompressible file, and lower percentages represent more compressed files. For example, a compression ratio of 20% means that the compressed file is only one-fifth the size of the original data file.

[0026] When the compression percentage prediction is not needed, such as when memory bandwidth is not conserved based on the predicted compressed file size, regression model 20 is not activated. Instead, input data 14 is sent to classification model 30. Classification model 30 is also implemented by a neural network. Classification model 30 classifies the input data as compressible or incompressible. A binary judgment may be easier and faster to obtain than predicting the compression ratio because there are only two possible outputs: yes or no. Classification model 30 outputs a compression decision 16, which can be a single binary bit or a Boolean value.

[0027] Figure 3The regression model is shown in more detail. The input data block is applied to counter 22, which counts the number of symbols appearing for each possible symbol value in the input data to obtain a count or frequency for each symbol. Counter 22 can generate a histogram or table, such as a symbol frequency table. This symbol frequency table is indexed by symbol value (X) and is not sorted according to frequency or count (Y), so frequency peaks can appear anywhere in the frequency table.

[0028] Then, sorter 24 sorts the frequency counts in the symbol frequency table generated by counter 22 to obtain a sorted frequency table. This sorted frequency table places the symbol with the highest count or frequency at the leftmost position, followed by the symbol with the second highest count, then the symbol with the third highest count, and so on. Therefore, in the sorted frequency table generated by sorter 24, the frequency or count value decreases along the X-axis or index.

[0029] Cutter 26 removes symbol entries with the lowest count values ​​from the sorted frequency table. For example, for 8-bit symbols with symbol values ​​from 1 to 256, the 128 symbols with the lowest frequency counts can be removed by cutter 26. Cutter 26 retains only the 128 symbols that appear most frequently. Cuttering reduces the size of the dataset and allows the neural network to process it faster.

[0030] The neural network implementing regression model 20 must first be trained using training data. Training data can include a large number of input data blocks, which also have compression ratios. For example, each training input data block can be compressed to determine the compression ratio, or another prediction calculator can be used to generate the compression ratio of the training data.

[0031] The trainer 28 can be a neural network configured for linear regression. A training input dataset with its compression ratio is fed into the trainer 28. As the trainer 28 processes more training sets, the weights and their feature maps in the neural network are adjusted until the training endpoint is reached or convergence occurs.

[0032] After trainer 28 has processed the training data and reached its endpoint, the final weights are applied to the neural network to configure it for compressed prediction. The trained neural network, acting as inferencer 29, uses the trained weights to generate compressed predictions for each input data block. During training, counter 22, sorter 24, cutter 26, and trainer 28 are used to adjust the weights of the neural network as the training data is processed. Then, during normal processing, counter 22, sorter 24, cutter 26, and inferencer 29 are used to generate the compression ratio of the predictions. Trainer 28 and inferencer 29 use the same neural network. When running as inferencer 29, the weights are fixed, but when running as trainer 28, the weights are adjusted during training.

[0033] Figure 4 The classification model is shown in more detail. Counter 22 generates a frequency count for each symbol value and can be the same module used by regression model 20. Counter 22 can be shared by regression model 20 and classification model 30.

[0034] By sorter 24 ( Figure 3 The sorting performed is time-consuming and computationally intensive, thus increasing computational latency. The inventors realized that for binary compressible decisions, sorting is unnecessary, resulting in faster decisions than when predicting compression ratios.

[0035] The concentration calculator 34 calculates the concentration of the input data block by squaring each frequency count and then summing all the squared frequency counts. For an 8-bit symbol k with a symbol value from 0 to 255, and a frequency count frequency[k], the concentration value (CV) can be expressed as:

[0036]

[0037] The concentration calculator 34 can be used to obtain the individual concentration (CV) of each data block.

[0038] The number of zero-value symbols in the input data block is counted by zero-value counter 36. This count is the same as the frequency count generated by counter 22 for symbols with zero values, so zero-value counter 36 can simply look up the symbol value 0 in the symbol frequency table generated by counter 22 to obtain the number of zero-values ​​(NZ). Alternatively, zero-value counter 36 can scan the input data block, generating a new count of the number of zero-value symbols in the input data block.

[0039] The inventors recognized that many compressible data blocks contain a large number of zero-value symbols or other symbols with high concentration. Incompressible blocks typically have few zero-value symbols and lack concentration of other symbols, resulting in low CV and NZ values. The inventors combined CV and NZ through multiplication to use as input to a threshold comparator.

[0040] Comparator 38 compares the product CV*NZ with a threshold THH to determine when the input data block is compressible or incompressible. The simple yes / no decision made by comparator 38 is fast and easy to implement. However, the usefulness and accuracy of the decision depend heavily on the threshold THH.

[0041] Figure 5This is a graph in which the classification model uses the Receiver Operating Characteristic (ROC) curve to set the compressibility decision threshold. During training, the training input data is applied to counter 22, concentration calculator 34 calculates the concentration (CV), and zero-value counter 36 finds the number of zero values ​​(NZ) (the sign of zero values) for each training data block.

[0042] Plotter 42 plots or creates a table of the product CV*NZ as a function of the compression ratio C% of compressor 40. Figure 9 This is an example of a plot generated by plotter 42, where each data point is for a different training dataset. The user sets a vertical threshold THV, which is a compression ratio percentage considered to be the compressible limit. THV is somewhat arbitrary and can be set to a value like 95% and then adjusted.

[0043] ROC generator 44 generates receiver operating characteristic (ROC) curves. ROC generator 44 tests various values ​​of the level threshold THH. For different tested THH values, ROC generator 44 obtains and plots the true positive rate as a function of the false positive rate. For example, Figure 14A This is a table generated by ROC Generator 44. Figure 14B This is the ROC curve generated by ROC generator 44.

[0044] THH selector 46 examines the ROC plot or table generated by ROC generator 44 and selects the THH value along the ROC curve that produces the point with the maximum Youden index. The Youden index can be expressed as the distance between the diagonal and the vertical line between the ROC curve, such as... Figure 13 As shown, the Youden index optimizes the discriminative power of the THH, giving equal importance to sensitivity and specificity.

[0045] Therefore, the level threshold THH is obtained from the CV*NZ and C% data of the training dataset by the ROC generator 44 and the THH selector.

[0046] The vertical and horizontal thresholds use different units. The vertical threshold THV is the compression percentage C%, while the horizontal threshold THH is a value of CV*NZ. THH is used to quickly make compression decisions, while THV is used to obtain the optimal THH value that can be used at runtime during training.

[0047] Figure 6 A confusion matrix is ​​displayed. Confusion matrices are useful for evaluating binary classification models, such as classification model 30 (…). Figure 2 The training data is used for... Figure 5A device is used to plot the relationship between CV*NZ and C%, and then a series of THH values and a fixed VTH value are applied to evaluate the effectiveness of the threshold.

[0048] The vertical threshold THV is set as the benchmark to judge true positive and negative samples. The horizontal threshold THH is the dividing line between positive and negative samples judged by a method. This means there are true positives and negatives, as well as detected positives and negatives. Therefore, there will be four groups: True Positive (TP), False Positive (FP), False Negative (FN), and True Negative (TN).

[0049] To obtain the optimal horizontal threshold THH for a dataset, the inventor plots the ROC curve based on the values in the confusion matrix. The optimal horizontal threshold THH is the upper left corner point of all ROC curves. The optimal C% is the optimal horizontal threshold.

[0050] The vertical threshold THV is expressed as a compression percentage %, and it separates the actual positive and negative results. The horizontal threshold THH is expressed as a CV*NZ value, rather than a compression percentage %. The predicted value can be positive or negative. Positive indicates non-compressible, while negative indicates compressible.

[0051] The prediction can be correct or incorrect. True Positive (TP) occurs when the prediction is positive (non-compressible) and the compression percentage is greater than the vertical threshold C% > THV. True Negative (TN) occurs when the prediction is negative (compressible) and the compression percentage is less than the vertical threshold C% < THV. For the TP and TN cases, the compression decision is correct.

[0052] Incorrect predictions occur in the cases of False Positive (FP) and False Negative (FN). False Positive (FP) occurs when the prediction is positive (non-compressible) but C% < THV, so the file is actually compressible (negative), or below the user-set vertical compression threshold. False Negative (FN) occurs when the prediction is negative (compressible) but C% > THV, so the file is actually non-compressible (positive), above the vertical threshold.

[0053] The vertical and horizontal thresholds THV, THH can be moved or adjusted to improve the prediction results and reduce false positives or false negatives. For example, moving the vertical threshold VTH to the right will increase True Positive (TP) and reduce False Positive (FP), but True Negative (TN) will also decrease, while False Negative (FN) will increase.

[0054] Figure 7It is a training data graph with predictions based on the number of zero values ​​(NZ). NZ is useful for compressed predictions.

[0055] Each point in the graph represents a different data block. When the vertical threshold THV62 is set to 90%, the horizontal threshold THH 64 can be set to a low value to provide a large number of true negatives, or compressible files with correct predictions. This low THH 64 can also capture a few true positives, or datasets that are correctly predicted as incompressible.

[0056] However, some false negatives were found in the upper right quadrants of THV 62 and THH 64. These were incompressible files that were incorrectly predicted as compressible. Very few false positives occurred in the lower left quadrant. While using only NZ as the level threshold was helpful, there were still too many false negatives.

[0057] Figure 8 This is a training data graph with concentration-based (CV) predictions. CV is also useful for compressible predictions. When the vertical threshold THV 62 is set to 90%, the horizontal threshold THH 64 can be set to a low value to provide a large number of true negatives, or compressible files with correct predictions. This low THH 64 also captures many true positives, or datasets that are correctly predicted as incompressible.

[0058] However, due to the relationship with NZ ( Figure 7 In contrast, the data tends to curve downwards, resulting in a large number of false positives. THH 64 must be set to a very low level to eliminate these false positives. True positives will then decrease, while false negatives will increase. Simply using CV to set the level threshold will produce too many false positives.

[0059] Figure 9 This is a plot of the training data based on the concentration (CV) multiplied by the number of zeros (NZ). The CV*NZ value for each training data block is plotted on the Y-axis as a function of the compression ratio C% for that block. Compared to the individual CV or NZ curves, the curve of the product CV*NZ is steeper and more suitable for (…). Figure 7-8 This steeper plot provides better differentiation between predictions.

[0060] The vertical threshold THV 62 can be set to a higher value, such as 97%. This allows more files to be compressed. True negatives increase. The horizontal threshold THH 64 can be set to a lower value to reduce false positives while still identifying true positives (incompressible files).

[0061] Figure 10 yes Figure 9 A magnified view of the graphic. Figure 10The area near the intersection of the vertical threshold THV 62 and the horizontal threshold THH 64 was magnified. By adjusting THV to 97.5%, false negatives were almost eliminated. Some false positives remained, but there were more true positives.

[0062] Figure 11A-11B A flowchart illustrating the operation of the neural network used for compressed prediction and decision-making is shown. Figure 11A In step 502, the symbol dataset is received and the frequencies of all symbols are counted. In step 504, when only one compression decision is needed, the process proceeds as follows: Figure 11B The process continues. In step 504, when a compressed prediction is needed instead of a decision, the symbol frequencies are sorted in step 506. When only a decision is needed, no sorting is required. Figure 11B This allows for faster decisions to make compression decisions.

[0063] exist Figure 11A Continuing in step 508, low-frequency symbols are pruned and discarded, resulting in a smaller sorted symbol frequency dataset. Due to the smaller dataset, subsequent processing may be reduced.

[0064] In step 510, the neural network is trained using the training data to obtain training weights. Then, in step 512, the neural network uses the trained weights to generate a prediction of the compression ratio of the input dataset. Step 510 can be repeated on many training datasets that have been sorted and pruned in steps 502, 506, and 508 before processing the actual dataset in step 512.

[0065] exist Figure 11B In this process, only compression decisions are needed; compression ratio prediction is not required. Symbol frequencies are unsorted but used to calculate the concentration (CV) (step 520) and the number of zero values ​​(NZ) of the unsorted input dataset (step 522).

[0066] The horizontal threshold THH has been used previously. Figure 12 The ROC process is obtained in step 530. In step 532, the product of CV and NZ (CV*NZ) is compared with the level threshold THH. In step 534, if CV*NZ is greater than THH, then the file is compressible and the NOCOMPRESS flag is set to false.

[0067] Figure 12 This is a flowchart illustrating the optimization of the horizontal threshold THH using a receiver operating characteristic (ROC) curve. The ROC process 530 is performed during training before the real-time input dataset is processed. In step 542, each training data block is compressed to obtain an actual compression ratio C%. For example, via... Figure 11BSteps 520 and 522 obtain the CV and NZ for each data block. Then, in step 544, the product CV*NZ is plotted or tabulated as a function of C%.

[0068] In step 546, points on the ROC curve are generated for different THH test values ​​by determining when each training data block is a true positive or a false positive. In step 548, the optimal THH value is selected by calculating the Youden exponent for each THH test value and selecting the THH test value with the highest Youden exponent. In step 550, the selected THH with the highest Youden exponent is returned. During the processing, a compression decision can be quickly made by comparing the selected THH with the product CV*NZ.

[0069] Figure 13 This is a graph of several Receiver Operational Characteristic (ROC) curves. Many training data blocks were processed, and the CV*NZ of each block was plotted or tabulated as a function of C%. The result is as follows: Figure 9 The graph shown represents a different data block in the training data, with each point representing a different data point. The vertical threshold THV is set to a user-defined value, such as 95%. However, the horizontal threshold THH moves up and down within the range of THH values ​​in the graph. As the horizontal threshold THH 64 moves upward, more data points are classified as false positives, but there may also be more true positives. Moving the horizontal threshold THH 64 downward will increase true negatives and decrease both false and true positives.

[0070] For the range of THH values, these different true positives and false positives are plotted or tabulated to form ROC curves 122. Each ROC curve represents the diagnostic ability of the binary classifier system. A random classifier produces a diagonal 120. Generally, ROC curves farther from the diagonal 120 represent better classifiers. Therefore, ROC curve 124 is a better classifier than ROC curve 122, while ROC curve 126 is worse than ROC curves 124 and 122.

[0071] For any given neural network classifier system, all training data blocks will generate points along only one ROC curve. For example, when the neural network generates points along ROC curve 124 for different THH test values, the optimal value of THH can be found at the point along ROC curve 124 that is furthest from the diagonal 120. This optimization can be found by finding the Youden exponent for each THH test value, or for each point along ROC curve 124. The point with the largest Youden exponent is the optimal value of THH.

[0072] In this example, the Yoden index 128 is the maximum value of the ROC curve 124. The Yoden index 128 points to the point on the ROC curve 124 with the optimal THH value.

[0073] Figures 14A-14B The generation of the ROC curve and selection of the optimal THH with the highest Yorkden index are highlighted. In this example, the vertical threshold THV is set to 95% because a file is considered compressible if it is at least 5% smaller when compressed.

[0074] exist Figure 14A In the process, many training data blocks have been processed to generate a table of CV*NZ values ​​as a function of C%, and in Figure 14A The paper presents statistics and lists the true positive (TP), false positive (FP), true negative (TN), and false negative (FN) values ​​for various THH test values. The THH test value ranges from 0.0 to 1.0. Figure 14A As can be seen from the table, as THH increases, FP increases while TN decreases.

[0075] from Figure 14A The true positive rate was calculated from the TP and FN values:

[0076] TPR = TP / (TP + FN)

[0077] TPR is also known as recall rate. Figure 14A The false positive rate is calculated from the FP and TN values ​​in the table:

[0078] FPR = FP / (FP + TN)

[0079] The calculated TPR is plotted as a function of FPR to obtain... Figure 14B The curve in the table. Each THH value in the table corresponds to... Figure 14B This refers to a point on the ROC curve 140. The point on the ROC curve 140 that has the largest distance to the diagonal 120 has the highest Youden exponent and is the optimal point. The point with a THH value of 0.1 is this optimal point because it is farther from the diagonal 120 than points with THH values ​​of 0.2, 0.3, 0.5, 0.9, etc. The actual Youden exponent for each point on the ROC curve 140 can be calculated as its vertical distance to the diagonal 120. The highest Youden exponent can then be selected, and its corresponding THH output can be used as the optimized THH.

[0080] Alternative embodiments

[0081] The inventors have also added several other embodiments. Compressor 40 can determine the compression ratio (C%) for each training data block, for example, by compressing the data and comparing the file sizes before and after compression, or by using a compression ratio algorithm without actually compressing the data. Alternatively, C% can be provided along with the input training data, thus eliminating the need for compressor 40.

[0082] While charts and graphs have been shown and described to aid reader comprehension, computer systems can implement these graphs or figures using tables or other data structures. Tables can be computer-implemented graphs.

[0083] While the optimal level threshold THH has been described as being found by the point in the ROC curve with the highest Yoden exponent, a suboptimal level threshold THH can also be used, for example, by a point in the ROC curve with the second highest Yoden exponent. Some applications allow for suboptimal thresholds. The level threshold THH can also be set once, for example, during initialization or during factory fabrication, and remain unchanged for all future datasets, although recalibrating the level threshold THH for new datasets can yield better compression decisions. The level threshold THH can also be set empirically or by the user.

[0084] Various parameters can be expressed in various ways and with various variations. For example, for an incompressible file, the compression ratio can be defined as 100%; for a file that is compressed to a size 80% smaller, the compression ratio can be defined as 20%; or for a file that is compressed to a size 20% smaller, the compression ratio can be defined as 20%. Other statistical data can also be changed, redefined, transferred, converted, etc.

[0085] ROC curves can be generated using different X and Y axis definitions. The true positive rate and false positive rate can be redefined. The optimal points on these alternative ROC curves can be obtained using methods similar to the Youden index, such as graphical methods.

[0086] The number of zero values ​​(NZ) can be tracked using a zero-value counter, memory such as semiconductor or disk storage, or by inputting or reading from a symbol frequency table. NZ can be stored in a zero-value counter, memory, table, register, or processing unit, and can be stored in multiple locations.

[0087] Various block sizes and symbol sizes can be used. Block sizes can be fixed or variable. An 8-bit symbol and an 8-byte block are just one of many possibilities. Symbol values ​​can be encoded in binary, two's complement, or other encoding codes.

[0088] Some implementations may not use all components. Additional components can be added. The loss function that converges during training can use various error / loss and cost generators, such as a weight decay term to prevent weights from growing too large over many training optimization cycles, and a sparsity penalty to encourage nodes to zero out their weights so that only a small fraction of the total nodes are used. Many substitutions, combinations, and variations are possible. Other variations and types of loss or cost terms can be added to the loss function. The values ​​of the relative scaling factors for different cost functions can be adjusted to balance the effects of various functions. The training endpoint of the neural network can be set for various combinations of conditions, such as the desired final accuracy, the accuracy-hardware cost product, the target hardware cost, etc.

[0089] Neural networks, loss functions, and other components can be implemented using various techniques, employing diverse combinations of software, hardware, firmware, routines, modules, functions, etc. The final result can be derived from the neural network with final weights and may be implemented as a program module or in application-specific integrated circuits (ASICs) or other hardware to improve processing speed and reduce power consumption.

[0090] The terms left, right, top, and bottom are relative and can be flipped, rotated, transformed, or transposed in various ways. Adders can add two's complement values ​​to perform subtraction. Therefore, subtraction and addition can be used interchangeably.

[0091] Various interleaving and mapping schemes can alter the actual storage location to optimize bandwidth or other design parameters. Many memory arrangements, both physical and logical, are possible. Various parallel processing techniques can be used to perform the basic operations described here serially for easier understanding.

[0092] The background section of this invention may include background information about the problem or environment of the invention, rather than a description of prior art. Therefore, the material included in the background section is not an admission of prior art by the applicant.

[0093] Any methods or processes described herein are machine-implemented or computer-implemented and are intended to be performed by machines, computers, or other devices, and not necessarily by humans alone without machine assistance. Tangible results may include reports or other machine-generated displays on display devices such as computer monitors, projection devices, audio generation devices, and related media devices, and may include hard-copy printouts that are also machine-generated. Computer control of other machines is another tangible result.

[0094] Any advantages and benefits described herein may not necessarily apply to all embodiments of the invention. When the word "device" appears in a claim element, the applicant intends that the claim element fall within the provisions of Section 112, Subsection 6 of 35 USC. Typically, one or more words precede the word "device." These words preceding "device" are a label intended to facilitate reference to the claim element, not to express structural limitations. Such device-plus-function claims must cover not only the structure described herein for performing that function and its structural equivalents, but also equivalent structures. For example, although nails and screws have different constructions, they are equivalent structures because they both perform the fastening function. Claims that do not use the word "device" do not fall within the provisions of Section 112, Subsection 6 of 35 USC. Signals are typically electronic signals, but can also be optical signals, for example, transmitted via fiber optic lines.

[0095] The above description of embodiments of the invention is provided for illustrative and descriptive purposes. It is not intended to be exhaustive, nor is it intended to limit the invention to the precise forms disclosed. Many modifications and variations are possible based on the above teaching. The purpose is that the scope of the invention is not limited by this detailed description, but rather by the appended claims.

Claims

1. A data compression analyzer, comprising: A counter is used to count the number of times a symbol appears in the input data block to generate a symbol frequency table and store the symbol frequencies of the input data block. A calculator for calculating the concentration CV of the input data block by summing the squares of the symbol frequencies in the symbol frequency table; A zero-value tracker having a zero-value count NZ, which indicates the number of symbols in the input data block whose symbol value is zero; A multiplier that multiplies the concentration CV by the number of zero values ​​NZ to obtain the product CV*NZ; and A comparator is used to compare the product CV*NZ with a horizontal threshold. When the product CV*NZ exceeds the horizontal threshold, a signal is sent to indicate that the input data block is compressible; when the product CV*NZ is less than the horizontal threshold, a signal is sent to indicate that the input data block is incompressible. Therefore, the product CV*NZ is compared with the level threshold to indicate whether the input data block is compressible or incompressible.

2. The data compression analyzer according to claim 1, further comprising: A zero-value counter counts the number of symbols with zero values ​​in the input data block to generate the number of zero values ​​NZ.

3. The data compression analyzer according to claim 1, wherein, The number of zero values, NZ, is obtained by reading the entries with a symbol value of zero in the symbol frequency table.

4. The data compression analyzer according to claim 1, further comprising: Threshold optimizer, including: A trainer that applies multiple training data blocks to the counter, causing the calculator and the multiplier to generate multiple CV*NZ products for the multiple training data blocks; Herein, a compression ratio is received or generated for each of the plurality of training data blocks, and the compression ratio of a training data block is the ratio of the size of the compressed training data block to the size of the training data block. A tabulator that tabulates the plurality of CV*NZ products as a function of the compression ratio of each of the plurality of training data blocks; The receiver operation feature ROC tabulator compares the multiple CV*NZ products with the horizontal threshold to obtain the prediction result, compares the compression ratio of each training data block with a vertical threshold to obtain the actual result, and generates the true positive rate and false positive rate from the prediction result and the actual result. The ROC optimizer selects the optimal combination of the true positive rate and the false positive rate, and outputs a test value of the level threshold corresponding to the optimal combination as the level threshold for use by the comparator for input data blocks.

5. The data compression analyzer according to claim 4, further comprising: A Youden index generator that generates multiple Youden indices, each of which corresponds to a point on the ROC curve of the combination of the true positive rate and the false positive rate; The ROC optimizer selects the Yoden index with the maximum value to determine an optimal combination and the level threshold that produces the optimal combination.

6. The data compression analyzer of claim 4, wherein the ROC tabulator receives a vertical threshold indicating a compression ratio, wherein training data blocks with a compression ratio greater than the vertical threshold are considered incompressible, and wherein training data blocks with a compression ratio less than the vertical threshold are considered compressible; The ROC tabulator tests multiple test values ​​of the level threshold and compares the test values ​​of the level threshold with the multiple CV*NZ products. Training data blocks whose CV*NZ products are greater than the test values ​​of the level threshold are considered predictably compressible, and training data blocks whose CV*NZ products are less than the test values ​​of the level threshold are considered predictably incompressible. The ROC tabulator lists the ROC curves, where each point on the ROC curve represents a different test value of the level threshold, and the ROC tabulator lists true positives that are considered incompressible predictive incompressible training data blocks, true negatives that are considered compressible predictive compressible training data blocks, false positives that are considered incompressible predictive compressible training data blocks, and false negatives that are considered compressible predictive incompressible training data blocks. in, The ROC tabulator tabulates the true positive rate of each of the plurality of test values ​​of the level threshold as a function of the false positive rate to determine points on the ROC curve. The true positive rate is calculated by dividing the true positive rate by the sum of the true positive and false negative rates. The false positive rate is calculated by dividing the false positive rate by the sum of the false positive and true negative rates.

7. The data compression analyzer according to claim 6, wherein the ROC optimizer selects an optimal point on the ROC curve and outputs a test value of a horizontal threshold corresponding to the optimal point as the horizontal threshold for use by the comparator for the input data block.

8. The data compression analyzer of claim 7, wherein the ROC optimizer calculates a plurality of Youden indices, wherein each Youden index corresponds to a point on the ROC curve, and the ROC optimizer selects the Youden index with the maximum value to determine the optimal point.

9. The data compression analyzer according to claim 4, further comprising: A sorter that sorts the symbol frequency table according to the symbol values ​​to generate a sorted symbol frequency table; A cutter that discards entries in the sorted symbol frequency table whose symbol values ​​are below a cutoff value to generate a cut sorted symbol frequency table, the cut sorted symbol frequency table having fewer entries than the sorted symbol frequency table. A neural network receives the cut sorted symbol frequency table as input and generates a predicted compression ratio for the input data block.

10. The data compression analyzer according to claim 9, wherein, For each of the plurality of training data blocks, a compression ratio is received or generated, wherein the compression ratio of a training data block is the ratio of the size of the compressed training data block to the size of the training data block; A neural network trainer uses the plurality of training data blocks to train the neural network, the training data blocks being processed by the counter, sorter, and cutter and input into the neural network, wherein the weights of the neural network are adjusted to converge the predicted compression ratio to the compression ratio of each training data block during training. Thus, the neural network is trained by the plurality of training data blocks to converge the predicted compression ratio to the compression ratio of each training data block during training.

11. A compression manager, comprising: A regression model that generates a predicted compression ratio for the input data block; A classification model that generates binary compression decisions for the input data block, wherein the classification model further includes: A counter generates a symbol frequency table by counting the number of occurrences of each symbol value in the input data block to produce a symbol frequency table; A concentration CV generator generates the CV of the input data block by summing the squares of the symbol frequencies in the symbol frequency table; A zero-value count (NZ) indicator, which indicates the number of symbols with zero sign values ​​in the input data block; A multiplier that multiplies the CV and the NZ to generate a CV*NZ product; and The decision module generates a binary compressible decision when the CV*NZ product is greater than a horizontal threshold, and generates an incompressible decision when the CV*NZ product is less than the horizontal threshold. Thus, the binary compression decision is generated by comparing the CV*NZ product with the level threshold.

12. The compression manager of claim 11, wherein the regression model further comprises: A sorter that sorts the symbol frequency table into a sorted table stored by symbol value; A cutter removes entries with low symbol values ​​from the symbol frequency table to generate a frequency table of reduced size; A neural network receives the frequency table with reduced size and generates the predicted compression ratio.

13. The compression manager of claim 12, wherein the regression model further comprises: A trainer adjusts the weights of the neural network by processing multiple blocks of training data with a training compression ratio, wherein the weights are adjusted so that the predicted compression ratio converges to the training compression ratio. Thus, the neural network is trained to predict the compression ratio.

14. The compression manager according to claim 11, wherein, The classification model also includes: A threshold generator generates the level threshold by processing multiple training data blocks, each training data block having a compression ratio, and the threshold generator generates a CV*NZ product for each of the multiple training data blocks.

15. The compression manager of claim 14, wherein the threshold generator further comprises: A first table generator generates a first table that uses the CV*NZ product as a function of the compression ratio; Vertical threshold, which represents the threshold compression ratio, wherein blocks with a compression ratio greater than the threshold compression ratio are classified as actually compressible blocks, and blocks with a compression ratio less than the threshold compression ratio are classified as actually incompressible blocks; A threshold tester sets a horizontal threshold to multiple test values, where blocks whose CV*NZ product is greater than the horizontal threshold are predicted compressible blocks, and blocks whose CV*NZ product is less than the horizontal threshold are predicted incompressible blocks. A second table generator generates a second table of true and false prediction rates, wherein each entry in the second table has a true prediction rate and a false prediction rate, as well as a test value for a level threshold that produces the true and false prediction rates for that entry.

16. The compression manager of claim 15, wherein the true prediction rate and the false prediction rate are functions of at least two of the following: Predict the number of compressible blocks; Predict the number of incompressible blocks; The actual number of compressible blocks; and The actual number of incompressible blocks; All of the above items are determined by the threshold tester for multiple test values ​​of the horizontal threshold.

17. The compression manager of claim 16, wherein the threshold generator further comprises: A level threshold selector selects an entry from the second table as the selected entry and outputs the level threshold from the selected entry for use by the decision module.

18. The compression manager of claim 15, wherein the threshold generator further comprises: Yoden index generator, which generates a Yoden index for each entry in the second table; The Yoden maximizer selects an entry from the second table as the selected entry, whose Yoden index is greater than that of other entries in the second table.

19. A computer-implemented method for managing compression, comprising: Receive input data blocks and store at least a portion of the input data blocks on a computer storage device; A symbol frequency table is generated by counting the number of occurrences of symbols in the input data block to generate symbol frequencies, and then stored in the computer memory. The concentration CV of the input data block is calculated by summing the squares of the symbol frequencies read from the symbol frequency table; The symbol frequency table includes zero-value entries for zero-value symbols, and the symbol frequency of the zero-value entries is the number of zero-values ​​NZ, which indicates the number of symbols with zero-value symbols in the input data block. The concentration CV is multiplied by the number of zero values ​​NZ using a computer to obtain the CV*NZ product; and The CV*NZ product is compared with a horizontal threshold. When the CV*NZ product exceeds the horizontal threshold, a signal is sent to notify that the input data block is compressible; when the CV*NZ product is less than the horizontal threshold, a signal is sent to notify that the input data block is incompressible. Therefore, the computer compares the CV*NZ product with the level threshold to indicate whether the input data block is compressible or incompressible.

20. The computer-implemented method of claim 19, further comprising, during training: The computer processes multiple training data blocks, each of which has a compression ratio, and generates a CV*NZ product for each of the multiple training data blocks. Generate a first table that uses the CV*NZ product as a function of the compression ratio; in, The vertical threshold represents the threshold compression ratio, where blocks with a compression ratio greater than the threshold compression ratio are classified as actually compressible blocks, and blocks with a compression ratio less than the threshold compression ratio are classified as actually incompressible blocks. The horizontal threshold is set to multiple test values, where the block whose CV*NZ product is greater than the horizontal threshold is a predicted compressible block, and the block whose CV*NZ product is less than the horizontal threshold is a predicted incompressible block. A second table of true and false prediction rates is generated, wherein each entry in the second table has a true prediction rate and a false prediction rate, as well as a test value for a level threshold that generates the true and false prediction rates for that entry; The true prediction rate and the false prediction rate are functions of at least two of the following: The predicted number of compressible blocks; The predicted number of incompressible blocks; The actual number of compressible blocks; and The actual number of incompressible blocks; All of the above items are determined by multiple test values ​​of the stated level threshold; Generate a Yoden index for each entry in the second table; Select one entry from the second table as the selected entry, wherein the Youden index of the selected entry is greater than the Youden index of any other entry in the second table; and Output the level threshold from the selected entries.

Images