A distributed compression storage method based on data sharing

Abstract

The present invention relates to the field of electronic digital data processing technology, and in particular to a distributed compression storage method based on data sharing. The method comprises: obtaining an uplink adjustment coefficient and a downlink adjustment coefficient of a data according to the uplink data and downlink data corresponding to the data in a feature sequence; obtaining a basic tolerance of the data according to the uplink adjustment coefficient, the downlink adjustment coefficient, the uplink data and the downlink data of the data in the feature sequence; obtaining a tolerance weight of the data according to the frequency of occurrence of a data in the feature sequence, the uplink data and the downlink data of the data; obtaining a final tolerance of each data based on the basic tolerance and the tolerance weight of each data; compressing the data sequence corresponding to the image using the final tolerance of each data, and sharing the compressed data to each secondary server for distributed storage. The present invention can improve the compression effect of image data by obtaining an adaptive final tolerance.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic digital data processing, and in particular to a distributed compression storage method based on data sharing. Background Art

[0002] In recent years, the Internet has developed rapidly, and big data has been applied in more and more industries. The data density is also getting higher and higher, and the amount of data generated is also correspondingly increasing. In particular, the storage of image data on the client has a particularly serious impact on the storage cost. Therefore, when storing big data, compression is needed to reduce the amount of data during storage, so as to reduce the storage cost.

[0003] When storing the client's image data, due to the unique visual recognition system of the human eye, the loss of part of the image information can be allowed, that is, the image data that the client needs to store can be compressed using a lossy compression algorithm, that is, the image data can be converted into one-dimensional data, and the image data is compressed using a revolving door compression algorithm. In the revolving door compression algorithm, tolerance is a factor that determines distortion and compression rate, and the tolerance is fixed in the existing revolving door compression algorithm. Therefore, the compression effect on data with different changes is not good, that is, it cannot guarantee that the compression rate and distortion rate of the data reach a reasonable balance point. Summary of the invention

[0004] In order to solve the problem that the tolerance in the existing revolving door compression algorithm is a fixed value and the compression effect of data with different change trend characteristics is not good, the purpose of the present invention is to provide a distributed compression storage method based on data sharing, and the technical solution adopted is as follows:

[0005] An embodiment of the present invention provides a distributed compression storage method based on data sharing:

[0006] The grayscale value of a channel of each pixel in the image forms a one-dimensional data sequence; the redundant data in the data sequence is removed to obtain a feature sequence;

[0007] Acquire uplink data and downlink data corresponding to a data of the characteristic sequence in the SDT compression algorithm process, and obtain an uplink adjustment coefficient and a downlink adjustment coefficient of the data based on the uplink data and the downlink data;

[0008] Obtaining a basic tolerance of a data according to an upward adjustment coefficient, a downward adjustment coefficient, the upward data and the downward data of the data in the characteristic sequence;

[0009] Obtain the tolerance weight of a data according to the frequency of occurrence of the data in the feature sequence, the upstream data and the downstream data of the data;

[0010] The final tolerance of each data is obtained based on the basic tolerance and tolerance weight of each data; the final tolerance of each data is used to compress the data sequence corresponding to the image, and the compressed data is shared to each secondary server for distributed storage.

[0011] Preferably, obtaining the uplink data and downlink data corresponding to a data in the characteristic sequence during the SDT compression algorithm includes: selecting a data, obtaining the data in the characteristic sequence that is equal to the data, and recording it as equal data; obtaining the data that is backward adjacent to each equal data and is greater than the equal data, and recording it as the uplink data of the data; obtaining the data that is backward adjacent to each equal data and is less than the equal data, and recording it as the downlink data of the data.

[0012] Preferably, obtaining the uplink adjustment coefficient and the downlink adjustment coefficient of the data based on the uplink data and the downlink data includes:

[0013] Obtain the ratio of the number of uplink data corresponding to a data in the feature sequence to the sum of the number of uplink data and the number of downlink data; obtain the reciprocal of the average value and the standard deviation of all uplink data; the product of the ratio, the reciprocal of the average value and the standard deviation is the first calculation parameter of the uplink data, and similarly obtain the second calculation parameter of the downlink data; obtain the result of adding the first calculation parameter and the second calculation parameter, the ratio of the first calculation parameter to the result of the addition is the uplink adjustment coefficient of the data, and the ratio of the second calculation parameter to the result of the addition is the downlink adjustment coefficient of the data.

[0014] Preferably, the basic tolerance of the data is obtained according to the uplink adjustment coefficient, the downlink adjustment coefficient, the uplink data and the downlink data of a data in the feature sequence, including: taking the uplink adjustment coefficient of a data in the feature sequence as the weight of the average value of the difference between each uplink data and the data, and taking the downlink adjustment coefficient as the weight of the average value of the difference between the data and each downlink data, summing them up to obtain a summation result; and taking the average value of the summation results as the basic tolerance of the data.

[0015] Preferably, the tolerance weight of a data is obtained according to the frequency of occurrence of a data in a feature sequence, the upstream data and the downstream data of the data, including: classifying the same data in the feature sequence into one category, obtaining the number of data in each category and the total number of categories; obtaining the difference between the upstream data and the downstream data and the data; obtaining the tolerance weight of the data according to the frequency of occurrence of a data, the number of data in each category in the feature sequence, the total number of categories, and the absolute value of the ratio of the difference to the data.

[0016] Preferably, the tolerance weight of the data is:

[0017]

[0018] in, Represents the tolerance weight of the jth data in the feature sequence; Indicates the frequency of the jth data in the feature sequence; Statement The number of data in the class data, The size of the data in the class data is ; Indicates the total number of categories in the data sequence; and Respectively represent the amount of uplink data and the amount of downlink data of the data; Indicates the first individual data; Represents an exponential function with the natural constant e as the base.

[0019] Preferably, obtaining the final tolerance of each data based on the basic tolerance and tolerance weight of each data includes: obtaining the sum of a first preset value and the tolerance weight, and multiplying the basic tolerance by the ratio of the sum to a second preset value to obtain the final tolerance.

[0020] Preferably, removing redundant data in the data sequence to obtain the characteristic sequence includes: in the characteristic sequence, only one data in the continuously repeated data is retained, and the non-continuously repeated data is directly retained, thereby obtaining the characteristic sequence.

[0021] The embodiments of the present invention have at least the following beneficial effects: the present invention converts image data into one-dimensional sequence data, can analyze the image data as a whole, and obtain a feature sequence of the data sequence at the same time. The feature sequence can not only characterize the characteristics of the data in the data sequence, but also has a small amount of data, and can more conveniently analyze each data; obtain the uplink data and downlink data of a data in the feature sequence, and then obtain the uplink adjustment coefficient and the downlink adjustment coefficient respectively according to the uplink data and the downlink data, as the adjustment coefficient when calculating the basic tolerance of the data, and then analyze the change characteristics of the data in the feature sequence in combination with the uplink data and the downlink data of the data, which can A more appropriate basic tolerance is obtained; then, according to the frequency of the data appearing in the feature sequence, the tolerance weight is obtained by combining the uplink data and the downlink data of the data, and the basic tolerance is adjusted to obtain the final tolerance. Here, the change characteristics of each data in the feature sequence are combined to obtain the adaptive final tolerance of each data, so as to improve the compression effect when compressing the data in the data sequence converted from the image. The data after edge-proof compression is shared to each secondary server for distributed storage, so that the compression rate and the distortion rate reach a reasonable balance, that is, while improving the compression rate, the distortion rate is kept within a reasonable range, which does not affect the human eye's observation of the decompressed image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 A method flow chart of a distributed compression storage method based on data sharing provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation, structure, features and effects of a distributed compression storage method based on data sharing proposed by the present invention in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures, or characteristics in one or more embodiments may be combined in any suitable form.

[0025] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0026] The following is a detailed description of a specific scheme of a distributed compression storage method based on data sharing provided by the present invention in conjunction with the accompanying drawings.

[0027] Example:

[0028] The main application scenario of the present invention is: when the image data of the client is shared and distributedly stored, in order to reduce the storage amount of data, the revolving door compression algorithm is used to compress the image data transmitted by the client. The present invention obtains the tolerance corresponding to each grayscale value in the image data uploaded by the client, and then uses the revolving door compression algorithm to compress and store the image data, where the English abbreviation of the revolving door compression algorithm is SDT compression algorithm.

[0029] The purpose of the present invention is to use image data to extract feature sequences, and then use the feature sequences to adapt the tolerance sizes of different grayscale values ​​in the image, and finally use the SDT compression algorithm to perform lossy compression of image data according to the adaptive tolerance of different grayscale values ​​to realize distributed storage of data sharing.

[0030] See also Figure 1 , which shows a method flow chart of a distributed compression storage method based on data sharing provided by an embodiment of the present invention, the method comprising the following steps:

[0031] Step S1, the grayscale value of a channel of each pixel point of the image forms a one-dimensional data sequence; and the redundant data in the data sequence is removed to obtain a feature sequence.

[0032] When the present invention compresses the image data uploaded by the user client using the SDT compression algorithm, it is first necessary to obtain the feature sequence of each image, and deduplication processing is required when obtaining the feature sequence.

[0033] The present invention needs to perform SDT compression based on adaptive tolerance on the image data to be compressed, and the SDT compression algorithm can achieve the best compression effect when compressing sequence data, so firstly, it is necessary to perform one-dimensional serialization on the two-dimensional image data, and when obtaining the tolerance of each data in the one-dimensional sequence data, repeated continuous data is unnecessary. Not only is it unnecessary but it will also increase the corresponding calculation amount, so it is necessary to remove the continuous repeated data to obtain a feature sequence of continuous non-repeated serialized image data. For example, the gray value corresponding to a row of pixels of a certain part of the existing image data is [a, a, a, a, a, b]. When the gray value of this part is used to quantify the tolerance size, essentially only the two gray values ​​a and b are needed. However, because the original gray value is [a, a, a, a, a, b], it is necessary to perform correlation calculations between each adjacent gray value a and the previous or next a, resulting in more data processing costs, and then a continuous feature sequence is used to reflect the changes between the pixels of the image, and finally the tolerance size is adapted according to the changes between the pixels.

[0034] First Take the image data to be compressed as an example, the grayscale value of each channel of each pixel in the image is combined into a one-dimensional data sequence, and the data in the continuously repeated data in the data sequence is processed, and only one data in the continuously repeated data is retained to obtain a feature sequence. There are several one-dimensional data sequences and corresponding feature sequences for the grayscale values ​​of several channels in an image. The specific way to obtain it is as follows:

[0035] First, The image data to be compressed are serialized into two-dimensional image data. The specific method is as follows: The first row of pixels of the image data to be compressed is taken as the basis, and then each subsequent row of pixels is arranged in sequence after the first row of pixels to obtain the first row of pixels. The serialized data sequence of the images to be compressed , as shown below:

[0036]

[0037] in, Indicates The first image to be compressed after serialization The size of the data, ,in Represents the total number of data in the sequence, that is, the number of The total number of pixels in the image to be compressed, Indicates the first The first image to be compressed The gray value of each pixel;

[0038] Furthermore, based on the data sequence, the corresponding feature sequence is obtained. The Data For example, the method of removing the backward duplicate data is as follows:

[0039] First, repeat the data backwards Calculation:

[0040]

[0041] in, The first The size of the data, The first The size of the next data after the data; For the The data duplication value of each data;

[0042] There are two possible values ​​for :

[0043]

[0044] in When and No repetition, The data is not If the number of data is repeated, the To find the backward repeated data of data, select data, find the Backward repetition of data; When and Repeat, The data is If the data is repeated backward, continue with the Data and Repeat the calculation of the data value to obtain Corresponding data repetition value, determine whether it is 0, if it is 0, then for If it is not 0, then it is not, and then start looking for the first Data The backward repeated data is found in the same way as the first The method of searching for the backward duplicate data of a data is the same. It should be noted that the backward here refers to the data after a data.

[0045] Then obtained the The backward duplicate data of the data are removed and only the first Data For example, if the data sequence is [A,B,B,B,C,C,A,A], then its characteristic sequence is [A,B,C,A], that is, the consecutively repeated data BBB, CC and AA are processed, and only one data is retained, which are B, C and A respectively. Because the first A is not repeated continuously, it is directly retained, so the characteristic sequence is [A,B,C,A].

[0046] So far, we can obtain The feature sequence after the data sequence corresponding to the image data to be stored is processed ,in As shown below:

[0047]

[0048] in Indicates the first data, Indicates the first data; , .

[0049] Step S2, obtaining uplink data and downlink data corresponding to a data of the characteristic sequence in the SDT compression algorithm process, and obtaining an uplink adjustment coefficient and a downlink adjustment coefficient of the data based on the uplink data and the downlink data.

[0050] When the image data is compressed using the SDT compression algorithm, the overall compression rate depends on the size of the tolerance. The larger the tolerance, the higher the compression rate, but the corresponding data distortion rate is also higher; on the contrary, the smaller the tolerance, the lower the compression rate, but the corresponding data distortion rate is also lower. Therefore, the present invention calculates the basic tolerance of each data by the difference between the data of different sizes in the feature sequence (corresponding to the different gray values ​​in the image) and the backward continuous data. Because the data compression using the SDT compression algorithm is a backward compression process, it is compressed according to the fluctuation trend of the data.

[0051] Therefore, it is necessary to obtain the backward adjacent data of each data for analysis, and then characterize the characteristics of the data. The so-called backward adjacent data refers to the data that is adjacent to a data and after this data, such as AB, B is the backward adjacent data of A. At the same time, for a data, the data in the feature sequence is mainly divided into three categories, one is greater than the data, one is less than the data, and one is equal to the data. Because the tolerance of equal data is consistent, it is necessary to combine the data greater than one data and less than one data for analysis, and then obtain the change characteristics of the data.

[0052] Thus, the uplink data and downlink data corresponding to a data in the feature sequence are obtained, specifically: select a data, obtain the data in the feature sequence that is equal to the data, and record it as equal data; obtain the data that is adjacent to each equal data in the backward direction and is larger than the equal data, and record it as the uplink data of the data; obtain the data that is adjacent to each equal data in the backward direction and is smaller than the equal data, and record it as the downlink data of the data. It should be noted that the uplink data and downlink data of equal data are the same, and the uplink data and downlink data in the present invention are just to give a name to the two types of data, and have nothing to do with the uplink data and downlink data in data transmission.

[0053] Furthermore, it is necessary to analyze the discreteness of the uplink data and the downlink data of a data respectively, and obtain the uplink adjustment coefficient and the downlink adjustment coefficient corresponding to the data; obtain the ratio of the number of uplink data corresponding to a data in the characteristic sequence to the sum of the number of uplink data and the number of downlink data; obtain the reciprocal and standard deviation of the average value of all uplink data; the product of the ratio, the reciprocal of the average value and the standard deviation is the first calculation parameter of the uplink data, and the second calculation parameter of the downlink data is obtained in the same way; the result of adding the first calculation parameter and the second calculation parameter is obtained, the ratio of the first calculation parameter to the result of the addition is the uplink adjustment coefficient of the data, and the ratio of the second calculation parameter to the result of the addition is the downlink adjustment coefficient of the data, specifically, first obtain the first calculation parameter corresponding to the uplink data, expressed by the formula:

[0054]

[0055] in, Indicates the first calculation parameter corresponding to the uplink data; and Respectively represent the number of uplink data and the number of downlink data of a data; Represents the mean of all upstream data; Represents a data set consisting of all uplink data corresponding to a data; Indicates that all uplink data corresponding to a data constitute the first Uplink data.

[0056] Among them, the first calculation parameter represents the discrete degree of all uplink data corresponding to a data and the proportion of uplink data. Because the tolerance of each data needs to be calculated later, when calculating the tolerance of each data, it is necessary to analyze the uplink data and downlink data of the data separately. When the proportion of uplink data is larger and the discrete degree is larger, the influence on the calculation tolerance is greater, that is, the larger the first calculation parameter is, the larger the corresponding data is. The greater the dispersion and proportion of the uplink data, the greater the tolerance needs to be adjusted using the uplink adjustment coefficient and the downlink adjustment coefficient. Therefore, the proportion of the uplink data in the overall uplink data and downlink data is added to the formula. The larger the value is, the more serious the impact of the upstream data will be on the tolerance. It indicates the degree of dispersion of the upstream data, because different data are calculated may be the same, so use Make a distinction.

[0057] Similarly, according to the method of obtaining the first calculation parameter of the uplink data, the second calculation parameter of the downlink data is calculated , just need to replace the parameters corresponding to the downlink data during calculation. So far, the first and second calculation parameters corresponding to the uplink data and downlink data of a data in the feature sequence are obtained. Further, they are normalized to obtain the uplink adjustment coefficient corresponding to the uplink data and the downlink adjustment coefficient corresponding to the downlink data, which are expressed by the formula:

[0058]

[0059]

[0060] in, and represent the uplink adjustment coefficient and the downlink adjustment coefficient respectively, and They represent the first calculation parameter and the second calculation parameter respectively. The purpose of normalization here is to convert them into weights for subsequent analysis of the uplink data and downlink data of the data, so as to obtain a more appropriate basic tolerance.

[0061] Step S3, obtaining a basic tolerance of a data according to an uplink adjustment coefficient, a downlink adjustment coefficient, uplink data and downlink data of the data in the feature sequence.

[0062] Because data compression using the SDT compression algorithm is a backward compression process, which is performed based on the fluctuation trend of the data, the present invention analyzes the weighted average difference between the data in the feature sequence and its corresponding uplink data and downlink data, and uses this as the basic tolerance for data of different sizes in the feature sequence.

[0063] The size of the feature sequence is the first Data Take the data of as an example, calculate its basic tolerance, take the uplink adjustment coefficient of a data in the feature sequence as the weight of the average value of the difference between each uplink data and the data, take the downlink adjustment coefficient as the weight of the average value of the difference between the data and each downlink data, sum them up, and obtain the sum result; calculate the average value of the sum result, which is the basic tolerance of the data, and is expressed by the formula:

[0064]

[0065] in, Represents the basic tolerance of the jth data in the feature sequence; represents the upward adjustment coefficient; represents the downward adjustment factor; Indicates the jth data in the uplink data corresponding to the jth data Uplink data, represents the jth data in the feature sequence, and Greater than , Indicates the first j-th data in the downlink data corresponding to the j-th data Downlink data, and Less than ; and They represent the number of uplink data and downlink data corresponding to the j-th data, respectively. , .

[0066] It is used to represent the first Data The basic tolerance without considering other environmental influences, for example, at this time, the number of data of this size in the overall feature sequence is not considered, because All data in the feature sequence of the image to be compressed represent the continuous and non-repetitive grayscale values ​​of pixels in a channel of the image to be compressed, and the SDT compression algorithm is a backward compression algorithm, so the present invention uses all data in the feature sequence that are related to the first The difference calculation is performed between the backward adjacent data of the same data size and the data of the same size. The difference calculation is also and , and then take the average of the difference values ​​as the basic tolerance for data of this size in the feature sequence.

[0067] In the feature sequence, The backward adjacent data of data of the same size includes not only the backward adjacent data larger than the data, but also the backward adjacent data smaller than the data, that is, the uplink data and the downlink data of the data. Based on the characteristics of the SDT compression algorithm, they need to be processed separately, considering the backward adjacent data larger than and less than The product of the discrete degree and the proportion of the data volume of the backward adjacent data in the two cases is used as the weight to adjust the average value of the difference value, that is, the upward adjustment coefficient and the downward adjustment coefficient are used to adjust the average value of the difference value in the two cases. For example, the number of the backward continuous data greater than The more data there is, the larger the size The base tolerance of the data should be greater than The tolerance should be higher and more likely to be greater than The principle of discreteness is similar. The more discrete the data is, the more When the data is used as the compression point, the volatility of the subsequent data is stronger, so a larger tolerance should be given to it so that the compression efficiency is better when the data is used as the compression point, and vice versa. Therefore, the average value of the difference value is weighted according to the number and discreteness of the adjacent data, and finally the weighted average value is used as the size of the feature sequence. It should be noted that the basic tolerance of data of the same size in the feature sequence is the same.

[0068] So far, the change characteristics of the uplink data and downlink data of a data are analyzed, and the uplink adjustment coefficient and the downlink adjustment coefficient are obtained. The difference between the uplink data and the downlink data of a data and the subsequent analysis is adjusted to obtain a more appropriate basic tolerance for the data.

[0069] Step S4, obtaining the tolerance weight of a data according to the frequency of occurrence of the data in the feature sequence, the uplink data and the downlink data of the data.

[0070] The basic tolerance calculated in step S3 is for data of the same size in the feature sequence, without considering the actual image situation, so it needs to be corrected to obtain a tolerance with better compression effect.

[0071] For example, if repeated data appears more times in the feature sequence, its interval in the image is relatively small. When the gray value is used as the compression point, in order to have a better visual effect or a smaller distortion rate in the smaller interval after compression, the tolerance should be smaller when compressing it. Conversely, data that appears less times means that the interval between its corresponding gray value in the image is longer, so the tolerance should be larger when compressing it to ensure the corresponding compression rate. Therefore, for data of different sizes, it is necessary to calculate the corresponding tolerance weights to adjust the basic tolerance and obtain the final tolerance.

[0072] Specifically, the size of the feature sequence is Take data as an example, classify the same data in the feature sequence into one category, obtain the number of data in each category and the total number of categories; obtain the uplink data and the difference between the downlink data and the data; according to the data The frequency of occurrence in the feature sequence, the number of each type of data in the feature sequence, the total number of categories, the difference between the data and the The absolute value of the ratio of is used to obtain the tolerance weight of the data, which is expressed as follows:

[0073]

[0074] in, Represents the tolerance weight of the jth data in the feature sequence; Indicates the frequency of the jth data in the feature sequence; Statement The number of data in the class data, The size of the data in the class data is ; Indicates the total number of categories in the data sequence; and Respectively represent the amount of uplink data and the amount of downlink data of the data; Indicates the first individual data; Represents an exponential function with the natural constant e as the base.

[0075] in, It can characterize the amount of data in each category when evenly distributed. yes The deformation of The relative number of occurrences, the larger the value, the more similar the data in the feature sequence is. The more times it appears, the smaller the interval in the image to be compressed. The smaller the interval, the more we need to consider its corresponding grayscale sensitivity. That is, when using the grayscale value as the compression point, we need to reduce the information loss rate between them to compensate for the visual perception of the human eye, which means that a smaller tolerance is needed. Similarly, Representation data Uplink data and downlink data and data The larger the value, the richer the grayscale changes in the image. When this data is used as the compression point, the tolerance will also become smaller. The tolerance weight for the data The basic tolerance is adjusted so that the adjusted basic tolerance can improve the compression effect when the SDT compression algorithm is used to compress the image data.

[0076] Step S5, obtaining the final tolerance of each data based on the basic tolerance and tolerance weight of each data; compressing the data sequence corresponding to the image using the final tolerance of each data, and sharing the compressed data to each secondary server for distributed storage.

[0077] In steps S3 and S4, the basic tolerance size corresponding to each data in the feature sequence and its corresponding tolerance weight are calculated. Now the basic tolerance is adjusted using the tolerance weight to obtain the final tolerance size of the data in the feature sequence.

[0078] The size of the feature sequence is As an example, the first preset value and the size are obtained. The sum of the tolerance weights corresponding to the data is compared with the second preset value, and the result of the comparison is multiplied by the basic tolerance to obtain a value of The final tolerance corresponding to the data, which corresponds to the final tolerance The calculation formula is as follows:

[0079]

[0080] in, Indicates that the size of the feature sequence is The basic tolerance of the data, Indicates size The tolerance weight of the data is preferably, in this embodiment, the first preset value is 1, and the second preset value is 2; on the basis of the basic tolerance, the corresponding tolerance weight is used to adjust it. The purpose of adjustment is to reduce the variation range of the tolerance weight to prevent the occurrence of some special final tolerances that are too large or too small, resulting in the first When compressing an image, if the local compression rate is insufficient or the local information loss rate is large when compressing with the final tolerance, the implementer can adjust the values ​​of the first preset value and the second preset value according to the specific situation.

[0081] At this point, the final tolerance of each data in the feature sequence can be obtained. The change of the data is taken into account when calculating the final tolerance of each data. Because the data in the feature sequence is the grayscale value of each pixel point in a channel in the image, it is combined with the change of the grayscale of the pixel points in one channel of the image. Therefore, when the image data is compressed using the SDT algorithm based on the final tolerance of each data, the compression rate can be improved, and the distortion of the image will not affect the normal use of the image.

[0082] In addition, the final tolerances of the same data in the feature sequence are the same, and if the image has several channels of grayscale values, there are several one-dimensional data sequences, so several data sequences need to be compressed.

[0083] The final tolerance of each data of different sizes in the feature sequence is adaptively adjusted so that the same data has the most suitable final tolerance, and each data of different sizes in the feature sequence corresponds to the first The different grayscale values ​​in the image to be compressed are obtained. The final tolerance corresponding to each gray value in the image to be compressed is then compressed using the SD compression algorithm. The one-dimensional data sequence corresponding to the image to be compressed can be compressed.

[0084] Special, data sequence The Data of data The compressed data is , its final value as follows:

[0085]

[0086] Because after compression using the SDT compression algorithm, the Data The compression curve point at the point is used to replace the gray value of the point, but the gray value is an integer, but the compression value at the point may be a non-integer, so the above method is used to perform membership calculation and the closest integer membership can be used. Indicates rounding down. The image to be compressed is compressed, and the The compressed data of the image.

[0087] After obtaining the compressed data, it needs to be distributedly stored. The distributed file system used in the present invention is MooseFS. The implementer can also choose other distributed file systems. The specific distribution is: a metadata management server, a metadata log management server, a main server, multiple secondary servers and an NFS server. The working principle of all servers is as follows:

[0088] NFS server: acts as a user client to upload, modify and view image data;

[0089] Metadata management server: manages the metadata generated in the storage system;

[0090] Metadata log management server: logs all metadata and manages the logs;

[0091] Main server: mainly stores all data;

[0092] Secondary server: Shares the data of the primary server and backs up the distributed storage.

[0093] The overall workflow is as follows:

[0094] First, an index is established on the metadata server, and then the NFS server is used as a client to communicate with the metadata server. All user operations are based on the NFS server. For example, to upload an image file, the user uploads the image on the client, and then the RGB image in the NFS server is channel separated to obtain the grayscale value of each channel. Then, the above method for obtaining the final tolerance is used to obtain the adaptive final tolerance of each data, and then the SDT compression algorithm is used for compression.

[0095] Then, the compressed image data uploaded by the user is stored on the main server, and then the compressed image data is backed up and shared to multiple secondary servers to realize distributed storage.

[0096] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The above is a description of a specific embodiment of this specification. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0097] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0098] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the present invention should be included in the protection scope of the present invention.

Claims

1. A distributed compression storage method based on data sharing, characterized in that: The method includes: The grayscale value of a channel of each pixel in the image forms a one-dimensional data sequence; the redundant data in the data sequence is removed to obtain a feature sequence; Acquire uplink data and downlink data corresponding to a data of the characteristic sequence in the SDT compression algorithm process, and obtain an uplink adjustment coefficient and a downlink adjustment coefficient of the data based on the uplink data and the downlink data; Obtaining a basic tolerance of a data according to an upward adjustment coefficient, a downward adjustment coefficient, the upward data and the downward data of the data in the characteristic sequence; Obtain the tolerance weight of a data according to the frequency of occurrence of the data in the feature sequence, the upstream data and the downstream data of the data; The final tolerance of each data is obtained based on the basic tolerance and tolerance weight of each data; the final tolerance of each data is used to compress the data sequence corresponding to the image, and the compressed data is shared to each secondary server for distributed storage; The obtaining, based on the uplink data and the downlink data, an uplink adjustment coefficient and a downlink adjustment coefficient of the data includes: Obtain the ratio of the number of uplink data corresponding to a data in the feature sequence to the sum of the number of uplink data and the number of downlink data; obtain the reciprocal of the average value and the standard deviation of all uplink data; the product of the ratio, the reciprocal of the average value and the standard deviation is the first calculation parameter of the uplink data, and similarly obtain the second calculation parameter of the downlink data; obtain the result of adding the first calculation parameter and the second calculation parameter, the ratio of the first calculation parameter to the result of the addition is the uplink adjustment coefficient of the data, and the ratio of the second calculation parameter to the result of the addition is the downlink adjustment coefficient of the data; The method of obtaining the tolerance weight of a data according to the frequency of occurrence of a data in a feature sequence, the upstream data and the downstream data of the data, comprises: classifying the same data in the feature sequence into one category, obtaining the number of data in each category and the total number of categories; obtaining the difference between the upstream data and the downstream data and the data; obtaining the tolerance weight of the data according to the frequency of occurrence of a data, the number of data in each category in the feature sequence, the total number of categories, and the absolute value of the ratio of the difference to the data; The tolerance weights for the data are: in, Represents the tolerance weight of the jth data in the feature sequence; Indicates the frequency of the jth data in the feature sequence; Statement The number of data in the class data, The size of the data in the class data is ; Indicates the total number of categories in the data sequence; and Respectively represent the amount of uplink data and the amount of downlink data of the data; Indicates the first individual data; Represents an exponential function with the natural constant e as the base.

2. A distributed compression storage method based on data sharing according to claim 1, characterized in that: The method of obtaining uplink data and downlink data corresponding to a data in a characteristic sequence during an SDT compression algorithm includes: selecting a data, obtaining data in the characteristic sequence that is equal to the data, and recording it as equal data; obtaining data that is backward adjacent to each equal data and is greater than the equal data, and recording it as uplink data of the data; obtaining data that is backward adjacent to each equal data and is less than the equal data, and recording it as downlink data of the data.

3. A distributed compression storage method based on data sharing according to claim 1, characterized in that: The method of obtaining the basic tolerance of the data according to the uplink adjustment coefficient, downlink adjustment coefficient, uplink data and downlink data of a data in the feature sequence includes: taking the uplink adjustment coefficient of a data in the feature sequence as the weight of the average value of the difference between each uplink data and the data, and taking the downlink adjustment coefficient as the weight of the average value of the difference between the data and each downlink data, summing them up to obtain a summation result; and calculating the average value of the summation results as the basic tolerance of the data.

4. The distributed compression storage method based on data sharing according to claim 1 is characterized in that: The method of obtaining the final tolerance of each data based on the basic tolerance and the tolerance weight of each data includes: obtaining the sum of a first preset value and the tolerance weight, and multiplying the basic tolerance by the ratio of the sum to a second preset value to obtain the final tolerance.

5. The distributed compression storage method based on data sharing according to claim 1 is characterized in that: The removing redundant data in the data sequence to obtain the characteristic sequence includes: in the characteristic sequence, only one data in the continuously repeated data is retained, and the non-continuously repeated data is directly retained, thereby obtaining the characteristic sequence.

Images