File compression method, computer device and computer program product

Abstract

The application relates to a file compression method, a computer device and a computer program product, and relates to the field of data processing.The method comprises the following steps: acquiring a to-be-compressed file, performing clustering processing on each data in the to-be-compressed file to obtain a plurality of clustering clusters; establishing an index of the centroid of each clustering cluster, and determining the index of the centroid of the clustering cluster to which each data belongs as the index of each data; the index of each centroid is used for representing the sequence of the centroid in the plurality of centroids; the number of bytes occupied by each index is less than the number of bytes occupied by each data; and the indexes of the centroids and the data are combined to form a compressed file corresponding to the to-be-compressed file. The method can improve the download speed of the file.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a file compression method and device, computer equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] Files often need to be downloaded before they can be used, for example, audio files need to be downloaded before they can be played. However, the download speed of a file is affected by the size of the file, and if the size of the file is large, the download speed of the file will be slow. SUMMARY

[0003] Therefore, it is necessary to provide a file compression method, computer equipment and computer program product capable of improving the download speed of a file to solve the technical problem of slow download speed of a file.

[0004] In a first aspect, the present application provides a file compression method, comprising:

[0005] obtaining a to-be-compressed file, clustering each data in the to-be-compressed file to obtain a plurality of clustering clusters;

[0006] establishing an index of a centroid of each clustering cluster, and determining the index of the centroid of the clustering cluster to which each data belongs as an index of the each data; the index of each centroid is used to represent the order of the centroid in a plurality of centroids; and the number of bytes occupied by each index is less than the number of bytes occupied by each data;

[0007] grouping each centroid and the index of each data to form a compressed file corresponding to the to-be-compressed file.

[0008] In one embodiment, the establishing an index of a centroid of each clustering cluster comprises:

[0009] storing the centroid of each clustering cluster into a first data set; the number of bytes occupied by each centroid is equal to the number of bytes occupied by each data;

[0010] determining the index of each centroid according to the storage order of each centroid in the first data set;

[0011] The grouping each centroid and the index of each data to form a compressed file corresponding to the to-be-compressed file comprises:

[0012] storing the index of each data into a second data set in sequence according to the storage order of each data in the to-be-compressed file;

[0013] combining the first data set and the second data set to obtain the compressed file corresponding to the to-be-compressed file.

[0014] In one of the embodiments, the combining the first data set and the second data set to obtain the compressed file corresponding to the to-be-compressed file further comprises:

[0015] determining a total number of preset centroids for the clustering processing;

[0016] combining the total number of preset centroids, the first data set and the second data set in sequence to obtain the compressed file.

[0017] In one of the embodiments, the number of bytes occupied by each index is at least the number of bytes required for representing the total number of preset centroids for the clustering processing in binary system.

[0018] In one of the embodiments, the clustering processing of each data in the to-be-compressed file to obtain a plurality of clustering clusters comprises:

[0019] initializing a plurality of centroids according to the data lower limit value and the data upper limit value of each data and a data distribution interval corresponding to the file type to which the to-be-compressed file belongs; the data distribution interval is used to represent the distribution of the data set of the file type;

[0020] performing clustering processing on the each data based on the plurality of initialized centroids to obtain the plurality of clustering clusters.

[0021] In one of the embodiments, the number of the plurality of initialized centroids is a preset number.

[0022] The initializing a plurality of centroids according to the data lower limit value and the data upper limit value of each data and a data distribution interval corresponding to the file type to which the to-be-compressed file belongs comprises:

[0023] initializing a first number of centroids according to the data lower limit value and the data upper limit value and an interval lower limit value and an interval upper limit value of the data distribution interval;

[0024] uniformly initializing a second number of centroids in the data distribution interval; the second number is obtained by subtracting the first number from the total number of preset centroids;

[0025] determining the first number of centroids and the second number of centroids as the plurality of initialized centroids.

[0026] In one of the embodiments, the initializing a first number of centroids according to the data lower limit value and the data upper limit value and an interval lower limit value and an interval upper limit value of the data distribution interval comprises:

[0027] initialize a centroid as an average of the data lower limit value and the interval lower limit value;

[0028] initialize a centroid as an average of the data lower limit value and the interval lower limit value in a case where the data lower limit value is less than or equal to the interval lower limit value, and initialize a centroid as an average of the interval lower limit value and a midpoint value of the data distribution interval in a case where the data lower limit value is greater than the interval lower limit value;

[0029] initialize a centroid as an average of the data upper limit value and the interval upper limit value in a case where the data upper limit value is greater than or equal to the interval upper limit value, and initialize a centroid as an average of the interval upper limit value and the midpoint value in a case where the data upper limit value is less than the interval upper limit value.

[0030] In one embodiment, the method further comprises:

[0031] for each file type, obtaining a plurality of files under the file type;

[0032] counting data distribution in each file to obtain a data distribution interval for representing data distribution in the data set of the file type.

[0033] In a second aspect, the present application further provides a computer device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0034] obtaining a to-be-compressed file, performing clustering processing on each data in the to-be-compressed file to obtain a plurality of clustering clusters;

[0035] establishing an index of a centroid of each clustering cluster, and determining an index of a centroid of a clustering cluster to which each data belongs as an index of the each data; the index of each centroid is used to represent an order of the centroid in a plurality of centroids; and a number of bytes occupied by each index is less than a number of bytes occupied by each data;

[0036] grouping the indexes of the data and the indexes of the centroids to form a compressed file corresponding to the to-be-compressed file.

[0037] In a third aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the following steps:

[0038] obtaining a to-be-compressed file, performing clustering processing on each data in the to-be-compressed file to obtain a plurality of clustering clusters;

[0039] An index of the centroid of each cluster is established, and an index of the centroid of the cluster to which each data belongs is determined as the index of the data; the index of each centroid is used to represent the order of the centroid in the plurality of centroids; and the number of bytes occupied by each index is less than the number of bytes occupied by each data.

[0040] The indexes of the respective centroids and the respective data are combined to form the compressed file corresponding to the to-be-compressed file.

[0041] The file compression method, the computer device, and the computer program product, first, a to-be-compressed file is obtained, and each data in the to-be-compressed file is clustered to obtain a plurality of cluster groups; then, an index of the centroid of each cluster group is established, and an index of the centroid of the cluster to which each data belongs is determined as the index of the data; the index of each centroid is used to represent the order of the centroid in the plurality of centroids; and the number of bytes occupied by each index is less than the number of bytes occupied by each data; and then, the indexes of the respective centroids and the respective data are combined to form the compressed file corresponding to the to-be-compressed file. Based on the file compression method, the index of each data is obtained through the clustering of the data and the establishment of the index of the centroid of the cluster group, and since the index of each data points to the centroid of the cluster group to which the data belongs, the information recorded in the file is changed from the data itself to the index of the centroid of the cluster group to which the data belongs and the centroid pointed to by the index before and after compression. Since the number of data is often large in actual business, the volume occupied by each centroid in the compressed file can be ignored, that is, the volume of the compressed file mainly considers the volume occupied by the index of each data, and the number of bytes occupied by each index is less than the number of bytes occupied by each data, so the volume of the compressed file is reduced compared with the to-be-compressed file, and the download speed of the file is improved. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other related drawings can be obtained by those skilled in the art without any creative effort.

[0043] Figure 1 A flowchart of the file compression method in one embodiment;

[0044] Figure 2 A flowchart of the step of establishing the index of the centroid of each cluster in one embodiment;

[0045] Figure 3A flowchart of a step of grouping the indexes of the respective centroids and the respective data into a compressed file corresponding to the to-be-compressed file in an embodiment;

[0046] Figure 4 A flowchart of a step of combining the first data set and the second data set to obtain a compressed file corresponding to the to-be-compressed file in an embodiment;

[0047] Figure 5 A comparison diagram of the to-be-compressed file and the compressed file in an embodiment;

[0048] Figure 6 A flowchart of a step of performing clustering processing on the respective data in the to-be-compressed file to obtain a plurality of clustering clusters in an embodiment;

[0049] Figure 7 A flowchart of a step of initializing a plurality of centroids according to the data lower limit value and the data upper limit value of the respective data and the data distribution interval corresponding to the file type to which the to-be-compressed file belongs in an embodiment;

[0050] Figure 8 A flowchart of a step of initializing the first number of centroids according to the data lower limit value and the data upper limit value and the interval lower limit value and the interval upper limit value of the data distribution interval in an embodiment;

[0051] Figure 9 A flowchart of a file compression method in another embodiment;

[0052] Figure 10 A block diagram of a file compression device in an embodiment;

[0053] Figure 11 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0055] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0056] In an embodiment, as shown in FIG. 1, a file compression method includes the following steps: Figure 1As shown, a file compression method is provided, and the embodiment takes the method applied to a server as an example. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a server and a terminal, and is realized through the interaction of the server and the terminal. The server can be a stand-alone physical server, a server cluster or a distributed system formed by multiple physical servers, or a cloud server providing cloud computing services. The terminal can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and the like. In the embodiment, the method includes the following steps:

[0057] In step S102, a to-be-compressed file is obtained, and each data in the to-be-compressed file is clustered to obtain a plurality of clustering clusters.

[0058] The file type to which the to-be-compressed file belongs is a file type with similar data distribution and relatively concentrated data distribution for each corresponding file. In actual application, the to-be-compressed file is a song voice posterior probability file, and the song voice posterior probability file can be used for song generation, tone correction and other applications in a song synthesis scenario.

[0059] As can be easily understood, binary is a number system widely used by computers for data storage and processing. Therefore, the to-be-compressed file and the compressed file in the present application are binary files, and the data, centroid, index and the like in the to-be-compressed file in the present application are also binary data. However, for the convenience of understanding, decimal data is used in one or more embodiments described below.

[0060] Specifically, the server obtains a to-be-compressed file, and clusters each data in the to-be-compressed file to cluster the data into a plurality of clustering clusters. Each clustering cluster has at least one data, and each clustering cluster has a corresponding centroid for representing the average level of each data in the clustering cluster.

[0061] For example, it is assumed that each data in the to-be-compressed file is {5, 23, 6, 15, 33, 17, 4, 28}. The server clusters the above data to obtain clustering cluster 1 {5, 6, 4}, clustering cluster 2 {15, 17} and clustering cluster 3 {23, 33, 28}, wherein the centroid of clustering cluster 1 is 5, the centroid of clustering cluster 2 is 16, and the centroid of clustering cluster 3 is 28.

[0062] In specific application, the data in the to-be-compressed file can exist in the form of a multi-dimensional array. Therefore, the server can regard each multi-dimensional array element in the multi-dimensional array as a data, or can perform a flatten operation on the multi-dimensional array in the to-be-compressed file, convert the multi-dimensional array into a corresponding one-dimensional array, and then regard each array element in the one-dimensional array as a data.

[0063] Step S104, establishing the index of the centroid of each cluster, determining the index of the centroid of the cluster to which each data belongs as the index of each data.

[0064] Wherein, the index of each centroid is used to represent the order of the centroid in the plurality of centroids.

[0065] Wherein, the number of bytes occupied by each index is less than the number of bytes occupied by each data.

[0066] Wherein, the number of bytes occupied by each data is related to the precision of the data, for example, if the data is fp32 (single-precision floating-point number), the number of bytes occupied by each data is 4 bytes (bytes), that is, 32 bits (bits), if the data is fp16 (half-precision floating-point number), the number of bytes occupied by each data is 2 bytes, that is, 16 bits.

[0067] Wherein, the number of bytes occupied by a single index is at least the number of bytes needed to represent a total number of preset centroids for clustering processing in binary system Different integers; for example, assuming that the total number of centroids is 256, then the number of centroid indexes is also 256, since computers store data in binary system, and in binary system, if 256 different integers need to be represented, at least 1 byte is needed, therefore, the number of bytes occupied by each index is 1 byte.

[0068] In specific applications, since the smallest unit of computer data storage is bit, and computers store data in binary system, the total number of centroids is preset to be a positive integer power of 2, so as to make full use of each bit and facilitate reading by other servers or terminals.

[0069] Specifically, first, for each centroid, the server determines the order of the centroid in the plurality of centroids, and determines the order as the index of the centroid, then, for each data, the server determines the index of the centroid of the cluster to which the data belongs as the index of the data.

[0070] As easily understood, determining the index of the centroid of the cluster to which each data belongs as the index of each data means that the index of each data is the index of the corresponding centroid (the centroid of the cluster to which the data belongs), that is, for each data, its index can point to its corresponding centroid, therefore, the server can determine the centroid corresponding to the data according to the index of the data.

[0071] In actual applications, the order of the plurality of centroids can be obtained in any way, for example, the order of the plurality of centroids is from large to small or from small to large according to the centroid value.

[0072] For example, assuming that the centroid of the cluster 1 {5, 1, 4} is 5, the centroid of the cluster 2 {15, 17} is 16, the centroid of the cluster 3 {23, 33, 28} is 28, and the order of the three centroids is {5, 16, 28}. Then, the server determines the index of the centroid 5 of the cluster 1 as "0", the index of the centroid 16 of the cluster 2 as "1", and the index of the centroid 28 of the cluster 3 as "2"; and then, the server determines the index of the data 5, the data 1 and the data 4 in the cluster 1 as "0", the index of the data 15 and the data 17 in the cluster 2 as "1", and the index of the data 22, the data 33 and the data 28 in the cluster 3 as "2".

[0073] In step S106, the index of each centroid and each data is grouped into the compressed file corresponding to the to-be-compressed file.

[0074] Specifically, the server first stores the centroid of each cluster in a blank file according to the order of the plurality of centroids, and then stores the index of each data behind the centroid of each cluster according to the order of each data in the to-be-compressed file, thereby obtaining the compressed file corresponding to the to-be-compressed file.

[0075] For example, the server first stores the centroids {5, 16, 28} of the cluster 1, the cluster 2 and the cluster 3 in a blank file, and then stores the index {0, 2, 0, 1, 2, 1, 0, 2} of each data behind, thereby obtaining the compressed file "{5, 16, 28}, {0, 2, 0, 1, 2, 1, 0, 2}".

[0076] In this way, when the computer parses the compressed file, the first data can be read as 5, the second data can be read as 28, and the third data can be read as 5, and so on. As can be easily understood, the amount of data in the to-be-compressed file is often large enough, and in the case where the data is distributed relatively concentratedly and the number of clusters is large enough, the difference between the centroid of each cluster and each data in the cluster will be small enough, so that the loss of accuracy of the to-be-compressed file compared with the to-be-compressed file can be ignored.

[0077] In actual application, since the expression of each data in the compressed file is converted from the data itself to the centroid corresponding to the data, in order to minimize the loss of accuracy before and after file compression, preferably, the number of bytes occupied by each centroid in the compressed file is equal to the number of bytes occupied by each data.

[0078] For example, assuming that there are 8 data in the to-be-compressed file, and the number of bytes occupied by each data is Byte, then the volume of the to-be-compressed file is Byte. Byte. Bytes.

[0079] Assuming that the total number of centroids is preset to and the number of bytes occupied by each centroid is equal to the number of bytes occupied by each data, which is also Bytes, then the volume occupied by each centroid in the compressed file is Bytes. Assuming that in the binary system, at least Bytes are required to represent different integers, that is, the number of bytes occupied by a single index is Bytes, then the volume occupied by the index of data in the compressed file is Bytes. In summary, the volume of the compressed file is Bytes. In actual business, the volume is often very large, so the volume occupied by each centroid Bytes can be ignored, that is, the volume of the compressed file is considered to be Bytes, and is less than , so the volume of the compressed file is reduced compared to the file to be compressed.

[0080] In the above file compression method, first, the server obtains the file to be compressed, performs clustering processing on each data in the file to be compressed, and obtains a plurality of clustering clusters; then, the server establishes an index of the centroid of each clustering cluster, and determines the index of the centroid of the clustering cluster to which each data belongs as the index of each data; the index of each centroid is used to represent the order of the centroid in the plurality of centroids; the number of bytes occupied by each index is less than the number of bytes occupied by each data; then, the server groups each centroid and the index of each data to form a compressed file corresponding to the file to be compressed. Based on the above process of the file compression method, through the clustering processing of the data and the establishment of the index of the centroid of the clustering cluster, the server can obtain the index of each data, since the index of each data points to the centroid of the clustering cluster to which the data belongs, therefore, before and after compression, the server converts the information recorded in the file from the data itself to the index pointing to the centroid of the clustering cluster to which the data belongs and the centroid pointed to by the index; since in actual business, the number of data is often large, therefore, the volume occupied by each centroid of the clustering cluster in the compressed file can be ignored, that is, the volume of the compressed file mainly considers the volume occupied by the index of each data, and the number of bytes occupied by each index is less than the number of bytes occupied by each data, therefore, the volume of the compressed file is reduced compared to the file to be compressed, and the download speed of the file is improved.

[0081] In an exemplary embodiment, as Figure 2As shown, the step S104 above, which involves establishing the index of the centroid of each cluster, specifically includes the following steps:

[0082] Step S202: Store the centroids of each cluster into the first data set.

[0083] Step S204: Determine the index of each centroid according to its storage order in the first data set.

[0084] It is easy to understand that, since the representation of each data point in the compressed file is transformed from the data itself into the centroid corresponding to the data, the precision of the centroid is preferably consistent with the precision of the data in order to minimize the loss of precision before and after file compression. Therefore, in this embodiment, the number of bytes occupied by each centroid is equal to the number of bytes occupied by each data point.

[0085] Specifically, the server stores the centroids of each cluster in a blank first data set, resulting in the first data set {5, 16, 28}. The storage order of each centroid in the first data set is used as the index of that centroid. For example, the index of centroid 5 is 0, the index of centroid 16 is 1, and the index of centroid 28 is 2.

[0086] In practical applications, the first data set is a LUT (lookup table), meaning that the server stores the centroids of each cluster in the LUT.

[0087] like Figure 3 As shown, step S106 above, which combines the indices of each centroid and each data point to form the compressed file corresponding to the file to be compressed, specifically includes the following steps:

[0088] Step S302: Store the indexes of each data item in the second data set according to the storage order of each data item in the file to be compressed.

[0089] Step S304: Combine the first data set and the second data set to obtain the compressed file corresponding to the file to be compressed.

[0090] Specifically, the server stores the indices of each data item in the blank second data set in the order they are stored in the file to be compressed: {5, 23, 6, 15, 33, 17, 4, 28}. This results in the second data set {0, 2, 0, 1, 2, 1, 0, 2}. Then, the server combines the first data set {5, 16, 28} and the second data set {0, 2, 0, 1, 2, 1, 0, 2} to obtain the compressed file "{5, 16, 28}, {0, 2, 0, 1, 2, 1, 0, 2}".

[0091] In this embodiment, since the index of each data points to the centroid of the cluster to which the data belongs, the server transforms the information recorded in the file from the data itself into an index pointing to the centroid of the cluster to which the data belongs and the centroid pointed to by the index. That is, the storage of the data itself is transformed into the storage of the centroid and the index of the centroid. Since the amount of data is often large, the volume occupied by each centroid in the compressed file can be ignored. That is, the volume of the compressed file mainly considers the volume occupied by the index of each data. The number of bytes occupied by each index is less than the number of bytes occupied by each data. Therefore, the volume of the compressed file is reduced compared to the file to be compressed, thereby improving the download speed of the file.

[0092] In one exemplary embodiment, such as Figure 4 As shown, step S304 above, which combines the first data set and the second data set to obtain the compressed file corresponding to the file to be compressed, further includes the following steps:

[0093] Step S402: Determine the total number of centroids preset for clustering processing.

[0094] Step S404: Combine the total number of centroids, the first data set, and the second data set in sequence to obtain the compressed file.

[0095] Specifically, the server determines the preset total number of centroids for clustering processing. Then, the server combines the total number of centroids, the first data set, and the second data set in the order of the first data set and the second data set to obtain a compressed file.

[0096] It's easy to understand that the number of bytes required to represent the total number of centroids is equal to the number of bytes required to represent the total number of integer centroids. For example, when there are 256 indices, the number of bytes required to represent 256 is also 1 byte, therefore, the number of bytes occupied by the total number of centroids 256 is also 1 byte.

[0097] For example, suppose the file to be compressed contains... There are 4 data points, each occupying 4 bytes. Therefore, the size of the file to be compressed is... Byte.

[0098] Assuming a total of 256 centroids, each index occupies 1 byte, and the total number of centroids also occupies 1 byte. Furthermore, assuming the number of bytes occupied by each centroid is equal to the number of bytes occupied by each data element, this is also... Byte. Therefore, in the compressed file, the total number of centroids occupies 1 Byte, and the volume occupied by each centroid is... Byte, The size occupied by the index of each data is Byte, thus obtaining the size of the compressed file as Byte. In actual business operations, The file size is often very large, therefore the compressed file size is considered... Bytes, compressed into the file to be compressed This results in a significant reduction.

[0099] See Figure 5 This is a comparison diagram of the file to be compressed and the compressed file.

[0100] In this embodiment, by combining the total number of centroids, the first data set, and the second data set in sequence to obtain the compressed file, other servers or terminals can facilitate subsequent reading of the first data set and the second data set by reading the total number of centroids at the beginning of the compressed file when parsing the compressed file (dividing the volume occupied by the first data set by the total number of centroids can determine the number of bytes occupied by a single centroid in the compressed file, thus facilitating the reading of the centroids; the total number of centroids can determine the number of bytes occupied by a single index, thus facilitating the reading of the index).

[0101] In one exemplary embodiment, the number of bytes occupied by each index in one or more embodiments of this application is, in binary number system, the minimum number of bytes required for a preset total number of centroids for clustering processing.

[0102] As is easily understood, the index is the index of the centroid. Therefore, the number of integers needed to represent the centroid indices corresponds to the number of centroids. Thus, the number of bytes occupied by each index is the minimum number of bytes required to represent the total number of centroid integers in binary. For example, assuming there are 256 centroids, then 256 integers are needed to represent the centroid indices. In binary, if at least 1 byte is needed to represent 256 different integers, then each index occupies 1 byte.

[0103] In this embodiment, the number of bytes occupied by each index is the minimum number of bytes required to represent the total number of integer centroids in binary number system. This can minimize the number of bytes occupied by a single index, thereby minimizing the size of the compressed file.

[0104] In one exemplary embodiment, such as Figure 6 As shown, in step S102 above, clustering is performed on the data in the file to be compressed to obtain multiple clusters. This specifically includes the following steps:

[0105] In step S602, the plurality of centroids are initialized according to the data lower limit value and the data upper limit value of each data, and the data distribution interval corresponding to the file type to which the file to be compressed belongs.

[0106] In step S604, the clustering processing is performed on each data based on the plurality of initialized centroids, and a plurality of clustering clusters are obtained.

[0107] The data distribution interval is used to represent the distribution of the data in the file type to which the data belongs. In a specific application, the data distribution interval is the interval in which the data in the file type to which the data belongs is distributed. For example, taking a song voice posterior probability file as an example, the data in the voice posterior probability files of different songs is basically consistent, close to a normal distribution, and the data is basically distributed in the interval [-3, 3]. Therefore, for the file type of the song voice posterior probability file, the corresponding data distribution interval is [-3, 3].

[0108] Specifically, before clustering, the server first determines the data lower limit value and the data upper limit value of each data in the file to be compressed, and obtains the data distribution interval corresponding to the file type to which the file to be compressed belongs. Then, the server initializes a plurality of centroids based on the data lower limit value, the data upper limit value, and the data distribution interval. Based on the plurality of initialized centroids, the server performs clustering processing on each data using a preset clustering algorithm, and obtains a plurality of clustering clusters.

[0109] In actual applications, the preset clustering algorithm can be any clustering algorithm capable of clustering discrete data. In specific applications, preferably, the present application uses a k-means clustering algorithm to cluster each data, and the specific process is as follows: after initializing the centroids, (1) for each data, the server calculates the Euclidean distance between the data and each centroid, and assigns the data to the clustering cluster corresponding to the nearest centroid; (2) after each data is assigned, for each clustering cluster, the server calculates the average value of the data in the clustering cluster, and takes the average value as the new centroid of the clustering cluster; repeat steps (1) and (2) until each clustering cluster converges or the clustering reaches the maximum iteration number, and end the clustering. The finally obtained each clustering cluster is taken as the plurality of clustering clusters corresponding to the clustering result.

[0110] In the present embodiment, the server initializes the centroids by the data lower limit value, the data upper limit value, and the data distribution interval, rather than randomly initializing the centroids, which can accelerate the calculation of the centroids and further accelerate the clustering processing. The server can divide each data into a plurality of clustering clusters through the clustering processing on each data, and further convert the expression of the data from the data itself to the centroid of the clustering cluster to which the data belongs.

[0111] In an exemplary embodiment, as shown inFigure 7 As shown, the plurality of centroids are initialized according to the data lower limit value and the data upper limit value of each data, and the data distribution interval corresponding to the file type to which the file to be compressed belongs, and specifically include the following steps:

[0112] In step S702, the first number of centroids are initialized according to the data lower limit value and the data upper limit value, and the interval lower limit value and the interval upper limit value of the data distribution interval.

[0113] In step S704, the second number of centroids are initialized uniformly within the data distribution interval.

[0114] In step S706, the first number of centroids and the second number of centroids are determined as the initialized plurality of centroids.

[0115] The second number is obtained by subtracting the first number from the total number of centroids determined in advance.

[0116] Specifically, first, the server determines the total number of centroids; then, the server determines the interval lower limit value and the interval upper limit value of the data distribution interval; then, the server initializes the first number of centroids according to the data lower limit value, the data upper limit value, the interval lower limit value, and the interval upper limit value; finally, the server initializes the remaining second number of centroids uniformly within the data distribution interval; in this way, the first number of centroids and the second number of centroids are the plurality of initialized centroids.

[0117] In actual application, the first number is 4.

[0118] For example, assuming that the total number of centroids is 256, the server first initializes 4 centroids according to the data lower limit value, the data upper limit value, the interval lower limit value, and the interval upper limit value, and then initializes the remaining 251 centroids uniformly within the data distribution interval.

[0119] In this embodiment, the server initializes the plurality of centroids through the actual distribution of each data of the file to be compressed (the data lower limit value and the data upper limit value), and the distribution in the data set of the file type to which the file to be compressed belongs (the interval lower limit value and the interval upper limit value), which can accelerate the centroid calculation in the clustering process, and further accelerate the clustering process.

[0120] In an exemplary embodiment, as shown in Figure 8 The step S702 of initializing the first number of centroids according to the data lower limit value and the data upper limit value, and the interval lower limit value and the interval upper limit value of the data distribution interval specifically includes the following steps:

[0121] In step S802, the data lower limit value and the data upper limit value are initialized as one centroid respectively.

[0122] Step S804, if the data lower limit value is less than or equal to the interval lower limit value, the average of the data lower limit value and the interval lower limit value is initialized as a centroid; if the data lower limit value is greater than the interval lower limit value, the average of the interval lower limit value and the midpoint value of the data distribution interval is initialized as a centroid.

[0123] Step S806, if the data upper limit value is greater than or equal to the interval upper limit value, the average of the data upper limit value and the interval upper limit value is initialized as a centroid; if the data upper limit value is less than the interval upper limit value, the average of the interval upper limit value and the midpoint value is initialized as a centroid.

[0124] Specifically, the server first initializes the data lower limit value and the data upper limit value as a centroid respectively.

[0125] Then, the server compares the data lower limit value and the interval lower limit value, if the data lower limit value is less than or equal to the interval lower limit value, the average of the data lower limit value and the interval lower limit value is calculated and initialized as a centroid; if the data lower limit value is greater than the interval lower limit value, the average of the data lower limit value and the midpoint value of the data distribution interval is calculated and initialized as a centroid.

[0126] For example, assuming that the data distribution interval is [-3, 3], the interval lower limit value is -3 and the midpoint value of the data distribution interval is 0; if the data lower limit value is -4, the server initializes -3.5 as a centroid; if the data lower limit value is -1, the server initializes -1.5 as a centroid.

[0127] Next, the server compares the data upper limit value and the interval upper limit value, if the data upper limit value is greater than or equal to the interval upper limit value, the average of the data upper limit value and the interval upper limit value is calculated and initialized as a centroid; if the data upper limit value is less than the interval upper limit value, the average of the data upper limit value and the midpoint value of the data distribution interval is calculated and initialized as a centroid.

[0128] For example, assuming that the data distribution interval is [-3, 3], the interval upper limit value is 3 and the midpoint value of the data distribution interval is 0; if the data upper limit value is 4, the server initializes 3.5 as a centroid; if the data lower limit value is 2, the server initializes 1.5 as a centroid.

[0129] In this embodiment, the server initializes four centroids based on the data lower limit value and the data upper limit value, and the data lower limit value and the data upper limit value and the interval lower limit value and the interval upper limit value, which can accelerate the centroid calculation in the clustering process, and further accelerate the clustering process.

[0130] In one example embodiment, the file compression method provided by the present application further includes the following: for each file type, obtaining a plurality of files under the file type; and counting data distribution in each file to obtain a data distribution interval representing data distribution in the data set of the file type.

[0131] Specifically, the file compression method provided by the present application can compress files of multiple file types; the server obtains a plurality of files under each file type, counts data distribution in each file, and obtains an interval representing data distribution in the data set of the file type as the data distribution interval corresponding to the file type.

[0132] In actual applications, the file type applicable to the file compression method provided by the present application is a file type having similarity in data distribution of corresponding files and concentrated data distribution, for example, a song voice posterior probability file.

[0133] In this embodiment, the server counts data distribution of files of the same file type to obtain a data distribution interval representing data distribution in the data set of the file type, and based on the data distribution interval, can accelerate the calculation of the centroid in the clustering process.

[0134] In one example embodiment, as shown in Figure 9 Another file compression method is provided, which is described by taking the server as an example and includes the following steps:

[0135] Step S902, obtaining a file to be compressed.

[0136] Step S904, initializing a lower limit value and an upper limit value of data in each data of the file to be compressed as a centroid, respectively.

[0137] Step S906, in a case where the lower limit value of data is less than or equal to a lower limit value of the interval of the data distribution interval, initializing an average value of the lower limit value of data and the lower limit value of the interval as a centroid, and in a case where the lower limit value of data is greater than the lower limit value of the interval, initializing an average value of the lower limit value of the interval and a midpoint value of the data distribution interval as a centroid.

[0138] Step S908, in a case where the upper limit value of data is greater than or equal to an upper limit value of the interval of the data distribution interval, initializing an average value of the upper limit value of data and the upper limit value of the interval as a centroid, and in a case where the upper limit value of data is less than the upper limit value of the interval, initializing an average value of the upper limit value of the interval and the midpoint value as a centroid.

[0139] Step S910, initializing a remaining number of centroids in the total number of predetermined centroids uniformly within the data distribution interval.

[0140] In step S912, the first number of centroids and the second number of centroids are determined as initialized centroids, and each data is clustered based on the initialized centroids to obtain a plurality of clustering clusters.

[0141] In step S914, the centroid of each clustering cluster is stored in the first data set, and the index of each centroid is determined according to the storage order of each centroid in the first data set.

[0142] In step S916, the index of the centroid of the clustering cluster to which each data belongs is determined as the index of each data.

[0143] In step S918, the index of each data is sequentially stored in the second data set according to the storage order of each data in the file to be compressed.

[0144] In step S920, the total number of centroids is determined.

[0145] In step S922, the total number of centroids, the first data set and the second data set are combined in sequence to obtain a compressed file.

[0146] In this embodiment, first, the server initializes the centroids by the data lower limit value, the data upper limit value and the data distribution interval, rather than randomly initializing the centroids, which can speed up the calculation of the centroids and further speed up the clustering process. Second, since the index of each data points to the centroid of the clustering cluster to which the data belongs, the server converts the information of the file record from the data itself to the index pointing to the centroid of the clustering cluster to which the data belongs and the centroid pointed by the index, i.e. converts the storage of the data itself to the storage of the centroid and the index of the centroid. Since the number of data is often large, the volume occupied by each centroid in the compressed file can be ignored, i.e. the volume of the compressed file mainly considers the volume occupied by each index, and the number of bytes occupied by each index is less than the number of bytes occupied by each data, so the volume of the compressed file is reduced compared with the file to be compressed. Third, by combining the total number of centroids, the first data set and the second data set in sequence to obtain the compressed file, other servers or other terminals can read the total number of centroids at the front of the compressed file when parsing the compressed file, which facilitates the subsequent reading of the first data set and the second data set.

[0147] In order to more clearly illustrate the file compression method provided by the embodiments of the present application, the file compression method will be specifically described in a specific embodiment, but it should be understood that the embodiments of the present application are not limited thereto. In one of the exemplary embodiments, the present application also provides a non-uniform discretization compression method based on k-means clustering, which specifically includes the following contents:

[0148] In the end-to-end inference process of the song synthesis scene, the generation of each song depends on a specific template file; among them, the largest resource file is the speech posterior probability file of the song. The precision of each data in the speech posterior probability file is fp32, that is, each data occupies 4 bytes. If a certain precision loss is accepted, the precision of the data can be compressed to fp16, so that the volume of the speech posterior probability file can be reduced by half.

[0149] In order to further compress the volume of the speech posterior probability file on the basis of fp16, the embodiment proposes a non-uniform discretization compression method based on k-means clustering implementation. The compression principle is to combine the characteristics of the data distribution in the speech posterior probability file, further discretize the original data, replace the data with the corresponding centroid, and ensure the distribution trend and precision of the original data as much as possible with the non-uniformly distributed centroids.

[0150] In this embodiment, the server first presets the total number of centroids in the clustering process. Since the smallest unit of computer storage data is bit, and computer storage data is mostly binary, the total number of centroids is determined as a positive integer power of 2 to fully utilize each bit and facilitate reading by other servers or terminals. Since the basic unit of computer storage data is Byte, the total number of centroids is finally determined as the number of integers that can be expressed by 1 Byte, which is 256.

[0151] The server finds that the data distribution of the speech posterior probability files of different songs is basically the same, close to the normal distribution, and concentrated in the interval [-3, 3] by statistically analyzing the data distribution of the speech posterior probability files of different songs.

[0152] For each speech posterior probability file to be compressed, in order to speed up the calculation of the centroids in the clustering process, the server does not initialize the centroids randomly, but initializes them according to the following strategy:

[0153] (1) Calculate the lower limit value and the upper limit value of the data in the speech posterior probability file to be compressed, and assign one centroid to the lower limit value and one centroid to the upper limit value.

[0154] (2) If the lower limit value is less than or equal to -3, calculate the average of the lower limit value and -3, and assign a centroid to the average; if the lower limit value is greater than -3, assign a centroid to -1.5.

[0155] (3) If the upper limit value is greater than or equal to 3, calculate the average of the upper limit value and 3, and assign a centroid to the average; if the upper limit value is less than 3, assign a centroid to 1.5.

[0156] (4) The remaining 251 centroids are evenly distributed within [-3, 3].

[0157] After initializing the centroids, the server uses the k-means clustering algorithm to cluster each data in the compressed speech posterior probability file to obtain a plurality of clustering clusters.

[0158] The server stores the centroid of each clustering cluster in the lut. In this embodiment, in order to retain the accuracy of the centroid as much as possible, the server still determines the number of bytes occupied by each centroid as 4 bytes.

[0159] In the compressed speech posterior probability file, the server stores the total number of determined centroids 256 in the first byte, which facilitates the parsing of the compressed speech posterior probability file by other servers or terminals; then stores the lut, and finally stores the index of the centroid of the clustering cluster to which each data belongs, to replace the data itself.

[0160] Suppose that a speech posterior probability file has a total of 1000 data, then the volume of the original speech posterior probability file is 1000*4 = 4000 Byte.

[0161] After compression, the total number of centroids 256 occupies one byte; the lut occupies 256*4 = 1024 Byte; since the number of indexes (i.e. the total number of centroids) is 256, each index occupies 1 byte, i.e. the number of bytes occupied by a single index is 1 byte, and thus the volume occupied by the indexes of 1000 data is 1000*1 = 1000 Byte; in summary, after compression, the volume of the speech posterior probability file is 1+1024+1000 = 2045 Byte. In actual business, the volume of the speech posterior probability file is often very large, so 2045 Byte can be ignored, and the volume after compression is regarded as 1024 Byte, and the final compression effect is 25% of the original.

[0162] In this embodiment, in combination with the distribution characteristics of the data in the speech posterior probability file, the originally discrete data is further discretized, the data is replaced by the corresponding centroid, and the non-uniformly distributed centroid can guarantee the distribution trend and accuracy of the original data as much as possible, thereby realizing the compression of the speech posterior probability file and reducing the download time of the speech posterior probability file and the bandwidth cost required for transmission.

[0163] ​​​​​​​​​It should be understood that although the steps in the flowcharts involved in the embodiments described above are shown in sequence according to the arrows, the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of the steps is not strictly limited in sequence, and the steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the embodiments described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of the steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps.

[0164] Based on the same inventive concept, the embodiments of the present application also provide a file compression device for implementing the above-mentioned file compression method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more file compression device embodiments provided below can refer to the limitations of the file compression method described above, which will not be repeated here.

[0165] In an exemplary embodiment, as shown in Figure 10 a file compression device is provided, comprising: a data clustering module 1002, an index determination module 1004, and a file compression module 1006, wherein:

[0166] The data clustering module 1002 is configured to obtain a to-be-compressed file, perform clustering processing on each data in the to-be-compressed file, and obtain a plurality of clustering clusters.

[0167] The index determination module 1004 is configured to establish an index of each centroid of the clustering clusters, determine an index of the centroid of the clustering cluster to which each data belongs as an index of each data, and use the index of each centroid to represent the order of the centroid in the plurality of centroids. The number of bytes occupied by each index is less than the number of bytes occupied by each data.

[0168] The file compression module 1006 is configured to group the indices of each centroid and each data to form a compressed file corresponding to the to-be-compressed file.

[0169] In an exemplary embodiment, the index determination module 1004 is further configured to store the centroids of the clustering clusters into a first data set, determine the index of each centroid according to the storage order of each centroid in the first data set, and equalize the number of bytes occupied by each centroid and the number of bytes occupied by each data.

[0170] The file compression module 1006 is further configured to store the indexes of the data in the second data set in sequence according to the storage sequence of the data in the file to be compressed; and combine the first data set and the second data set to obtain a compressed file corresponding to the file to be compressed.

[0171] In an example embodiment, the file compression module 1006 is further configured to determine a total number of preset centroids for the clustering processing; and combine the total number of centroids, the first data set and the second data set in sequence to obtain the compressed file.

[0172] In an example embodiment, the number of bytes occupied by each index is at least the number of bytes required to represent the total number of preset centroids for the clustering processing in binary.

[0173] In an example embodiment, the data clustering module 1002 is further configured to initialize a plurality of centroids according to the data lower limit value and the data upper limit value of each data and a data distribution interval corresponding to the file type to which the file to be compressed belongs; the data distribution interval is used to represent the distribution of the data set of the file type; and the clustering processing is performed on the data based on the initialized plurality of centroids to obtain a plurality of clustering clusters.

[0174] In an example embodiment, the data clustering module 1002 is further configured to initialize a first number of centroids according to the data lower limit value and the data upper limit value and an interval lower limit value and an interval upper limit value of the data distribution interval; initialize a second number of centroids evenly within the data distribution interval; the second number is obtained by subtracting the first number from the total number of preset centroids; and the first number of centroids and the second number of centroids are determined as the initialized plurality of centroids.

[0175] In an example embodiment, the data clustering module 1002 is further configured to initialize the data lower limit value and the data upper limit value as one centroid; in a case where the data lower limit value is less than or equal to the interval lower limit value, initialize an average of the data lower limit value and the interval lower limit value as one centroid; in a case where the data lower limit value is greater than the interval lower limit value, initialize an average of the interval lower limit value and a midpoint value of the data distribution interval as one centroid; in a case where the data upper limit value is greater than or equal to the interval upper limit value, initialize an average of the data upper limit value and the interval upper limit value as one centroid; and in a case where the data upper limit value is less than the interval upper limit value, initialize an average of the interval upper limit value and the midpoint value as one centroid.

[0176] In an example embodiment, the file compression apparatus further includes a data distribution statistics module configured to obtain a plurality of files of each file type; and obtain the data distribution interval representing the distribution of the data set of the file type by statistically analyzing the data distribution in each file.

[0177] Each module in the file compression device can be implemented by software, hardware, and a combination thereof, in whole or in part. Each module can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in a computer device in software form, so as to be called and executed by a processor to perform operations corresponding to each module.

[0178] In an exemplary embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in Figure 11 The computer device includes a processor, a memory, an input / output interface, and a communication interface. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store a file to be compressed. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with a terminal outside through a network connection. The computer program is executed by the processor to implement a file compression method.

[0179] Those skilled in the art can understand that Figure 11 The structure shown in the above

[0180] In an exemplary embodiment, a computer device is also provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in each method embodiment.

[0181] In an exemplary embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps in each method embodiment.

[0182] In an exemplary embodiment, a computer program product is provided, which includes a computer program. The computer program is executed by a processor to implement the steps in each method embodiment.

[0183] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0184] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0185] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A file compression method characterized by, The method comprises: obtaining a to-be-compressed file; initializing a first number of centroids according to data lower limit values and data upper limit values of each data in the to-be-compressed file and interval lower limit values and interval upper limit values of a data distribution interval corresponding to a file type to which the to-be-compressed file belongs; the data distribution interval is used to represent distribution in a data set of the file type; initializing a second number of centroids evenly in the data distribution interval; the second number is obtained by subtracting the first number from a preset total number of centroids; determining the first number of centroids and the second number of centroids as a plurality of initialized centroids; performing clustering processing on the each data based on the plurality of initialized centroids to obtain a plurality of clustering clusters; establishing an index of a centroid of each clustering cluster, and determining an index of a centroid of a clustering cluster to which each data belongs as an index of the each data; the index of each centroid is used to represent an order of the centroid in the plurality of centroids; a number of bytes occupied by each index is less than a number of bytes occupied by each data; grouping the indexes of the each data and the each centroid to form a compressed file corresponding to the to-be-compressed file.

2. The method of claim 1, wherein, The establishing of the index of the centroid of each clustering cluster comprises: storing the centroid of each clustering cluster into a first data set; determining the index of each centroid according to a storage order of the each centroid in the first data set; a number of bytes occupied by each centroid is equal to a number of bytes occupied by each data; The grouping of the indexes of the each data and the each centroid to form the compressed file corresponding to the to-be-compressed file comprises: storing the indexes of the each data into a second data set in a storage order of the each data in the to-be-compressed file; combining the first data set and the second data set to obtain the compressed file corresponding to the to-be-compressed file.

3. The method of claim 2, wherein, The combining of the first data set and the second data set to obtain the compressed file corresponding to the to-be-compressed file further comprises: determining a preset total number of centroids for the clustering processing; combining the total number of centroids, the first data set and the second data set in sequence to obtain the compressed file.

4. The method of claim 1, wherein, A number of bytes occupied by each index is at least a number of bytes required for representing the total number of centroids for the clustering processing in binary numeration.

5. The method of claim 1, wherein, The initializing of the first number of centroids according to the data lower limit values and the data upper limit values of the each data in the to-be-compressed file and the interval lower limit values and the interval upper limit values of the data distribution interval corresponding to the file type to which the to-be-compressed file belongs comprises: initializing the data lower limit values and the data upper limit values as one centroid respectively; in a case where the data lower limit value is less than or equal to the interval lower limit value, initializing an average of the data lower limit value and the interval lower limit value as one centroid, and in a case where the data lower limit value is greater than the interval lower limit value, initializing an average of the interval lower limit value and a midpoint value of the data distribution interval as one centroid; In the case that the data upper limit value is greater than or equal to the interval upper limit value, an average of the data upper limit value and the interval upper limit value is initialized as a centroid; in the case that the data upper limit value is less than the interval upper limit value, an average of the interval upper limit value and the midpoint value is initialized as a centroid.

6. The method of claim 1, wherein, The method further comprises: For each file type, a plurality of files under the file type are obtained; Data distribution of each file is counted to obtain a data distribution interval for representing data distribution in the data set of the file type.

7. The method according to any one of claims 1 to 6, characterized in that, The order of the plurality of centroids is obtained according to centroid numerical values.

8. The method according to any one of claims 1 to 6, characterized in that, In the compressed file, a number of bytes occupied by each centroid is equal to a number of bytes occupied by each data. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method of any one of claims 1 to 8.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 8.

Images