File compression method, computer equipment and computer program product

Abstract

The invention relates to a file compression method, computer equipment and a computer program product, and relates to the field of data processing. The method comprises the following steps: acquiring a to-be-compressed file, and clustering data in the to-be-compressed file to obtain a plurality of clusters; establishing an index of the centroid of each cluster, and determining the index of the centroid of the cluster to which each datum belongs as the index of each datum; the index of each centroid is used for representing the sequence of the centroid in the plurality of centroids; the number of bytes occupied by each index is smaller than the number of bytes occupied by each piece of data; and forming a compressed file corresponding to the to-be-compressed file by using the centroids and the indexes of the data. By adopting the method, the file downloading speed can be increased.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a file compression method, device, computer equipment, computer-readable storage medium and computer program product. Background Art

[0002] Files often need to be downloaded before they can be used, for example, audio files need to be downloaded before they can be played. However, the download speed of files is affected by the size of the files. If the size of the file is large, the download speed of the file will be slow. Summary of the invention

[0003] Based on this, it is necessary to provide a file compression method, computer device and computer program product that can improve the download speed of files in order to address the technical problem of the slow download speed of the above files.

[0004] In a first aspect, the present application provides a file compression method, comprising:

[0005] Acquire a file to be compressed, and perform clustering processing on each data in the file to be compressed to obtain multiple clusters;

[0006] Establishing the index of the centroid of each cluster, determining the index of the centroid of the cluster to which each data belongs as the index of each data; the index of each centroid is used to characterize the order of the centroid among multiple centroids; the number of bytes occupied by each index is less than the number of bytes occupied by each data;

[0007] The centroids and the indexes of the data are combined into a compressed file corresponding to the file to be compressed.

[0008] In one embodiment, the step of establishing the index of the centroid of each cluster includes:

[0009] The centroids of the clusters are stored in the first data set; the number of bytes occupied by each centroid is equal to the number of bytes occupied by each data;

[0010] Determine the index of each centroid according to the storage order of each centroid in the first data set;

[0011] The step of forming a compressed file corresponding to the file to be compressed by combining the centroids and the indexes of the data includes:

[0012] According to the storage order of the data in the to-be-compressed file, the indexes of the data are sequentially stored in the second data set;

[0013] The first data set and the second data set are combined to obtain a compressed file corresponding to the file to be compressed.

[0014] In one embodiment, combining the first data set and the second data set to obtain a compressed file corresponding to the file to be compressed further includes:

[0015] Determining a total number of centroids preset for the clustering process;

[0016] The total number of centroids, the first data set, and the second data set are combined in order to obtain the compressed file.

[0017] In one of the embodiments, the number of bytes occupied by each index is the minimum number of bytes required to represent the total number of centroids preset for the clustering process in a binary number system.

[0018] In one embodiment, clustering the data in the to-be-compressed file to obtain a plurality of clusters includes:

[0019] Initialize multiple centroids according to the data lower limit value and the data upper limit value of each data and the data distribution interval corresponding to the file type to which the to-be-compressed file belongs; the data distribution interval is used to characterize the data center distribution of the file type to which the to-be-compressed file belongs;

[0020] Based on the multiple centroids obtained by initialization, clustering processing is performed on the data to obtain the multiple cluster clusters.

[0021] In one of the embodiments, the number of the plurality of initialized centroids is a preset number;

[0022] Initializing multiple centroids according to the data lower limit value and the data upper limit value of each data and the data distribution interval corresponding to the file type to which the to-be-compressed file belongs includes:

[0023] Initialize a first number of centroids according to the data lower limit value and the data upper limit value, and the interval lower limit value and the interval upper limit value of the data distribution interval;

[0024] Uniformly initializing a second number of centroids within the data distribution interval; the second number is obtained by subtracting a preset total number of centroids from the first number;

[0025] The first number of centroids and the second number of centroids are determined as the initialized plurality of centroids.

[0026] In one embodiment, initializing a first number of centroids according to the data lower limit value and the data upper limit value, and the interval lower limit value and the interval upper limit value of the data distribution interval includes:

[0027] Initializing the data lower limit value and the data upper limit value as a centroid respectively;

[0028] When the data lower limit value is less than or equal to the interval lower limit value, the average value of the data lower limit value and the interval lower limit value is initialized as a centroid; when the data lower limit value is greater than the interval lower limit value, the average value of the interval lower limit value and the midpoint value of the data distribution interval is initialized as a centroid;

[0029] When the data upper limit value is greater than or equal to the interval upper limit value, the average of the data upper limit value and the interval upper limit value is initialized as a centroid. When the data upper limit value is less than the interval upper limit value, the average of the interval upper limit value and the midpoint value is initialized as a centroid.

[0030] In one embodiment, the method further comprises:

[0031] For each file type, obtain multiple files of the file type;

[0032] The data distribution in each file is counted to obtain a data distribution interval for characterizing the distribution of the file type in the data set.

[0033] In a second aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0034] Acquire a file to be compressed, and perform clustering processing on each data in the file to be compressed to obtain multiple clusters;

[0035] Establishing the index of the centroid of each cluster, determining the index of the centroid of the cluster to which each data belongs as the index of each data; the index of each centroid is used to characterize the order of the centroid among multiple centroids; the number of bytes occupied by each index is less than the number of bytes occupied by each data;

[0036] The centroids and the indexes of the data are combined into a compressed file corresponding to the file to be compressed.

[0037] In a third aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0038] Acquire a file to be compressed, and perform clustering processing on each data in the file to be compressed to obtain multiple clusters;

[0039] Establishing the index of the centroid of each cluster, determining the index of the centroid of the cluster to which each data belongs as the index of each data; the index of each centroid is used to characterize the order of the centroid among multiple centroids; the number of bytes occupied by each index is less than the number of bytes occupied by each data;

[0040] The centroids and the indexes of the data are combined into a compressed file corresponding to the file to be compressed.

[0041] The above-mentioned file compression method, computer device and computer program product first obtain the file to be compressed, cluster the data in the file to be compressed, and obtain multiple clusters; then, establish the index of the centroid of each cluster, and determine the index of the centroid of the cluster to which each data belongs as the index of each data; the index of each centroid is used to characterize the order of the centroid among multiple centroids; the number of bytes occupied by each index is less than the number of bytes occupied by each data; then, the indexes of each centroid and each data are combined into a compressed file corresponding to the file to be compressed. The file compression method based on the above process can obtain the index of each data by clustering the data and establishing the index of the centroid of the cluster cluster. Since the index of each data points to the centroid of the cluster cluster to which the data belongs, before and after compression, the information recorded in the file is converted from the data itself to the index pointing to the centroid of the cluster cluster to which the data belongs and the centroid pointed to by the index; since the amount of data is often large in actual business, the volume occupied by each centroid in the compressed file can be ignored, that is, the volume of the compressed file mainly considers the volume occupied by the index of each data, and the number of bytes occupied by each index is less than the number of bytes occupied by each data. Therefore, the volume of the compressed file is reduced compared to the file to be compressed, thereby increasing the download speed of the file. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A schematic diagram of a flow chart of a file compression method in an embodiment;

[0044] Figure 2 A schematic flow chart of the steps of establishing an index of the centroid of each cluster in one embodiment;

[0045] Figure 3A flowchart of a step of forming a compressed file corresponding to a file to be compressed by combining the indexes of each centroid and each data in an embodiment;

[0046] Figure 4 A schematic flow chart of the steps of combining a first data set and a second data set to obtain a compressed file corresponding to a file to be compressed in one embodiment;

[0047] Figure 5 is a schematic diagram showing a comparison between a file to be compressed and a compressed file in one embodiment;

[0048] Figure 6 A schematic flow chart of a step of clustering the data in a compressed file to obtain a plurality of clusters in one embodiment;

[0049] Figure 7 It is a flowchart of the steps of initializing multiple centroids according to the data lower limit value and the data upper limit value of each data and the data distribution interval corresponding to the file type to which the to-be-compressed file belongs in one embodiment;

[0050] Figure 8 A flowchart of a step of initializing a first number of centroids according to a data lower limit value and a data upper limit value, and an interval lower limit value and an interval upper limit value of a data distribution interval in an embodiment;

[0051] Fig. 9 is a flowchart of a file compression method in another embodiment;

[0052] Fig.10 is a structural block diagram of a file compression device in one embodiment;

[0053] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0056] In one embodiment, Figure 1As shown, a file compression method is provided. This embodiment uses the method applied to a server as an example for illustration. It is understandable that the method can also be applied to a terminal, and can also be applied to a system including a server and a terminal, and is implemented through the interaction between the server and the terminal; wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services; the terminal can be, but is not limited to, various personal computers, laptops, smart phones, and tablet computers, etc. In this embodiment, the method includes the following steps:

[0057] Step S102: obtaining a file to be compressed, and performing clustering processing on each data in the file to be compressed to obtain a plurality of clusters.

[0058] The file type to which the to-be-compressed file belongs is a file type whose data distribution of the corresponding files is similar and the data distribution is relatively concentrated. In practical applications, the to-be-compressed file is a song's voice posterior probability file, which can be used in applications such as singing generation and tuning in singing synthesis scenarios.

[0059] It is easy to understand that binary is a number system widely used by computers for data storage and processing. Therefore, the files to be compressed and the compressed files in this application are both binary files, and the data, centroids, indexes, etc. in the files to be compressed in this application are also binary data. However, for ease of understanding, decimal data is used as an example in one or more of the following embodiments of this application.

[0060] Specifically, the server obtains the file to be compressed, and clusters each data in the compressed file into multiple clusters; each cluster has at least one data, and each cluster has a corresponding centroid for representing the average level of each data in the cluster.

[0061] For example, suppose the data in the file to be compressed is {5, 23, 6, 15, 33, 17, 4, 28}. The server clusters the data to obtain cluster 1 {5, 6, 4}, cluster 2 {15, 17} and cluster 3 {23, 33, 28}, where the centroid of cluster 1 is 5, the centroid of cluster 2 is 16, and the centroid of cluster 3 is 28.

[0062] In a specific application, the data in the file to be compressed may exist in the form of a multidimensional array. Therefore, the server may regard each multidimensional array element in the multidimensional array as one data, or may flatten the multidimensional array in the file to be compressed (flatten operation), convert the multidimensional array into a corresponding one-dimensional array, and then regard each array element in the one-dimensional array as one data.

[0063] Step S104: establishing an index of the centroid of each cluster, and determining the index of the centroid of the cluster to which each data belongs as the index of each data.

[0064] The index of each centroid is used to represent the order of the centroid among multiple centroids.

[0065] The number of bytes occupied by each index is smaller than the number of bytes occupied by each data.

[0066] The number of bytes occupied by each data is related to the precision of the data. For example, if the data is fp32 (single-precision floating point number), the number of bytes occupied by each data is 4Bytes (bytes), that is, 32bits (bits). If the data is fp16 (half-precision floating point number), the number of bytes occupied by each data is 2Bytes, that is, 16bits.

[0067] Among them, the number of bytes occupied by a single index is the minimum number of bytes required to represent the total number of different integers of centroids preset for clustering processing in the binary number system; for example, assuming that the total number of centroids is 256, then the number of centroid indexes is also 256. Since computers mostly use the binary number system to store data, and in the binary number system, at least 1 Byte is required to represent 256 different integers. Therefore, the number of bytes occupied by each index is 1 Byte.

[0068] In specific applications, since the smallest unit of computer storage data is bit, and computer storage data mostly uses binary number system, the total number of centroids is preset to a positive integer power of 2 to make full use of each bit and facilitate reading by other servers or terminals.

[0069] Specifically, first for each centroid, the server determines the order of the centroid among multiple centroids and determines the order as the index of the centroid. Then, for each data, the server determines the index of the centroid of the cluster to which the data belongs as the index of the data.

[0070] It is easy to understand that determining the index of the centroid of the cluster to which each data belongs as the index of each data means that the index of each data is the index of the corresponding centroid (the centroid of the cluster to which the data belongs), that is, for each data, its index can point to its corresponding centroid. Therefore, the server can determine the centroid corresponding to the data based on the index of the data.

[0071] In practical applications, the order of the multiple centroids may be obtained in any manner, for example, the order of the multiple centroids may be from large to small or from small to large according to the centroid values.

[0072] For example, suppose the centroid of cluster 1 {5, 1, 4} is 5, the centroid of cluster 2 {15, 17} is 16, the centroid of cluster 3 {23, 33, 28} is 28, and the order of the three centroids is {5, 16, 28}. Then, the server determines the index of the centroid 5 of cluster 1 as "0", the index of the centroid 16 of cluster 2 as "1", and the index of the centroid 28 of cluster 3 as "2"; then, the server determines the indexes of data 5, data 1, and data 4 in cluster 1 as "0", the indexes of data 15 and data 17 in cluster 2 as 1, and the indexes of data 22, data 33, and data 28 in cluster 3 as "2".

[0073] Step S106, combining the indexes of each centroid and each data into a compressed file corresponding to the file to be compressed.

[0074] Specifically, the server first stores the centroids of each cluster in a blank file in the order of multiple centroids, and then stores the index of each data behind the centroid of each cluster in the order of each data in the file to be compressed, thereby obtaining a compressed file corresponding to the file to be compressed.

[0075] For example, the server first stores the centroids {5, 16, 28} of cluster 1, cluster 2, and cluster 3 in a blank file, and then stores the indexes of each data {0, 2, 0, 1, 2, 1, 0, 2} thereafter, thereby obtaining the compressed file "{5, 16, 28}, {0, 2, 0, 1, 2, 1, 0, 2}".

[0076] In this way, when the computer parses the compressed file, it can read the first data as 5, the second data as 28, the third data as 5 according to the index and centroid of the data... It is easy to understand that the amount of data in the file to be compressed is often large enough, and when the distribution of each data is relatively concentrated and the number of clusters is large enough, the difference between the centroid of each cluster and the data in the cluster will be small enough, so the loss of accuracy of the file to be compressed compared to the compressed file can be ignored.

[0077] In practical applications, since the expression of each data in the compressed file is converted from the data itself to the centroid corresponding to the data, in order to minimize the accuracy loss before and after file compression, preferably, in the compressed file, the number of bytes occupied by each centroid is equal to the number of bytes occupied by each data.

[0078] For example, suppose there is data, each data occupies the number of bytes Byte, then the size of the file to be compressed is Byte.

[0079] Assume that the total number of centroids is preset to And the number of bytes occupied by each centroid is equal to the number of bytes occupied by each data, which is also Byte, then the volume occupied by each centroid in the compressed file is Byte. Assuming the binary number system, it is represented by The minimum number of bytes required for different integers is Byte, that is, the number of bytes occupied by a single index is Byte, then, in the compressed file The volume occupied by the index of data is Byte. In summary, the size of the compressed file is Byte. In actual business, is often large, so the volume occupied by each centroid Byte can be ignored, that is, the size of the compressed file is considered Byte, and Less than Therefore, the size of the compressed file is reduced compared to the file to be compressed.

[0080] In the above file compression method, first, the server obtains the file to be compressed, clusters each data in the file to be compressed, and obtains multiple clusters; then, the server establishes the index of the centroid of each cluster, and determines the index of the centroid of the cluster to which each data belongs as the index of each data; the index of each centroid is used to characterize the order of the centroid among multiple centroids; the number of bytes occupied by each index is less than the number of bytes occupied by each data; then, the server combines the indexes of each centroid and each data into a compressed file corresponding to the file to be compressed. Based on the file compression method of the above process, the server can obtain the index of each data through clustering processing of the data and establishing the index of the centroid of the cluster cluster. Since the index of each data points to the centroid of the cluster cluster to which the data belongs, before and after compression, the server converts the information recorded in the file from the data itself into an index pointing to the centroid of the cluster cluster to which the data belongs and the centroid pointed to by the index; since the amount of data is often large in actual business, the volume occupied by the centroid of each cluster cluster in the compressed file can be ignored, that is, the volume of the compressed file mainly considers the volume occupied by the index of each data, and the number of bytes occupied by each index is less than the number of bytes occupied by each data. Therefore, the volume of the compressed file is reduced compared to the file to be compressed, thereby increasing the download speed of the file.

[0081] In an exemplary embodiment, Figure 2As shown, the establishment of the index of the centroid of each cluster in the above step S104 specifically includes the following steps:

[0082] Step S202: storing the centroid of each cluster in the first data set.

[0083] Step S204: determine the index of each centroid according to the storage order of each centroid in the first data set.

[0084] It is easy to understand that, since the expression of each data in the compressed file is converted from the data itself to the centroid corresponding to the data, in order to minimize the precision loss before and after the file is compressed, the precision of the centroid is preferably consistent with the precision of the data. Furthermore, in this embodiment, the number of bytes occupied by each centroid is equal to the number of bytes occupied by each data.

[0085] Specifically, the server stores the centroids of each cluster in a blank first data set to obtain a first data set {5, 16, 28}, and determines the storage order of each centroid in the first data set as the index of the centroid, for example, the index of centroid 5 is 0, the index of centroid 16 is 1, and the index of centroid 28 is 2.

[0086] In practical applications, the first data set is a LUT (look up table), that is, the server stores the centroid of each cluster in the LUT.

[0087] like Figure 3 As shown, the above step S106, which forms a compressed file corresponding to the file to be compressed by combining the indexes of each centroid and each data, specifically includes the following steps:

[0088] Step S302: store the index of each data in the second data set in sequence according to the storage order of each data in the file to be compressed.

[0089] Step S304: combining the first data set and the second data set to obtain a compressed file corresponding to the file to be compressed.

[0090] Specifically, the server stores the index of each data in the blank second data set in sequence according to the storage order of each data in the to-be-compressed file {5, 23, 6, 15, 33, 17, 4, 28}, and obtains the second data set {0, 2, 0, 1, 2, 1, 0, 2}. Then, the server combines the first data set {5, 16, 28} and the second data set {0, 2, 0, 1, 2, 1, 0, 2} to obtain the compressed file "{5, 16, 28}, {0, 2, 0, 1, 2, 1, 0, 2}".

[0091] In this embodiment, since the index of each data points to the centroid of the cluster to which the data belongs, before and after compression, the server converts the information recorded in the file from the data itself into an index pointing to the centroid of the cluster to which the data belongs and the centroid pointed to by the index, that is, the storage of the data itself is converted into the storage of the centroid and the index of the centroid. Since the amount of data is often large, the volume occupied by each centroid in the compressed file can be ignored, that is, the volume of the compressed file mainly considers the volume occupied by the index of each data, and the number of bytes occupied by each index is less than the number of bytes occupied by each data. Therefore, the volume of the compressed file is reduced compared to the file to be compressed, thereby increasing the download speed of the file.

[0092] In an exemplary embodiment, Figure 4 As shown, the above step S304, combining the first data set and the second data set to obtain a compressed file corresponding to the file to be compressed, specifically also includes the following steps:

[0093] Step S402, determining the total number of centroids preset for clustering processing.

[0094] Step S404, combining the total number of centroids, the first data set, and the second data set in order to obtain a compressed file.

[0095] Specifically, the server determines the total number of centroids preset for clustering processing, and then combines the total number of centroids, the first data set and the second data set in order to obtain a compressed file.

[0096] It is easy to understand that the number of bytes required to represent the total number of centroids is equal to the number of bytes required to represent the total number of centroids integer. For example, when the number of indexes is 256, the number of bytes required to represent 256 is also 1 Byte, so the number of bytes occupied by the total number of centroids 256 is also 1 Byte.

[0097] For example, suppose there is data, each data occupies 4 bytes, then the size of the file to be compressed is Byte.

[0098] Assuming that the total number of centroids is 256, the number of bytes occupied by each index is 1 Byte, the number of bytes occupied by the total number of centroids is also 1 Byte, and assuming that the number of bytes occupied by each centroid is equal to the number of bytes occupied by each data, it is also Byte. Then, in the compressed file, the total volume occupied by the number of centroids is 1 Byte, and the volume occupied by each centroid is Byte, The volume occupied by the index of data is Byte, and the size of the compressed file is Byte. In actual business, are often large, so the size of the compressed file is considered Byte, compressed to the file to be compressed , has been significantly reduced.

[0099] See also Figure 5 , which is a schematic diagram for comparing the file to be compressed and the compressed file.

[0100] In this embodiment, the compressed file is obtained by sequentially combining the total number of centroids, the first data set, and the second data set. This allows other servers or terminals to read the total number of centroids at the front of the compressed file when parsing the compressed file, which facilitates the subsequent reading of the first data set and the second data set (dividing the volume occupied by the first data set by the total number of centroids can clearly determine the number of bytes occupied by a single centroid in the compressed file, thereby facilitating the reading of the centroid; the total number of centroids can determine the number of bytes occupied by a single index, thereby facilitating the reading of the index).

[0101] In an exemplary embodiment, the number of bytes occupied by each index in one or more embodiments of the present application is the minimum number of bytes required to represent the total number of centroids preset for clustering processing in a binary number system.

[0102] It is easy to understand that the index is the index of the centroid. Therefore, the same number of integers are needed to represent the index of the centroid for the number of centroids preset. Therefore, the number of bytes occupied by each index is the minimum number of bytes required to represent the total number of centroids in the binary number system. For example, assuming that the total number of centroids is 256, then 256 integers are required to represent the index of the centroid. In the binary number system, if at least 1 Byte is required to represent 256 different integers, then the number of bytes occupied by each index is 1 Byte.

[0103] In this embodiment, the number of bytes occupied by each index is the minimum number of bytes required to represent the total number of centroids integers in the binary number system, which can minimize the number of bytes occupied by a single index and thus minimize the size of the compressed file.

[0104] In an exemplary embodiment, Figure 6 As shown, in the above step S102, clustering is performed on each data in the compressed file to obtain multiple clusters, which specifically includes the following steps:

[0105] Step S602 , initializing a plurality of centroids according to the data lower limit value and the data upper limit value of each data and the data distribution interval corresponding to the file type to which the to-be-compressed file belongs.

[0106] Step S604: clustering the data based on the multiple centroids obtained by initialization to obtain multiple clusters.

[0107] Among them, the data distribution interval is used to characterize the distribution of the data set of the file type to which it belongs; in a specific application, the data distribution interval is the interval of the distribution of the data set of the file type to which it belongs; for example, taking the speech posterior probability file of a song as an example, the data distribution in the speech posterior probability files of different songs is basically the same, close to the normal distribution, and the data is basically concentrated in the interval of [-3, 3]. Therefore, for the file type of the speech posterior probability file of a song, the corresponding data distribution interval is [-3, 3].

[0108] Specifically, before clustering, the server first determines the data lower limit value and data upper limit value of each data in the file to be compressed, and obtains the data distribution range corresponding to the file type to which the file to be compressed belongs; then, the server initializes multiple centroids based on the data lower limit value, data upper limit value and data distribution range; and based on the multiple centroids obtained by initialization, a preset clustering algorithm is used to cluster each data to obtain multiple cluster clusters.

[0109] In practical applications, the preset clustering algorithm can be any clustering algorithm that can cluster discrete data. In specific applications, preferably, the present application uses the k-means clustering algorithm to cluster each data, and the specific process is as follows: after initializing the centroid, (1) for each data, the server calculates the Euclidean distance between the data and each centroid, and assigns the data to the cluster corresponding to the nearest centroid; (2) after each data is assigned, for each cluster, the server calculates the average value of the data in the cluster, and uses the average value as the new centroid of the cluster; repeat the above steps (1) and (2) until each cluster converges or the cluster reaches the maximum number of iterations, and then terminate the clustering, and use the final clusters as multiple clusters corresponding to the clustering results.

[0110] In this embodiment, the server initializes the centroid by the data lower limit, the data upper limit and the data distribution interval, rather than randomly initializing the centroid, which can speed up the calculation of the centroid and thus speed up the clustering process. The server can divide each data into multiple clusters by clustering each data, and can then convert the data expression from the data itself into the centroid of the cluster to which the data belongs.

[0111] In an exemplary embodiment, Figure 7 As shown, according to the data lower limit value and the data upper limit value of each data, and the data distribution interval corresponding to the file type to which the to-be-compressed file belongs, multiple centroids are initialized, specifically including the following steps:

[0112] Step S702, initialize a first number of centroids according to the data lower limit value and the data upper limit value, and the interval lower limit value and the interval upper limit value of the data distribution interval.

[0113] Step S704, uniformly initialize a second number of centroids within the data distribution interval.

[0114] Step S706: Determine the first number of centroids and the second number of centroids as the initialized multiple centroids.

[0115] The second number is obtained by subtracting a predetermined total number of centroids from the first number.

[0116] Specifically, first, the server determines the total number of centroids; then, the server determines the interval lower limit value and the interval upper limit value of the data distribution interval; then, the server initializes a first number of centroids based on the data lower limit value, the data upper limit value, the interval lower limit value and the interval upper limit value; finally, the server uniformly initializes the remaining second number of centroids within the data distribution interval; in this way, the first number of centroids and the second number of centroids are the multiple centroids obtained by initialization.

[0117] In practical applications, the first number is 4.

[0118] For example, assuming that the total number of centroids is 256, the server first initializes 4 centroids based on the data lower limit, data upper limit, interval lower limit, and interval upper limit, and then evenly initializes the remaining 251 centroids within the data distribution interval.

[0119] In this embodiment, the server initializes multiple centroids through the actual distribution of each data of the file to be compressed (data lower limit value and data upper limit value) and the data concentration distribution of the file type to which the file to be compressed belongs (interval lower limit value and interval upper limit value), which can accelerate the centroid calculation in the clustering process and further accelerate the clustering processing.

[0120] In an exemplary embodiment, Figure 8 As shown, the above step S702, according to the data lower limit value and the data upper limit value, and the interval lower limit value and the interval upper limit value of the data distribution interval, initializes the first number of centroids, specifically including the following steps:

[0121] Step S802, initializing the data lower limit value and the data upper limit value as a centroid respectively.

[0122] Step S804, when the data lower limit value is less than or equal to the interval lower limit value, initialize the average of the data lower limit value and the interval lower limit value as a centroid; when the data lower limit value is greater than the interval lower limit value, initialize the average of the interval lower limit value and the midpoint value of the data distribution interval as a centroid.

[0123] Step S806, when the data upper limit value is greater than or equal to the interval upper limit value, the average of the data upper limit value and the interval upper limit value is initialized as a centroid; when the data upper limit value is less than the interval upper limit value, the average of the interval upper limit value and the midpoint value is initialized as a centroid.

[0124] Specifically, the server first initializes the data lower limit value and the data upper limit value as a centroid respectively.

[0125] Then, the server compares the data lower limit value and the interval lower limit value. If the data lower limit value is less than or equal to the interval lower limit value, the server calculates the average value of the data lower limit value and the interval lower limit value, and initializes the average value as a centroid; if the data lower limit value is greater than the interval lower limit value, the server calculates the average value of the data lower limit value and the midpoint value of the data distribution interval, and initializes the average value as a centroid.

[0126] For example, assuming that the data distribution interval is [-3, 3], then the lower limit of the interval is -3, and the midpoint value of the data distribution interval is 0; if the data lower limit is -4, then the server initializes -3.5 as a centroid, and if the data lower limit is -1, then the server initializes -1.5 as a centroid.

[0127] Next, the server compares the data upper limit value and the interval upper limit value. If the data upper limit value is greater than or equal to the interval upper limit value, the server calculates the average of the data upper limit value and the interval upper limit value, and initializes the average value as a centroid; if the data upper limit value is less than the interval upper limit value, the server calculates the average of the data upper limit value and the midpoint value of the data distribution interval, and initializes the average value as a centroid.

[0128] For example, assuming that the data distribution interval is [-3, 3], then the upper limit value of the interval is 3, and the midpoint value of the data distribution interval is 0; if the upper limit value of the data is 4, then the server initializes 3.5 as a centroid, and if the lower limit value of the data is 2, then the server initializes 1.5 as a centroid.

[0129] In this embodiment, the server initializes four centroids based on the data lower limit value and the data upper limit value, and the numerical comparison of the data lower limit value and the data upper limit value with the interval lower limit value and the interval upper limit value, which can accelerate the centroid calculation in the clustering process and thus accelerate the clustering processing.

[0130] In an exemplary embodiment, the file compression method provided by the present application also includes the following contents: for each file type, multiple files under the file type are obtained; and data distribution in each file is counted to obtain a data distribution interval for characterizing the distribution of the file type in a data set.

[0131] Specifically, the file compression method provided by the present application can compress files of multiple file types; for each file type, the server obtains multiple files under the file type, and counts the data distribution in each file to obtain an interval representing the distribution of the data set of the file type as the data distribution interval corresponding to the file type.

[0132] In practical applications, the file types that can be applied to the file compression method provided in this application are file types whose data distributions of corresponding files are similar and whose data distributions are relatively concentrated, for example, speech posterior probability files of songs.

[0133] In this embodiment, the server can obtain data distribution intervals representing the distribution of the data set of the same file type by statistics on the data distribution of the files of the same file type. Based on the data distribution intervals, the centroid calculation in the clustering process can be accelerated.

[0134] In an exemplary embodiment, if Fig. 9 As shown, another file compression method is provided, which is described by taking the application of the method to a server as an example, and includes the following steps:

[0135] Step S902, obtaining the file to be compressed.

[0136] Step S904: Initialize the data lower limit value and the data upper limit value in each data of the to-be-compressed file as a centroid respectively.

[0137] Step S906, when the data lower limit value is less than or equal to the interval lower limit value of the data distribution interval, the average value of the data lower limit value and the interval lower limit value is initialized as a centroid; when the data lower limit value is greater than the interval lower limit value, the average value of the interval lower limit value and the midpoint value of the data distribution interval is initialized as a centroid.

[0138] Step S908, when the data upper limit value is greater than or equal to the interval upper limit value of the data distribution interval, the average of the data upper limit value and the interval upper limit value is initialized as a centroid; when the data upper limit value is less than the interval upper limit value, the average of the interval upper limit value and the midpoint value is initialized as a centroid.

[0139] Step S910, uniformly initialize the remaining number of centroids in the predetermined total number of centroids within the data distribution interval.

[0140] Step S912: Determine the first number of centroids and the second number of centroids as multiple initialized centroids, and perform clustering processing on each data based on the multiple initialized centroids to obtain multiple cluster clusters.

[0141] Step S914: store the centroids of each cluster in the first data set, and determine the index of each centroid according to the storage order of each centroid in the first data set.

[0142] Step S916: determine the index of the centroid of the cluster to which each data belongs as the index of each data.

[0143] Step S918: store the index of each data in the second data set in sequence according to the storage order of each data in the file to be compressed.

[0144] Step S920, determining the total number of centroids.

[0145] Step S922, combining the total number of centroids, the first data set, and the second data set in order to obtain a compressed file.

[0146] In this embodiment, first, the server initializes the centroid by the data lower limit value, data upper limit value and data distribution interval, rather than randomly initializing the centroid, which can accelerate the calculation of the centroid and thus accelerate the clustering process. Second, since the index of each data points to the centroid of the cluster to which the data belongs, before and after compression, the server converts the information recorded in the file from the data itself to the index pointing to the centroid of the cluster to which the data belongs and the centroid pointed to by the index, that is, the storage of the data itself is converted into the storage of the centroid and the index of the centroid. Since the amount of data is often large, the volume occupied by each centroid in the compressed file can be ignored, that is, the volume of the compressed file mainly considers the volume occupied by the index of each data, and the number of bytes occupied by each index is less than the number of bytes occupied by each data, so the volume of the compressed file is reduced compared to the file to be compressed. Third, by sequentially combining the total number of centroids, the first data set and the second data set to obtain the compressed file, other servers or other terminals can read the total number of centroids at the front of the compressed file when parsing the compressed file, which facilitates the subsequent reading of the first data set and the second data set.

[0147] In order to more clearly illustrate the file compression method provided by the embodiment of the present application, the file compression method is specifically described below with a specific embodiment, but it should be understood that the embodiment of the present application is not limited thereto. In one exemplary embodiment, the present application also provides a non-uniform discretization compression method based on k-means clustering, which specifically includes the following contents:

[0148] In the end-side reasoning process of the singing synthesis scenario, the generation of each song depends on a specific template file. Among them, the largest resource file is the song's speech posterior probability file. The precision of each data in the speech posterior probability file is fp32, that is, the number of bytes occupied by each data is 4 bytes. If a certain precision loss is acceptable, the data precision can be compressed to fp16, so that the size of the speech posterior probability file can be reduced by half.

[0149] In order to further compress the volume of the speech posterior probability file on the basis of fp16, this embodiment proposes a non-uniform discretization compression method based on k-means clustering. The compression principle is to further discretize the originally discretized data in combination with the characteristics of the concentrated distribution of the data of the speech posterior probability file, and replace the data with the corresponding centroid, so as to ensure the distribution trend and accuracy of the original data as much as possible with the non-uniformly distributed centroid.

[0150] In this embodiment, the server first presets the total number of centroids in the clustering process. Since the smallest unit of computer storage data is bit, and computer storage data mostly uses binary number system, this application determines the total number of centroids as a positive integer power of 2 to make full use of each bit and facilitate reading by other servers or terminals; and because the basic unit of computer storage data is Byte, this embodiment ultimately determines the total number of centroids as the number of integers that can be expressed by 1 Byte - 256.

[0151] The server collects statistics on the data distribution of speech posterior probability files of different songs and finds that the data distribution of speech posterior probability files of different songs is basically consistent, close to the normal distribution, and concentrated in the interval [-3, 3].

[0152] For each speech posterior probability file to be compressed, in order to speed up the calculation of the centroid in the clustering process, the server does not initialize the centroid randomly, but initializes it according to the following strategy:

[0153] (1) Calculate the data lower limit value and data upper limit value in the speech posterior probability file to be compressed, assign one centroid to the data lower limit value, and assign one centroid to the data upper limit value.

[0154] (2) If the data lower limit value is less than or equal to -3, calculate the average of the data lower limit value and -3, and assign a centroid to the average; if the data lower limit value is greater than -3, assign a centroid to -1.5.

[0155] (3) If the data upper limit value is greater than or equal to 3, calculate the average of the data upper limit value and 3, and assign a centroid to the average; if the data upper limit value is less than 3, assign a centroid to 1.5.

[0156] (4) Distribute the remaining 251 centroids evenly into the range [-3, 3].

[0157] After initializing the centroid, the server uses the k-means clustering algorithm to cluster the data in the compressed speech posterior probability file to obtain multiple clusters.

[0158] The server stores the centroid of each cluster in the LUT. In this embodiment, in order to retain the accuracy of the centroid as much as possible, the server still determines the number of bytes occupied by each centroid to be 4 bytes.

[0159] In the compressed speech posterior probability file, the server stores the total number of determined centroids 256 in the first byte to facilitate other servers or terminals to parse the compressed speech posterior probability file; then stores the LUT, and finally stores the index of the centroid of the cluster to which each data belongs to replace the data itself.

[0160] Assume that a speech posterior probability file has data, then the volume of the original speech posterior probability file is Byte.

[0161] After compression, the total number of centroids 256 occupies one byte; LUT occupies Byte; Since the number of indexes (i.e. the total number of centroids) is 256, each index only takes 1 Byte, that is, the number of bytes occupied by a single index is 1 Byte, and then, The volume occupied by the index of data is Byte; In summary, after compression, the size of the speech posterior probability file is Byte. In actual business, Often large, so Byte can be ignored, and the compressed size is considered Byte, the final compression effect is 25% of the original.

[0162] In this embodiment, in combination with the characteristics of concentrated distribution of data in the speech posterior probability file, the originally discretized data is further discretized, and the data is replaced with the corresponding centroid. The distribution trend and accuracy of the original data are guaranteed as much as possible with the non-uniformly distributed centroid, thereby achieving compression of the speech posterior probability file, which can reduce the download time of the speech posterior probability file and the bandwidth cost required for transmission.

[0163] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0164] Based on the same inventive concept, the embodiment of the present application also provides a file compression device for implementing the file compression method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more file compression device embodiments provided below can refer to the limitations on the file compression method above, and will not be repeated here.

[0165] In an exemplary embodiment, Fig.10 As shown, a file compression device is provided, including: a data clustering module 1002, an index determination module 1004 and a file compression module 1006, wherein:

[0166] The data clustering module 1002 is used to obtain the file to be compressed, and perform clustering processing on each data in the file to be compressed to obtain multiple clusters.

[0167] The index determination module 1004 is used to establish the index of the centroid of each cluster, and determine the index of the centroid of the cluster to which each data belongs as the index of each data; the index of each centroid is used to characterize the order of the centroid among multiple centroids; the number of bytes occupied by each index is less than the number of bytes occupied by each data.

[0168] The file compression module 1006 is used to combine the indexes of each centroid and each data into a compressed file corresponding to the file to be compressed.

[0169] In an exemplary embodiment, the index determination module 1004 is also used to store the centroids of each cluster in the first data set; determine the index of each centroid according to the storage order of each centroid in the first data set; the number of bytes occupied by each centroid is equal to the number of bytes occupied by each data.

[0170] The file compression module 1006 is further used to store the index of each data in the second data set in sequence according to the storage order of each data in the file to be compressed; and combine the first data set and the second data set to obtain a compressed file corresponding to the file to be compressed.

[0171] In an exemplary embodiment, the file compression module 1006 is further used to determine the total number of centroids preset for clustering processing; and to combine the total number of centroids, the first data set, and the second data set in order to obtain a compressed file.

[0172] In an exemplary embodiment, the number of bytes occupied by each index is the number of bytes at least required to represent the total number of centroids preset for clustering processing in a binary number system.

[0173] In an exemplary embodiment, the data clustering module 1002 is also used to initialize multiple centroids based on the data lower limit value and the data upper limit value of each data, and the data distribution interval corresponding to the file type to which the to-be-compressed file belongs; the data distribution interval is used to characterize the data concentration distribution of the corresponding file type; based on the multiple centroids obtained by initialization, each data is clustered to obtain multiple cluster clusters.

[0174] In an exemplary embodiment, the data clustering module 1002 is also used to initialize a first number of centroids based on a data lower limit value and a data upper limit value, and an interval lower limit value and an interval upper limit value of a data distribution interval; uniformly initialize a second number of centroids within the data distribution interval; the second number is obtained by subtracting a predetermined total number of centroids from the first number; and determine the first number of centroids and the second number of centroids as the initialized multiple centroids.

[0175] In an exemplary embodiment, the data clustering module 1002 is also used to initialize the data lower limit value and the data upper limit value as a centroid respectively; when the data lower limit value is less than or equal to the interval lower limit value, the average value of the data lower limit value and the interval lower limit value is initialized as a centroid; when the data lower limit value is greater than the interval lower limit value, the average value of the interval lower limit value and the midpoint value of the data distribution interval is initialized as a centroid; when the data upper limit value is greater than or equal to the interval upper limit value, the average value of the data upper limit value and the interval upper limit value is initialized as a centroid; when the data upper limit value is less than the interval upper limit value, the average value of the interval upper limit value and the midpoint value is initialized as a centroid.

[0176] In an exemplary embodiment, the file compression device also includes a data distribution statistics module, which is used to obtain multiple files under each file type; count the data distribution in each file to obtain a data distribution interval for characterizing the distribution of the file type in the data set.

[0177] Each module in the above-mentioned file compression device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0178] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.11 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store files to be compressed. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a file compression method is implemented.

[0179] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0180] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0181] In an exemplary embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0182] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0183] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0184] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0185] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A file compression method, characterized in that: The method comprises: Acquire a file to be compressed, and perform clustering processing on each data in the file to be compressed to obtain multiple clusters; Establishing the index of the centroid of each cluster, determining the index of the centroid of the cluster to which each data belongs as the index of each data; the index of each centroid is used to characterize the order of the centroid among multiple centroids; the number of bytes occupied by each index is less than the number of bytes occupied by each data; The centroids and the indexes of the data are combined into a compressed file corresponding to the file to be compressed.

2. The method according to claim 1, characterized in that The index of the centroid of each cluster is established, including: Storing the centroids of each cluster in a first data set; Determine the index of each centroid according to the storage order of each centroid in the first data set; the number of bytes occupied by each centroid is equal to the number of bytes occupied by each data; The step of forming a compressed file corresponding to the file to be compressed by combining the centroids and the indexes of the data includes: According to the storage order of the data in the to-be-compressed file, the indexes of the data are sequentially stored in the second data set; The first data set and the second data set are combined to obtain a compressed file corresponding to the file to be compressed.

3. The method according to claim 2, characterized in that The combining the first data set and the second data set to obtain a compressed file corresponding to the file to be compressed further includes: Determining a total number of centroids preset for the clustering process; The total number of centroids, the first data set, and the second data set are combined in order to obtain the compressed file.

4. The method according to claim 1, characterized in that: The number of bytes occupied by each index is, in the binary number system, the minimum number of bytes required to represent the total number of centroids preset for the clustering process.

5. The method according to any one of claims 1 to 4, characterized in that: The clustering process is performed on each data in the to-be-compressed file to obtain a plurality of clusters, including: Initialize multiple centroids according to the data lower limit value and the data upper limit value of each data and the data distribution interval corresponding to the file type to which the to-be-compressed file belongs; the data distribution interval is used to characterize the data center distribution of the file type to which the to-be-compressed file belongs; Based on the multiple centroids obtained by initialization, clustering processing is performed on the data to obtain the multiple cluster clusters.

6. The method according to claim 5, characterized in that Initializing multiple centroids according to the data lower limit value and the data upper limit value of each data and the data distribution interval corresponding to the file type to which the to-be-compressed file belongs includes: Initialize a first number of centroids according to the data lower limit value and the data upper limit value, and the interval lower limit value and the interval upper limit value of the data distribution interval; Uniformly initializing a second number of centroids within the data distribution interval; the second number is obtained by subtracting a preset total number of centroids from the first number; The first number of centroids and the second number of centroids are determined as the initialized plurality of centroids.

7. The method according to claim 6, characterized in that Initializing a first number of centroids according to the data lower limit value and the data upper limit value, and the interval lower limit value and the interval upper limit value of the data distribution interval, comprises: Initializing the data lower limit value and the data upper limit value as a centroid respectively; When the data lower limit value is less than or equal to the interval lower limit value, the average value of the data lower limit value and the interval lower limit value is initialized as a centroid; when the data lower limit value is greater than the interval lower limit value, the average value of the interval lower limit value and the midpoint value of the data distribution interval is initialized as a centroid; When the data upper limit value is greater than or equal to the interval upper limit value, the average of the data upper limit value and the interval upper limit value is initialized as a centroid. When the data upper limit value is less than the interval upper limit value, the average of the interval upper limit value and the midpoint value is initialized as a centroid.

8. The method according to claim 5, characterized in that The method further comprises: For each file type, obtain multiple files of the file type; The data distribution in each file is counted to obtain a data distribution interval for characterizing the distribution of the file type in the data set.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images