A mirror repository and compression method and device thereof

Abstract

The present application discloses an image repository and a compression method and device thereof. The method comprises: obtaining files contained in an image data layer in an image repository; re-aggregating the files contained in the obtained image data layer to form at least two file clusters, wherein there are no identical files between the at least two file clusters; generating a new image data layer based on the at least two file clusters; and updating the image meta-information layer of the image repository according to the new image data layer. The present invention can effectively compress the size of an image repository, save storage resources, and improve the deployment and upgrade efficiency of cloud services.

Description

Technical Field

[0001] The present invention relates to the field of computing technology, and in particular to an image repository and a compression method and device thereof. Background Art

[0002] With the development of cloud computing, traditional industries such as finance, medical care, and industrial manufacturing are moving to the cloud at a faster pace. As a very important virtualization technology in the field of cloud computing, Docker can well implement the encapsulation and isolation of processes and meet the requirements of these industries for data security and reliability. Therefore, Docker has also been widely used and promoted accordingly.

[0003] Docker images are the basis for building virtualized containers. As Docker is increasingly used, the size of Docker image repositories that collect Docker images is also growing, resulting in excessive use of storage space and other resources, and seriously affecting the deployment and upgrade efficiency of cloud services. Summary of the invention

[0004] In view of the above problems, the present invention is proposed to provide an image repository and a compression method and device thereof that overcome the above problems or at least partially solve the above problems.

[0005] In a first aspect, an embodiment of the present invention provides a method for compressing an image repository, comprising:

[0006] Get the files contained in the image data layer in the image repository;

[0007] Reaggregating the files contained in the acquired image data layer into at least two file clusters, wherein no identical files exist between the at least two file clusters;

[0008] Based on the at least two file clusters, generating a new mirror data layer;

[0009] According to the new image data layer, the image meta information layer of the image repository is updated.

[0010] In one embodiment, generating a new mirror data layer based on the at least two file clusters includes: generating at least one new mirror data layer corresponding to the at least two file clusters based on a mapping relationship between the at least two file clusters and the new mirror data layer to be generated; and including at least one of the file clusters in the new mirror data layer.

[0011] In one embodiment, obtaining files contained in an image data layer in an image repository includes:

[0012] Create a temporary storage directory;

[0013] Obtain a file compression package contained in the image data layer in the image repository, decompress the compression package; and store the decompressed file in the temporary storage directory;

[0014] Based on the at least two file clusters, generating a new mirror data layer includes:

[0015] Create a compressed image directory;

[0016] The files in the temporary storage directory are recompressed according to the file clusters, and according to the mapping relationship between the file clusters and the new image data layer to be generated, the compressed packages obtained by recompression are stored in the subdirectory corresponding to the new image data layer to be generated in the compressed image directory to form a new image data layer.

[0017] In one embodiment, the method further includes: reading image meta information in the image meta information layer of the image repository, and storing the image meta information in a subdirectory corresponding to the image meta information layer in the compressed image directory;

[0018] According to the new image data layer, updating the image meta information layer includes:

[0019] The image meta information in the subdirectory corresponding to the image meta information layer is updated using the mapping relationship between the image and the new image data layer.

[0020] In one embodiment, according to the files contained in the acquired image data layer, re-aggregating to form at least two file clusters includes:

[0021] According to the mapping relationship between the image and the files contained in the image data layer in the image repository, at least two rounds of aggregation operations are performed on the files in sequence to obtain at least two corresponding file clusters until all the files are aggregated into the corresponding file clusters.

[0022] In one embodiment, the aggregation operation includes:

[0023] According to the mapping relationship between the image and the file, at least one image cluster and at least one file cluster corresponding to the at least one image cluster are determined respectively;

[0024] Calculating and comparing the total data volume of the at least one file cluster, and taking the file cluster with the largest total data volume as the file cluster obtained by the current round of aggregation operation;

[0025] The mirror cluster is a mirror set containing files, wherein the mirror set includes at least one mirror; the file cluster is a set of files contained in the mirror in the mirror cluster.

[0026] In a second aspect, an embodiment of the present invention provides a compression device for an image repository, including:

[0027] The file acquisition module is used to obtain the files contained in the image data layer in the image repository;

[0028] An aggregation module, used for re-aggregating the files contained in the acquired image data layer to form at least two file clusters, wherein no identical files exist between the at least two file clusters;

[0029] A generating module, used for generating a new mirror data layer based on the at least two file clusters;

[0030] The image meta information updating module is used to update the image meta information layer of the image warehouse according to the new image data layer.

[0031] In one embodiment, the generation module is specifically used to generate at least one new mirror data layer corresponding to the at least two file clusters based on the mapping relationship between the at least two file clusters and the new mirror data layer to be generated; the new mirror data layer includes at least one of the file clusters.

[0032] In a third aspect, an embodiment of the present invention provides an image repository, wherein the image repository is obtained by the aforementioned image repository compression method; the image repository contains at least one image data layer, and there are no identical files between the at least one image data layers.

[0033] In a fourth aspect, an embodiment of the present invention provides a server, comprising: a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the instructions, when executed by the processor, can implement the aforementioned image repository compression method.

[0034] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, can implement the aforementioned image repository compression method.

[0035] The beneficial effects of the above technical solution provided by the embodiment of the present invention include at least:

[0036] In the image repository and compression method and device thereof provided by the embodiment of the present invention, the files contained in the image data layer in the image repository are first obtained, and the files contained in the obtained image data layer are re-aggregated to form at least two file clusters, and there are no identical files between at least two file clusters. The image meta-information layer of the image repository is updated according to the mapping relationship between the image, file and image data layer after layer reorganization. After the re-aggregation operation, it is ensured that there are no duplicate files between the new image data layers, the redundant data stored in the original image repository is reduced or eliminated, and the storage space size of the image data layer is effectively compressed. After the image repository is effectively reduced, on the one hand, storage resources can be saved and unnecessary consumption of storage resources can be avoided. On the other hand, because redundant files in the image repository are reduced or eliminated, the deployment and upgrade efficiency of cloud services can also be improved.

[0037] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0038] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0040] Figure 1 A flowchart of a method for compressing an image repository provided by an embodiment of the present invention;

[0041] Figure 2 Another flow chart of the image repository compression method provided by an embodiment of the present invention;

[0042] Figure 3 A schematic flow chart of an example of a method for compressing an image repository provided by an embodiment of the present invention;

[0043] Figure 4 A schematic flow chart of file re-aggregation by the mirror data layer reorganization algorithm provided in an embodiment of the present invention;

[0044] Figure 5 A structural block diagram of a compression device for an image repository provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0046] In order to solve the problem that the current Docker image repository is getting larger and larger and may cause excessive storage resources occupation, affecting the efficiency of cloud service deployment and upgrade, etc., the embodiment of the present invention provides an image repository and a compression method and device thereof.

[0047] Before describing the specific implementation of the image repository and its compression method and device provided by the embodiment of the present invention, a brief description is given of the architecture of the existing image repository. The image repository includes an image metadata layer and at least one image data layer, which can be collectively referred to as blob layers.

[0048] Each image data layer is used to store the actual storage content of the image. For example, the image data layer contains various files storing the actual content of the image.

[0049] The image meta information layer is used to store image meta information, including, for example, the Docker version (Docker version), information of each layer, and the identifier of each image (Image ID).

[0050] Docker images are constructed in a layered manner, and each image consists of a series of "image data layers". Layering achieves the effect of sharing image data layers between different images. Each image in the image repository has a mapping relationship with at least one image data layer, and each image data layer stores files with actual storage content. Due to various reasons, there will be duplicate data in the image repository. For example, different images are different versions of the same software, and the data overlaps with each other. For example, two images of different software contain the same data at the same time, etc., then, for the image repository, since these data may be stored in different image data layers, there will be more redundant data in the image repository.

[0051] Take a simple Docker image repository as an example. Assume that the Docker image repository contains 5 Docker images, which are defined as I1, I2…I5. Refer to the image composition and image data layer data size information (assuming the unit is GB) shown in Table 1 below, and the size information of each file shown in Table 2. For example, the image of I1 consists of two layers {L1, L5}, where the L1 layer contains files {F1, F2, F3, F4}, the size of the L1 layer is 25, and the L5 layer contains files {F6}, the size of the L5 layer is 3; the image of I2 consists of two layers {L2, L6}, where the L2 layer contains files {F1, F2, F3, F5}, the size of the L2 layer is 22, and the L6 layer contains files {F6, F7}, the size of the L6 layer is 8; the images of I3 to I5 are similar to this, which will not be repeated here. By calculation, the total size of the Docker image repository is: Sum{L1+L2+…+L10}=120. The number of duplicate files accounts for 17 / 26 = 65.4%, and the size of duplicate data accounts for 53 / 120 = 44.2%. It can be seen that some files may be reused between different images and between different image data layers, which leads to a lot of duplicate data in the image warehouse. As the number of images increases, the size of the entire image warehouse will become larger and larger. In view of the above problems, an embodiment of the present invention proposes a compression method for an image warehouse.

[0052] Table 1

[0053] <![CDATA[I1]]> <![CDATA[L1{F1,F2,F3,F4},25]]> <![CDATA[L5{F6},3]]> <![CDATA[I2]]> <![CDATA[L2{F1,F2,F3,F5},22]]> <![CDATA[L6{F6,F7},8]]> <![CDATA[I3]]> <![CDATA[L2{F1,F2,F3,F5},22]]> <![CDATA[L5{F6},3]]> <![CDATA[I4]]> <![CDATA[L3{F1,F2,F3},17]]> <![CDATA[L7{F5},5]]> <![CDATA[L9{F9},9]]> <![CDATA[I5]]> <![CDATA[L4{F1,F2},13]]> <![CDATA[L8{F3,F4},12]]> <![CDATA[L 10 {F8},6]]>

[0054] Table 2 (Unit: GB)

[0055]

[0056] The specific implementation of the image repository and its compression method and device are described in detail below with reference to the accompanying drawings.

[0057] Reference Figure 1 As shown, the image repository compression method provided by the embodiment of the present invention includes:

[0058] S11. Obtain the files contained in the image data layer in the image repository.

[0059] The image repository contains an image meta-information layer and at least one image data layer. In specific implementation, both can be implemented using blob layers, but the file contents stored in the two are different. The former often stores files with actual image contents, while the latter stores meta-information related to the image. Therefore, in one or more possible embodiments, the difference in file types between the two can be used to determine which blob layer in the image repository is the image meta-information layer and which is the image data layer.

[0060] For example, first find the storage address of the blob layer in the image repository, and then determine whether it is an image data layer or an image meta-information layer by the file type of the blob layer. For example, if the file type is tar or gzip (indicating a compressed file), it can be determined that the layer belongs to the image data layer. For other types, such as text information type such as txt type, it can be determined that the layer belongs to the image meta-information layer.

[0061] Get the files contained in the image data layer from the image data layer. If necessary, for example, for tar or gzip types, you can also decompress the files before performing subsequent operations.

[0062] S12: Re-aggregate the files contained in the acquired image data layer to form at least two file clusters.

[0063] In this step S12, there are no identical files between the at least two file clusters. For example, file cluster 1 includes files F1 and F2, and file cluster 2 includes files F3 and F4. The files included in the two file clusters have no intersection.

[0064] S13: Generate a new mirror data layer based on at least two file clusters.

[0065] S14. Update the image metadata layer of the image repository according to the new image data layer.

[0066] After the above steps S11 to S14, the files in the original image repository are reassembled into a new image data layer, and the image data layers do not contain the same files. Based on this, the size of the image repository can be effectively compressed. On the one hand, storage resources can be saved and unnecessary consumption of storage resources can be avoided. On the other hand, because redundant files in the image repository are reduced or eliminated, unnecessary occupation of resources is reduced, and the deployment and upgrade efficiency of cloud services can also be improved.

[0067] In one embodiment, when the above step S13 is implemented, at least one new mirror data layer corresponding to the at least two file clusters may be generated based on the mapping relationship between the at least two file clusters and the new mirror data layer to be generated; the new mirror data layer includes at least one file cluster.

[0068] There may be various mapping relationships between the at least two file clusters and the mirror data layers to be generated, such as a one-to-one relationship, that is, one file cluster generates one corresponding new mirror data layer, and the number of file clusters is equal to the number of new mirror data layers.

[0069] For example, assuming there are three file clusters: {F1, F2, F3}, {F4, F5}, and {F6}, then corresponding new image data layers blob1, blob2, and blob3 are generated respectively.

[0070] For another example, it can be a one-to-many relationship, that is, multiple file clusters can correspond to a new image data layer. In other words, more than two file clusters are used to generate the same new image data layer. Because there are no identical files between different file clusters, there will still be no identical files between the new image data layers generated in this way. In this case, the number of file clusters will be less than the number of new image data layers.

[0071] Still taking three file clusters: {F1, F2, F3}, {F4, F5} and {F6} as an example, the file clusters {F1, F2, F3}, {F4, F5} can generate a new image data layer blob1, and another file cluster {F6} generates a corresponding new image data layer blob2.

[0072] In one embodiment, referring to Figure 2 As shown, step S11 in the above process can be further implemented by the following process:

[0073] S21, create a temporary storage directory;

[0074] This temporary storage directory is used to store the files in the original image repository in preparation for subsequent file re-aggregation.

[0075] S22, obtaining a file compression package contained in the image data layer in the image repository, decompressing the compression package; and storing the decompressed file in a temporary storage directory;

[0076] Usually the files in the image data layer are compressed. Of course, it is conceivable that if the obtained file is not a compressed file but an uncompressed file, then the decompression step can be skipped directly and the obtained file can be directly stored in the temporary storage directory.

[0077] Correspondingly, refer to Figure 2 As shown, step S13 in the above process can be further implemented by the following process:

[0078] S23, creating a compressed image directory;

[0079] In the compressed image directory, there is at least one subdirectory, which corresponds to the new image data layer to be generated, and is prepared for generating the corresponding new image data layer.

[0080] S24, re-compress the files in the temporary storage directory according to the file clusters, and store the re-compressed compressed packages in the subdirectory corresponding to the new image data layer to be generated in the compressed image directory according to the mapping relationship between the file clusters and the new image data layer to be generated, so as to form a new image data layer.

[0081] In one embodiment, continue to refer to Figure 2 As shown, the above process may also include the following steps:

[0082] S25. Read the image meta information in the image meta information layer of the image repository, and store the image meta information in a subdirectory corresponding to the image meta information layer in the compressed image directory.

[0083] After the compressed image directory is created, in addition to creating a subdirectory for the new image data layer to be generated, a subdirectory corresponding to the image meta information may also be created.

[0084] Correspondingly, refer to Figure 2 The above step S14 can be implemented in the following way:

[0085] S26. Use the mapping relationship between the image and the new image data layer to update the image meta information in the subdirectory corresponding to the image meta information layer.

[0086] The compressed image directory obtained after the above steps S21 to S26 is the compressed image repository.

[0087] In one embodiment, when the above step S12 is implemented, at least two rounds of aggregation operations can be performed on the files in sequence according to the mapping relationship between the image and the files contained in the image data layer in the image repository to obtain at least two corresponding file clusters until all the files are aggregated into the corresponding file clusters.

[0088] According to the mapping relationship between the image and the image data layer, and the information of the files contained in the image layer, the mapping relationship between each image and each file in the image data layer can be determined.

[0089] Taking Table 1 as an example, there is a mapping relationship between image I1 and image data layers L1 and L5, and L1 contains files F1, F2, F3 and F4, and L5 contains file F6, so we can know the mapping relationship between image I1 and F1, F2, F3, F4 and F6. The others are similar to this and will not be repeated here.

[0090] The above-mentioned aggregation operation can be implemented in detail by referring to the following process: according to the mapping relationship between the image and the file, at least one image cluster and at least one file cluster corresponding to the at least one image cluster are determined respectively;

[0091] Calculate and compare the total data size of at least one file cluster, and use the file cluster with the largest total data size as the file cluster obtained by this round of aggregation operation;

[0092] The image cluster is a collection of images containing files, and the image collection includes at least one image; the file cluster is a collection of files contained in the images in the image cluster.

[0093] For each round of aggregation operation, when the aggregation operation is the first round of aggregation operation, the above files are all files contained in the image data layer in the image repository; when the aggregation operation is not the first round of aggregation operation, the above files are: among the files contained in the image data layer in the image repository, the remaining files after excluding the file clusters obtained by each completed round of aggregation operation. In other words, each round of aggregation operation (not the first round) needs to be performed by excluding the file clusters obtained in the previous round, so that it can be ensured that the file clusters obtained in each round will not contain the file clusters that have been aggregated in the previous round, and that the same files will not exist between file clusters.

[0094] In each round of aggregation operation, each mirror cluster (a mirror cluster is a set of various possible mirrors containing files, and each mirror cluster contains one or more mirrors) and the file cluster corresponding to each mirror cluster (a file cluster is a set of all files contained in each mirror in the mirror cluster) are determined according to the mapping relationship between each mirror and file; then the total data size of each file cluster corresponding to each mirror cluster is calculated and compared, and the file cluster with the largest total data size is mapped to the same mirror data layer, and then the remaining files are processed again according to the mapping relationship between each mirror and file, until all files have been mapped to the corresponding mirror data layer, and each new mirror data layer and the corresponding files can be obtained.

[0095] After such a file re-aggregation operation, there will no longer be duplicate redundant data between each image data layer, which will greatly reduce the overall data volume of the image warehouse.

[0096] In order to better illustrate the above-mentioned image repository compression method provided by an embodiment of the present invention, a specific example is used for illustration below.

[0097] The process of this example may include the following steps:

[0098] 1. Create a temporary storage directory TempImageDir, which is used to store temporary files generated in the middle. At the same time, create a compressed image directory CompressImageDir to store the compressed Docker image repository;

[0099] 2. In the Docker image repository, find the storage address of the blob layer and analyze the file types of all blob layer files under the address;

[0100] 3. Determine whether the file type under the blob layer is tar or gzip;

[0101] 4. If it is either of these types, create a subdirectory corresponding to the blob layer in TempImageDir and unzip the tar or gzip file into the same subdirectory;

[0102] 5. If it is another type of file, move the file to the subdirectory corresponding to CompressImageDir;

[0103] 6. According to the image granularity, the mapping relationship between each image and the new image data layer (blob layer) to be generated, and the mapping relationship between the image and the file are counted respectively, and the preset image blob layer reorganization algorithm is called to re-aggregate the files, that is, the mapping relationship between the image and the new blob layer to be generated is recalculated;

[0104] 7. Package the files in TempImageDir according to the latest blob layer, and store the packaged files in the directory corresponding to CompressImageDir.

[0105] The packaged files are stored in the directory corresponding to the corresponding new blob layer.

[0106] 8. Update the blob layer information in the image manifest information (image meta information), and record the new manifest information in CompressImageDir;

[0107] 9. Delete TempImageDir. CompressImageDir is the compressed image repository obtained at the end.

[0108] Reference Figure 3 What is shown is a schematic diagram of the above process, and the above process can be performed when the Docker repository is offline. Figure 3 After analyzing the file type of the image blob layer, such as whether it is a tar or gzip file, it is determined whether it is an image data layer. If so, the layer is decompressed, and the distribution information of the image (the mapping relationship between the image and the file) is counted for layer reorganization. After the layer is reorganized, the layer information in the image meta information is updated. If not, for example, it is an ASCII text file, the layer is considered to be an image meta information layer.

[0109] Next, we will use the example of the Docker image repository shown in Table 1 and Table 2 to explain the specific implementation process of the aforementioned image blob layer reorganization algorithm to re-aggregate files:

[0110] First, according to the mapping relationship information of each image, blob layer and file in Table 1 and Table 2, it is converted into a relationship matrix between image and file (Image-File) (hereinafter referred to as Image-File relationship matrix) according to the image granularity. The conversion result can be shown in Table 3. Each element value in Table 3 is the size value of each file (File). After each round, all processed elements (files) will be marked. The marked elements are called aggregated elements. In the initial state, all elements are in an unprocessed state and are marked as isolated elements (i.e., elements that have not undergone aggregation operations).

[0111] Table 3

[0112]

[0113] The first round of aggregation operation process is as follows:

[0114] According to the Image-File relationship matrix shown in Table 3 above, we can calculate the image set containing File as a non-zero element, called the image cluster, represented by C1, C2...Cn. For example, the C1 image cluster is {I1, I2, I3, I4, I5}, and the C2 image cluster is {I1, I5}. Then, we can find the set of all files that satisfy C1, C2...Cn (called file cluster) based on the image cluster;

[0115] For example, the file cluster that satisfies C1 is {F1, F2, F3}. The file cluster that satisfies C2 is {F1, F2, F3, F4}... and so on. Calculate the size of the file cluster and the mirror cluster. For example, the size of C1 is 85, expressed in terms of density. Table 4 shows all mirror cluster combinations, as well as the corresponding file clusters and density values.

[0116] Table 4

[0117]

[0118] In Table 4 above, only some mirror clusters with higher density are listed, and mirror clusters with lower density, such as {I1}, etc., can be ignored in the calculation.

[0119] Compare the density values ​​of all the image clusters in Table 4 above, and determine that the file cluster corresponding to the image cluster with the largest density is used to form a new image blob layer, namely, the L1 layer. For example, the density value of C1 in Table 4 is the largest, so the file clusters in C1 will form a new blob layer. The corresponding relationship between the Docker image, blob layer and size based on the new blob layer is shown in Table 5 below.

[0120] Table 5

[0121]

[0122]

[0123] After this first round of aggregation operation, the processed C1 column in Table 4 does not need to be processed again. At the same time, the Image-File relationship matrix is ​​updated, and the processed file elements are classified as aggregated elements and no longer isolated elements. The remaining C2 to C6 continue to enter the next round of aggregation operation.

[0124] The second round of aggregation operation is based on the Image-File relationship matrix in Table 3 above. After removing the aggregated file clusters {F1, F2, F3}, the mapping relationship between each image and file is determined again, and the relationship matrix between the image cluster, file cluster and density is shown in Table 6 below:

[0125] Table 6

[0126]

[0127] From Table 6 above, it can be seen that the image cluster with the highest density is {I1, I5}, and the corresponding file cluster is {F4}. Therefore, it is determined that the file cluster {F4} corresponding to the image cluster with the highest density is used to form a new image blob layer, namely, layer L2.

[0128] Update the Docker image composition and size information based on the new blob layer as shown in Table 7:

[0129] Table 7

[0130] <![CDATA[I1]]> <![CDATA[L1{F1,F2,F3},17]]> <![CDATA[L2{F4},8]]> <![CDATA[I2]]> <![CDATA[L1{F1,F2,F3},17]]> <![CDATA[I3]]> <![CDATA[L1{F1,F2,F3},17]]> <![CDATA[I4]]> <![CDATA[L1{F1,F2,F3},17]]> <![CDATA[I5]]> <![CDATA[L1{F1,F2,F3},17]]> <![CDATA[L2{F4},8]]>

[0131] The third round of aggregation operation is similar to the first and second rounds. Based on the Image-File relationship matrix in Table 3 above, after removing the aggregated F1, F2, F3 and F4, the matrix of image clusters, file clusters and density can be obtained as shown in Table 8 below:

[0132] Table 8

[0133]

[0134] It can be seen from Table 8 above that the image cluster with the highest density is {I2, I3, I4}, and the corresponding file cluster is {F5}. Therefore, the file cluster {F5} corresponding to the image cluster with the highest density is formed into a new image blob layer, namely, layer L3.

[0135] Update the Docker image composition and size information based on the new blob layer as shown in Table 9:

[0136] Table 9

[0137] <![CDATA[I1]]> <![CDATA[L1{F1,F2,F3},17]]> <![CDATA[L2{F4},8]]> <![CDATA[I2]]> <![CDATA[L1{F1,F2,F3},17]]> <![CDATA[L3{F5},5]]> <![CDATA[I3]]> <![CDATA[L1{F1,F2,F3},17]]> <![CDATA[L3{F5},5]]> <![CDATA[I4]]> <![CDATA[L1{F1,F2,F3},17]]> <![CDATA[L3{F5},5]]> <![CDATA[I5]]> <![CDATA[L1{F1,F2,F3},17]]> <![CDATA[L2{F4},8]]>

[0138] The fourth round of aggregation operation is similar to the previous rounds. Based on the Image-File relationship matrix in Table 3 above, after removing the aggregated F1, F2, F3, F4 and F5, the matrix of image clusters, file clusters and density can be obtained as shown in Table 10 below:

[0139] Table 10

[0140]

[0141] It can be seen from Table 10 above that the image cluster with the highest density is {I1, I2, I3}, and the corresponding file cluster is {F6}. Therefore, the file cluster {F6} corresponding to the image cluster with the highest density is formed into a new image blob layer, namely, layer L4.

[0142] Update the Docker image composition and size information based on the new blob layer as shown in Table 11:

[0143] Table 11

[0144] <![CDATA[I1]]> <![CDATA[L1{F1,F2,F3},17]]> <![CDATA[L2{F4},8]]> <![CDATA[L4{F6},3]]> <![CDATA[I2]]> <![CDATA[L1{F1,F2,F3},17]]> <![CDATA[L3{F5},5]]> <![CDATA[L4{F6},3]]> <![CDATA[I3]]> <![CDATA[L1{F1,F2,F3},17]]> <![CDATA[L3{F5},5]]> <![CDATA[L4{F6},3]]> <![CDATA[I4]]> <![CDATA[L1{F1,F2,F3},17]]> <![CDATA[L3{F5},5]]> <![CDATA[I5]]> <![CDATA[L1{F1,F2,F3},17]]> <![CDATA[L2{F4},8]]>

[0145] The fifth round of aggregation operation is similar to the previous rounds. Based on the Image-File relationship matrix in Table 3 above, after removing the aggregated F1, F2, F3, F4, F5 and F6, the matrix of image clusters, file clusters and density can be obtained as shown in Table 12 below:

[0146] Table 12

[0147]

[0148] As can be seen from Table 12 above, the image cluster with the highest density is {I4}, and the corresponding file cluster is {F9}. Therefore, the file cluster {F9} corresponding to the image cluster with the highest density is used to form a new image blob layer, namely, layer L5. The remaining {F8} is used to form a new image blob layer, namely, layer L6. The remaining {F7} is used to form a new image blob layer, namely, layer L7.

[0149] Update the Docker image composition and size information based on the new blob layer as shown in Table 13:

[0150] Table 13

[0151] <![CDATA[I1]]> <![CDATA[L1{F1,F2,F3},17]]> <![CDATA[L2{F4},8]]> <![CDATA[L4{F6},3]]> <![CDATA[I2]]> <![CDATA[L1{F1,F2,F3},17]]> <![CDATA[L3{F5},5]]> <![CDATA[L4{F6},3]]> <![CDATA[L7{F7},5]]> <![CDATA[I3]]> <![CDATA[L1{F1,F2,F3},17]]> <![CDATA[L3{F5},5]]> <![CDATA[L4{F6},3]]> <![CDATA[I4]]> <![CDATA[L1{F1,F2,F3},17]]> <![CDATA[L3{F5},5]]> <![CDATA[L5{F9},9]]> <![CDATA[I5]]> <![CDATA[L1{F1,F2,F3},17]]> <![CDATA[L2{F4},8]]> <![CDATA[L6{F8},6]]>

[0152] After the above five rounds of aggregation operations, the aggregation results of all files in the 7 new blob layers and the mapping relationship between the new blob layers to be generated and the images have been determined. All elements (files) in Table 3 above have been aggregated, and there are no isolated elements, so the calculation process ends.

[0153] The flowchart of the above algorithm can be simply summarized as Figure 4 ,exist Figure 4 In the process, the image-file relationship matrix is ​​first determined, and then it is determined whether there are isolated unaggregated elements in the matrix. If so, all image clusters are calculated based on the isolated elements (files) in the matrix, and then the corresponding file clusters and density values ​​are calculated based on the image clusters. All density values ​​are compared, and the file clusters with the maximum density value are formed into a new image data layer, the information of the new image data layer is recorded, and the files in the aggregated file cluster are marked as aggregated. This is until all elements in the image-file relationship matrix are aggregated.

[0154] The total size of each image data layer in the aggregated image repository is the sum of the sizes of F1 to F7, that is, 7+6+4+8+5+3+5+6+9=53, which is more than half the size of 120 before compression.

[0155] Based on the same inventive concept, an embodiment of the present invention also provides an image repository and a compression device, a server and a computer-readable storage medium. Since the principles of the problems solved by these devices and servers are similar to the compression method of the aforementioned image repository, the implementation of the device and server, etc. can refer to the implementation of the aforementioned method, and the repeated parts will not be repeated.

[0156] Reference Figure 5 As shown, an embodiment of the present invention provides a compression device for an image repository, including:

[0157] The file acquisition module 51 is used to acquire the files contained in the image data layer in the image repository;

[0158] An aggregation module 52, configured to aggregate the acquired files contained in the image data layer into at least two file clusters, wherein no identical files exist between the at least two file clusters;

[0159] A generating module 53, used for generating a new mirror data layer based on at least two file clusters;

[0160] The image meta information updating module 54 is used to update the image meta information layer of the image repository according to the new image data layer.

[0161] In one embodiment, referring to Figure 5 As shown, the above-mentioned generation module 53 is specifically used to generate at least one new mirror data layer corresponding to the at least two file clusters based on the mapping relationship between the at least two file clusters and the new mirror data layer to be generated; the new mirror data layer contains at least one of the file clusters.

[0162] In one embodiment, the file acquisition module 51 is further used to create a temporary storage directory; obtain a file compression package contained in the image data layer in the image repository, decompress the compression package; and store the decompressed file in the temporary storage directory;

[0163] Correspondingly, the generation module 53 is further used to create a compressed image directory; re-compress the files in the temporary storage directory according to the file clusters, and store the re-compressed compressed packages in the subdirectory corresponding to the new image data layer to be generated in the compressed image directory according to the mapping relationship between the file clusters and the new image data layer to be generated, so as to form a new image data layer.

[0164] In one embodiment, referring to Figure 5 As shown, the image meta-information update module 54 in the compression device of the above-mentioned image warehouse is also used to read the image meta-information in the image meta-information layer of the image warehouse, and store the image meta-information in the sub-directory corresponding to the image meta-information layer in the compressed image directory; use the mapping relationship between the image and the new image data layer to update the image meta-information in the sub-directory corresponding to the image meta-information layer.

[0165] In one embodiment, the aggregation module 52 is further used to perform at least two rounds of aggregation operations on the files in sequence according to the mapping relationship between the image and the files contained in the image data layer in the image repository, to obtain at least two corresponding file clusters, until all the files are aggregated into the corresponding file clusters.

[0166] In one embodiment, the above-mentioned aggregation operation includes: determining at least one mirror cluster and at least one file cluster corresponding to the at least one mirror cluster according to the mapping relationship between the mirror and the file; calculating and comparing the total data volume of the at least one file cluster, and taking the file cluster with the largest total data volume as the file cluster obtained in this round of aggregation operation; the mirror cluster is a mirror set containing files, and the mirror set includes at least one mirror; the file cluster is a set of files contained in the mirror in the mirror cluster.

[0167] The embodiment of the present invention further provides an image warehouse, which is obtained by the compression method of the image warehouse described above, and the image warehouse contains at least two image data layers, and there are no identical files between at least two image data layers.

[0168] An embodiment of the present invention further provides a server, comprising: a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the instructions, when executed by the processor, can implement the aforementioned image repository compression method.

[0169] The above-mentioned servers may be servers in various cloud computing fields, such as servers in various private clouds, public clouds or hybrid clouds (or proprietary clouds).

[0170] An embodiment of the present invention further provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the aforementioned image repository compression method can be implemented.

[0171] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0172] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0173] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0174] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0175] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for compressing an image repository, characterized in that: include: Get the files contained in the image data layer in the image repository; Reaggregating the files contained in the acquired image data layer into at least two file clusters, wherein no identical files exist between the at least two file clusters; Based on the at least two file clusters, generating a new mirror data layer; According to the new image data layer, updating the image metadata layer of the image repository; The generating a new mirror data layer based on the at least two file clusters includes: Based on the mapping relationship between the at least two file clusters and the new mirror data layer to be generated, at least one new mirror data layer corresponding to the at least two file clusters is generated; the new mirror data layer includes at least one of the file clusters.

2. The method according to claim 1, characterized in that Get the files contained in the image data layer in the image repository, including: Create a temporary storage directory; Obtain a file compression package contained in the image data layer in the image repository, decompress the compression package; and store the decompressed file in the temporary storage directory; Based on the at least two file clusters, generating a new image data layer, including: creating a compressed image directory; The files in the temporary storage directory are recompressed according to the file clusters, and according to the mapping relationship between the file clusters and the new image data layer to be generated, the compressed packages obtained by recompression are stored in the subdirectory corresponding to the new image data layer to be generated in the compressed image directory to form a new image data layer.

3. The method according to claim 2, characterized in that The method further comprises: reading the image meta information in the image meta information layer of the image warehouse, and storing the image meta information in a subdirectory corresponding to the image meta information layer in the compressed image directory; According to the new image data layer, updating the image meta information layer includes: The image meta information in the subdirectory corresponding to the image meta information layer is updated using the mapping relationship between the image and the new image data layer.

4. The method according to claim 2 or 3, characterized in that According to the files contained in the acquired image data layer, re-aggregating to form at least two file clusters, including: According to the mapping relationship between the image and the files contained in the image data layer in the image repository, at least two rounds of aggregation operations are performed on the files in sequence to obtain at least two corresponding file clusters until all the files are aggregated into the corresponding file clusters.

5. The method according to claim 4, characterized in that The aggregation operation comprises: According to the mapping relationship between the image and the file, at least one image cluster and at least one file cluster corresponding to the at least one image cluster are determined respectively; Calculating and comparing the total data volume of the at least one file cluster, and taking the file cluster with the largest total data volume as the file cluster obtained by the current round of aggregation operation; The mirror cluster is a mirror set containing files, wherein the mirror set includes at least one mirror; the file cluster is a set of files contained in the mirror in the mirror cluster.

6. A compression device for an image repository, characterized in that: include: The file acquisition module is used to obtain the files contained in the image data layer in the image repository; An aggregation module, used for re-aggregating the files contained in the acquired image data layer to form at least two file clusters, wherein no identical files exist between the at least two file clusters; A generating module, used for generating a new mirror data layer based on the at least two file clusters; An image meta information updating module, used for updating the image meta information layer of the image warehouse according to the new image data layer; The generation module is specifically configured to generate at least one new mirror data layer corresponding to the at least two file clusters based on a mapping relationship between the at least two file clusters and the new mirror data layer to be generated; the new mirror data layer includes at least one of the file clusters.

7. A mirror repository, characterized in that: The image repository is obtained by the image repository compression method according to any one of claims 1 to 5; the image repository contains at least two image data layers, and there are no identical files between the at least two image data layers.

8. A server, characterized in that: include: A memory, a processor, and a computer instruction stored in the memory and executable on the processor, wherein the instruction, when executed by the processor, can implement the image repository compression method as described in any one of claims 1-5.

9. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by the processor, the image repository compression method as described in any one of claims 1 to 5 can be implemented.

Images