A data deletion and reorganization method based on file grouping and image popularity

Abstract

The present invention provides a data deletion and reorganization method based on file grouping and image popularity, comprising the following steps: finding and clearing duplicate data files in an image file; dividing the data files by a preset grouping strategy to obtain multiple file groups; using the popularity of each image file to select a corresponding layer reorganization mode, and performing a tiering operation on the file group in combination with a preset selection strategy to complete the layer reorganization; wherein, in the process of performing the tiering operation, the overhead in each layer after the file group is tiered is calculated, and the layer with the smallest overhead is selected as the placement layer of the file group. The present invention has the following beneficial effects: avoiding the problem of layer disorder caused by layer reorganization, and solving the problem of excessive time overhead of layer reorganization.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a data deletion and reorganization method based on file grouping and image popularity. Background Art

[0002] With the advent of the cloud computing and big data era, application development and deployment have become increasingly complex, requiring consideration of compatibility across various operating systems and dependencies. This has led to operational efficiency and resource utilization issues with traditional virtualization technologies. This has necessitated the emergence of containerization technology. Compared to traditional virtual machines, containers offer advantages such as high resource utilization, fast startup and shutdown, and simplified management and deployment, leading to their widespread adoption. Docker was one of the first companies to develop and promote container technology. Its open-source Docker Engine, released in 2013, quickly became the epitome of container technology. To date, all major cloud platforms support containers as a core deployment technology, and Docker containers offer excellent performance across a wide range of hardware environments.

[0003] Docker can encapsulate an application's executable files and a complete set of dependency files into an image. An image consists of one or more layers, and each image layer can be considered a subset of the application's total file set. When running a container from an image, each layer of the image is merged into a file system tree to start the application. Docker uses the Docker registry to provide image storage and distribution services. Clients can download images from the Docker registry or upload local images to the Docker registry for storage. Each image has a JSON file called manifest in the Docker registry, which is used to record the relevant configuration information and corresponding layer information of the image dependencies.

[0004] The fundamental principle of layer reorganization is to abandon the layer structure in the current registry and focus on the relationship between an image and the files it contains, reorganizing the files to form a new layer structure. During layer reorganization, it is crucial to place common files between different images in the same layer whenever possible, leveraging layer sharing to reduce the storage overhead caused by redundant files between layers. Current registries are massive, containing billions of files and tens of millions of layers and images. At such a scale, computing the optimal layer reorganization is impractical, necessitating more efficient methods to solve the layer reorganization problem. These methods include, but are not limited to, relaxing minor constraints, using fast heuristics, and computing layer reorganizations on a smaller subset of images.

[0005] Currently, only the article by D. Skourtis et al. proposes the concept of image layer reorganization. They initially implemented layer reorganization based on a greedy algorithm. However, their layer reorganization algorithm requires traversing and calculating the increased overhead caused by placing each file in each layer. This file-based processing method results in extremely high time overhead for layer reorganization. The challenge in solving this problem lies in finding a more efficient algorithm for layer reorganization and considering how to optimize computational efficiency, especially the time overhead when processing large-scale images.

[0006] In order to solve the problem of excessive time overhead of layer reorganization, the present invention provides a layer reorganization algorithm based on file grouping. Before layer reorganization, DARR (a Docker registry system based on data deduplication and layer reorganization technology) will divide the files into corresponding file groups according to the image collection they belong to, and then perform layer reorganization based on the greedy algorithm on the file group. This optimization reduces the time complexity of layer reorganization from the number of files to the number of file groups. Experiments have shown that the number of file groups is much smaller than the number of files, thereby greatly reducing the time overhead required for layer reorganization.

[0007] In actual workloads, images suffer from severe access skew. For example, the 1% most frequently accessed layers in dal account for 42% of all requests, while in syd, this proportion increases to 59%. Frequently accessed "hot images" require reduced network overhead and image depth, thereby reducing network traffic consumption and the number of layer download requests and improving image distribution efficiency. Low-frequency accessed "cold images" require reduced storage overhead and have low requirements for image distribution efficiency. Therefore, the present invention proposes a flexible layer reorganization mode based on image popularity. During layer reorganization, DARR flexibly adjusts the weights of the three dimensions of Storage, Network, and Depth according to the popularity of the image to meet different needs.

[0008] In Docker, the image layer is read-only. When a file in the image needs to be modified, Docker will add a writable layer to the original image layer structure and create a file with the same path and file name as the modified file. When the container is running, file access is retrieved layer by layer from top to bottom. Once the target file is found, it will return and no longer access the files in the lower layer that are overwritten by the higher layer. Therefore, there are usually many useless files in the image, that is, those files in the lower layer that are overwritten by the same-name files in the higher layer. According to statistics, among the 30 most popular images on DockerHub, the total size of useless files reaches 115MB. Considering that the weekly download volume of these images is between millions and tens of millions, it can be seen that the additional network overhead caused by these useless files is very large.

[0009] At the same time, based on the above situation, the image is composed layer by layer in the order from low to high layers. Docker creates a file with the same name in the high layer to overwrite the low-layer file to implement file modification and deletion operations. After the image layer is reorganized, the order relationship of the layers may be broken, causing the low-layer files to be placed in the high layer, thereby affecting the effectiveness of the file modification or deletion operations. How to solve the layer disorder problem is the key factor in the success of the layer reorganization algorithm. Summary of the Invention

[0010] In view of this, the present invention aims to propose a data deletion and reorganization method based on file grouping and image popularity, in order to solve at least one of the above-mentioned technical problems.

[0011] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0012] A data deletion and reorganization method based on file grouping and image popularity includes the following steps:

[0013] Find and remove duplicate data files in the image file;

[0014] Divide the data files by a preset grouping strategy to obtain multiple file groups;

[0015] The popularity of each image file is used to select a corresponding layer reorganization mode, and a tiering operation is performed on the file group in combination with a preset selection strategy to complete the layer reorganization. During the tiering operation, the overhead of each layer of the file group after tiering is calculated, and the layer with the lowest overhead is selected as the placement layer of the file group.

[0016] Furthermore, the process of searching and clearing duplicate data files in the image file includes:

[0017] Traverse all data files in each image file and obtain the level, location, path, and file name of each data file in its own image file;

[0018] Get all data files with the same path and file name in the image file, clean up the multiple data files obtained, and only keep the data file with the highest level.

[0019] Furthermore, the preset grouping strategy is specifically as follows:

[0020] Obtain the image file to which each data file belongs, divide the data files belonging to the same image file into the same file group, and add a group index map and a layer reorganization map for storing file group information to each file group in the metadata database of the registry system.

[0021] Furthermore, the process of selecting a corresponding layer reorganization mode using the popularity of each image file and performing a layering operation on the file group according to a preset selection strategy includes:

[0022] A classification operation, wherein the classification operation uses a popularity algorithm to calculate the popularity of each image file, sets a classification threshold, and divides the image files into different types based on the classification threshold and popularity;

[0023] A selection operation is performed, wherein the selection operation selects a corresponding layer reorganization mode based on the type of the current image file, and performs a layering operation using a preset selection strategy and the selected layer reorganization mode.

[0024] Furthermore, the classification operation specifically includes:

[0025] The formula for setting the classification threshold is: , ;

[0026] Among them, ε hot is the threshold of thermal image, ε cold is the threshold of cold images, x is the sorting number of each image after descending sorting, P(x) is the popularity of the image at the xth position after descending sorting, and n is the number of images in the registry;

[0027] The popularity is higher than ε hot The mirror files with popularity lower than ε are divided into hot mirrors. cold The mirror files are divided into cold mirrors, and the popularity is in ε hot and ε cold The image files between them are classified as general images.

[0028] Furthermore, the layer reorganization mode includes:

[0029] High-performance mode corresponding to hot mirroring, high-deduplication mode corresponding to cold mirroring, and normal mode corresponding to general mirroring;

[0030] In the high-performance mode, the weight ratio of storage, network, and depth is 1:2:2;

[0031] In the high deduplication mode, the weight ratio of storage, network, and depth is 2:1:1;

[0032] In the normal mode, the weight ratio of the three dimensions of storage, network, and depth is 1:1:1.

[0033] Furthermore, the preset selection strategy includes:

[0034] In an ideal mode, multiple data layers are divided into a public layer and a private layer. The public layer stores data files shared by multiple image files, and the private layer stores data files belonging to only one image file.

[0035] In the actual mode, the private layer and the public layer divided in the ideal mode are merged.

[0036] Furthermore, the formula for calculating the overhead in each layer after the file group is layered is:

[0037] cost=Storageweight*Storage+Networkweight*Network+Depthweight*Depth;

[0038] Among them, cost is the cost of each layer, Storage is the storage cost, Network is the redundant network cost, Depth is the image depth, Storageweight is the storage dimension weight value, Networkweight is the network dimension weight value, and Depthweight is the depth dimension weight value.

[0039] Furthermore, the process of adding a group index map and a layer reorganization map for storing file group information to each file group includes:

[0040] The group index mapping is composed of a plurality of key-value pairs, where the key is the file group ID and the value is the file group object; wherein the file object at least includes the name of the image file to which the file group belongs;

[0041] The layer reorganization map consists of multiple key-value pairs, where the key is the layer ID after the layer reorganization and the value is the set of all file groups in the layer.

[0042] Furthermore, the process of selecting the tier with the lowest overhead as the placement tier for the file group includes:

[0043] Traversing the group index map to obtain the corresponding relationship between the file group and the mirror file, and traversing the layer reorganization map to obtain the corresponding relationship between the file group and the layer;

[0044] Based on the relationship between file groups, layers, and image files, the storage overhead, network overhead, and depth increase corresponding to the current layer when the data group is placed in the current layer are calculated;

[0045] The total cost of the current layer is weighted by using the storage cost, network cost, and depth increase values. The layer with the lowest cost is determined by traversing and calculating the total cost of each layer.

[0046] Compared with the prior art, the data deletion and reorganization method based on file grouping and image popularity described in the present invention has the following beneficial effects:

[0047] By clearing duplicate data files in the image file, the layer disorder problem caused by layer reorganization is avoided;

[0048] At the same time, by combining the layer reorganization mode with the selection strategy, the file group layering method with the minimum overhead can be calculated, thereby solving the problem of excessive time overhead of layer reorganization. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings, which constitute part of the present invention, are provided to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are provided to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0050] Figure 1 A flowchart of a data deletion and reorganization method based on file grouping and image popularity according to an embodiment of the present invention;

[0051] Figure 2 Schematic diagram of file grouping and layered structure during the layer reorganization process according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0053] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0054] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0055] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0056] A data deletion and reorganization method based on file grouping and image popularity includes the following steps:

[0057] S1. Find and delete duplicate data files in the image file.

[0058] The process of searching and clearing duplicate data files in the image file in step S1 includes:

[0059] When performing deduplication of image files, DARR records the images to which each file belongs and their hierarchical position in each image in fileIndex;

[0060] Before layer reorganization, traverse all data files in each image file and obtain the layer, location, path, and file name of each data file in its own image file;

[0061] Get all data files with the same path and file name in the image file, clean up the multiple data files obtained, and only keep the data file with the highest level.

[0062] The process of cleaning up the obtained multiple data files is specifically to delete the association between the current image and the overwritten file in the metadata database, specifically:

[0063] If the current data file belongs to both image file a and image file b, and the current data file has a higher file hierarchy within image file a, then only the association between the current data file and image file a is retained in the metadata repository, while the association between the current data file and image file b is deleted. The "current image" mentioned above is image file b, and the "overwritten file" is a mapping reference of the current data file at a lower hierarchy level.

[0064] This ensures that there are no more useless files in the layers after layer reorganization, eliminating the problems of increased resources and layer disorder caused by this.

[0065] S2. Divide the data files by a preset grouping strategy to obtain multiple file groups.

[0066] The preset grouping strategy in step S2 is specifically:

[0067] Obtain the image file to which each data file belongs, divide the data files belonging to the same image file into the same file group, and add a group index map and a layer reorganization map for storing file group information to each file group in the metadata database of the registry system.

[0068] By grouping data files, the time complexity of layer reorganization is reduced from being based on the number of files and layers to being based on the number of file groups and layers, thereby alleviating the problem of excessive time overhead caused by large-scale layer reorganization.

[0069] Furthermore, the process of adding a group index map and a layer reorganization map for storing file group information to each file group includes:

[0070] The group index mapping is composed of a plurality of key-value pairs, where the key is the file group ID and the value is the file group object; wherein the file object at least includes the name of the image file to which the file group belongs;

[0071] Among them, the group index mapping is specifically: a key-value pair named groupIndex, the key is the file group ID, and the value is the file group object; the file group ID is calculated through SHA-256 encoding based on the image file to which the file group belongs, and the file group object includes: the file fingerprint set contained in the file group and the image name set to which the file group belongs.

[0072] The layer reorganization map consists of multiple key-value pairs, where the key is the layer ID after the layer reorganization and the value is the set of all file groups in the layer;

[0073] The layer reorganization map is specifically a key-value pair named Lgmap, where the key is the new layer ID after the layer reorganization and the value is the set of file groups contained in the layer. When the layer reorganization starts, DARR will traverse each data file, calculate the image set to which the data file belongs through SHA-256 to obtain the file group ID, and then use the file group ID as the key to put the fingerprint of the current file into the groupIndex corresponding to the file group ID.

[0074] S3. Select a corresponding layer reorganization mode using the popularity of each image file, and perform a layering operation on the file group in combination with a preset selection strategy.

[0075] The specific working process in step S3 includes:

[0076] S31, a classification operation, wherein the classification operation uses a popularity algorithm to calculate the popularity of each image file, and sets a classification threshold, and classifies the image files into different types according to the classification threshold and popularity.

[0077] The classification operation in step S31 specifically includes:

[0078] The formula for setting the classification threshold is: , ;

[0079] Among them, ε hot is the threshold of thermal image, ε cold is the threshold of cold images, x is the sorting number of each image after descending sorting, P(x) is the popularity of the image at the xth position after descending sorting, and n is the number of images in the registry (for example, if the registry has 30 images, according to the formula, the hot image threshold is P(10), that is, the popularity of the 10th image among all 30 images sorted from high to low in popularity);

[0080] The popularity is higher than ε hot The mirror files with popularity lower than ε are divided into hot mirrors. cold The mirror files are divided into cold mirrors, and the popularity is in ε hot and ε cold The image files between them are classified as general images.

[0081] S32: A selection operation is performed, wherein the selection operation selects a corresponding layer reorganization mode based on the type of the current image file, and adjusts the selected layer reorganization mode according to a preset selection strategy.

[0082] The layer reorganization mode in step S32 includes:

[0083] High-performance mode corresponding to hot mirroring, high-deduplication mode corresponding to cold mirroring, and normal mode corresponding to general mirroring;

[0084] In the high-performance mode, the weight ratio of storage, network, and depth is 1:2:2 (Storageweight:Networkweight:Depthweight=1:2:2);

[0085] In the high deduplication mode, the weight ratio of storage, network, and depth is 2:1:1 (Storageweight:Networkweight:Depthweight=2:1:1).

[0086] In the normal mode, the weight ratio of the three dimensions of storage, network, and depth is 1:1:1 (Storageweight:Networkweight:Depthweight=1:1:1).

[0087] High-performance mode can be used to process hot images with the highest download frequency, providing the greatest savings in network transmission costs and the smallest image depth; normal mode can provide both network and storage cost savings for images after layer reorganization; high-deduplication mode is suitable for cold images with lower download frequency, relaxing the restrictions on increased network costs to obtain the smallest image storage costs.

[0088] The preset selection strategy in step S32 includes an ideal mode and an actual mode;

[0089] In the ideal mode, multiple data layers are divided into public layers and private layers. The public layers store data files shared by multiple image files (hereinafter referred to as public files), and the private layers store data files belonging to only one image file (hereinafter referred to as private files).

[0090] In the actual mode, the private layer and the public layer divided in the ideal mode are merged.

[0091] Preferably, a specific execution process of the selection strategy is as follows:

[0092] Ideal mode: Theoretically, the ideal result of layer reorganization is that all images have their public files in one layer and their private files in another layer.

[0093] For example, for existing image files 1, 2, and 3, the public files a, b, and c shared by image files 1, 2, and 3 are divided into the public layer α1, the public file d shared by image files 2 and 3 is divided into the public layer α2, the private files f and g of image file 1 are divided into the private layer β1, the private file e of image file 2 is divided into the private layer β2, and the private files h and i of image file 3 are divided into the private layer β3.

[0094] Actual mode: The 200 most popular images on Docker Hub currently have 5,786 layers, with an average image depth of approximately 29. This results in higher image index management costs, higher layer request times, and decreased container performance. Therefore, the actual result of layer reorganization is a trade-off between storage, network, and image depth. This involves merging some of the ideal layers in the previous example.

[0095] For example, merging the public layer α2 and the private layer β2 into one layer reduces the image depth. However, when the user pulls image 3, file e will also be downloaded locally, resulting in additional network overhead.

[0096] Alternatively, the public layer α2 and the private layer β2 are merged into one layer, and the public layer α2 and the private layer β3 are merged into one layer. This reduces the image depth and does not increase the network overhead, but the storage overhead increases due to the storage of redundant d files.

[0097] S33: Calculate the cost of each layer after the file group is layered, and select the layer with the smallest cost as the placement layer of the file group.

[0098] The formula for calculating the overhead in each layer after the file group is layered in step S33 is:

[0099] cost=Storageweight*Storage+Networkweight*Network+Depthweight*Depth;

[0100] Among them, cost is the cost of each layer, Storage is the storage cost, Network is the redundant network cost, Depth is the image depth, Storageweight is the storage dimension weight value, Networkweight is the network dimension weight value, and Depthweight is the depth dimension weight value.

[0101] When reorganizing layers, we need to consider the increase in overhead in three dimensions. Storage represents the total file size in all layers; Network represents the total size of unused files in the image file. When the client downloads the image, these unused files are also downloaded, resulting in additional network overhead. When reorganizing layers, we calculate the increase in overhead caused by placing the file group in each layer, and then place the files in the layer with the lowest overhead.

[0102] The process of selecting the tier with the lowest overhead as the placement tier of the file group in step S33 includes:

[0103] Traversing the group index map to obtain the corresponding relationship between the file group and the mirror file, and traversing the layer reorganization map to obtain the corresponding relationship between the file group and the layer;

[0104] Based on the relationship between file groups, layers, and image files, the storage overhead, network overhead, and depth increase corresponding to the current layer when the data group is placed in the current layer are calculated;

[0105] The total cost of the current layer is weightedly calculated using the values ​​of storage cost, network cost, and depth increase. The layer with the lowest cost is determined by traversing and calculating the total cost of each layer. At the same time, the current file group is added to the corresponding layer ID in the layer reorganization map.

[0106] After executing the above step S33, when the layer is restored, it is necessary to additionally retrieve the layer reorganization map and the group index map to obtain the relationship between the layer and the data file, and then restore the image file based on the relationship.

[0107] Preferably, a specific execution process of step S33 is as follows:

[0108] The cost calculation process of placing file groups in each layer during layer reorganization:

[0109] Input: group, image, output: cost;

[0110] Loop through each layer.

[0111] 1) If the group belongs to the layer but the layer does not belong to the image, then:

[0112] Depth=1, Network=size(p);

[0113] Here, p is a file group that does not belong to the image but to the layer.

[0114] 2) If the group does not belong to the layer but the layer belongs to the image, then:

[0115] Storage=size(group),Network=size(group)×N p ;

[0116] Among them, N p The number of images that contain this layer but do not contain this file group.

[0117] 3) If group belongs to the layer and the layer belongs to image, then:

[0118] Cost=0.

[0119] 4) If the group does not belong to the layer and the layer does not belong to the image, then

[0120] Storage = size(group);

[0121] Network=size(q)+size(group)×(N q );

[0122] Among them, q is the file group that does not belong to the image but belongs to the layer, N q The number of images that contain this layer but do not contain this file group.

[0123] Depth=Depth+1

[0124] Cost = Storage × storageWeight + Network × networkWeight + Depth × depthWeight

[0125] Compare the overhead of the file group in each layer, select the layer with the lowest overhead as the placement layer of the file group, and add the current file group for the corresponding layer ID in the layer reorganization map.

[0126] In order to further illustrate the technical solution of the present invention in detail, an implementation process of a data deletion and reorganization method based on file grouping and image popularity is given as follows:

[0127] Clean up useless files.

[0128] like Figure 2 The file grouping shown is as follows: after file grouping, a, b, c belong to file group A; f, g belong to file group B; d belongs to file group C; e belongs to file group D; h, i belong to file group E.

[0129] Get the image popularity and select the layer reorganization mode: Assume that the image popularity is image1>image2>image3. n is 3; then the threshold of the hot image is p(3 / 3)=p(1), which is the popularity of the first image in descending order; then the threshold of the cold image is p(3*2 / 3)=p(2), which is the popularity of the second image in descending order; so image1 is the hot image, selected; image2 is the general image; image3 is the cold image.

[0130] For file group-based layer reorganization, select high-performance mode for Image1, normal mode for Image2, and high-deduplication mode for Image3.

[0131] 1) For image1:

[0132] Calculate the minimum cost of file group A:

[0133] If placed on the first layer, cost = 1*3+2*0+2*0=3, if placed on the second layer, cost = 1*3+2*0+2*1=5, so A is placed on the first layer;

[0134] Calculate the minimum cost of file group B:

[0135] If A is placed on the second layer, cost=1*2+2*0+2*1=4. If A is placed on the third layer, cost=1*2+2*0+2*2=6. Therefore, B is placed on the second layer.

[0136] 2) For image2:

[0137] Calculate the cost of file group C:

[0138] If placed on the second layer, cost = 1*1+1*0+1*1=2, if placed on the third layer, cost = 1*1+1*0+1*2=3, so C is placed on the second layer;

[0139] Calculate the cost of file group D:

[0140] If placed on the third layer, cost = 1*1+1*0+1*2=3, if placed on the fourth layer, cost = 1*1+1*0+1*3=4, so C is placed on the third layer.

[0141] 3) For image3:

[0142] Calculate the cost of file group E:

[0143] If placed on the third layer, cost = 2*2+1*0+1*2=6; if placed on the fourth layer, cost = 2*2+1*0+1*3=7, so C is placed on the third layer.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and description of the present invention.

[0145] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data deletion and reorganization method based on file grouping and image popularity, characterized in that: The steps include: Find and remove duplicate data files in the image file; Divide the data files by a preset grouping strategy to obtain multiple file groups; Selecting a corresponding layer reorganization mode using the popularity of each image file, and performing a tiering operation on the file group in combination with a preset selection strategy to complete the layer reorganization; wherein, during the tiering operation, calculating the overhead of each layer after the file group is tiered, and selecting the layer with the smallest overhead as the placement layer of the file group; The process of selecting a corresponding layer reorganization mode using the popularity of each image file and performing a layering operation on the file group in combination with a preset selection strategy includes: A classification operation, wherein the classification operation uses a popularity algorithm to calculate the popularity of each image file, and sets a classification threshold, and classifies the image files into different types according to the classification threshold and popularity; A selection operation, wherein the selection operation selects a corresponding layer reorganization mode based on the type of the current image file, and performs a layering operation using a preset selection strategy and the selected layer reorganization mode; The classification operation specifically includes: The formula for setting the classification threshold is: Among them, ε hot is the threshold of thermal image, ε cold is the threshold of cold images, x is the sorting number of each image after descending sorting, P(x) is the popularity of the image at the xth position after descending sorting, and n is the number of images in the registry; The popularity is higher than ε hot The image files with popularity lower than ε are divided into hot images. cold The image files of hot and ε cold The image files between are divided into general images; The layer reorganization modes include: The high-performance mode corresponds to the hot mirror, the high-deduplication mode corresponds to the cold mirror, and the normal mode corresponds to the general mirror; In the high-performance mode, the weight ratio of storage, network, and depth is 1:2:2; In the high deduplication mode, the weight ratio of the three dimensions of storage, network, and depth is 2:1:1; In the normal mode, the weight ratio of storage, network, and depth is 1:1:1; The formula for calculating the cost in each layer after the file group is layered is: cost=Storageweight*Storage+Networkweight*Network+Depthweight*Depth; Among them, cost is the cost of each layer, Storage is the storage cost, Network is the redundant network cost, Depth is the image depth, Storageweight is the storage dimension weight value, Networkweight is the network dimension weight value, and Depthweight is the depth dimension weight value.

2. A data deletion and reorganization method based on file grouping and image popularity according to claim 1, characterized in that: The process of finding and clearing duplicate data files in the image file includes: Traverse all data files in each image file and obtain the level, location, path, and file name of each data file in the image file to which it belongs; Get all data files with the same path and file name in the image file, clean up the multiple data files obtained, and only keep the data files with the highest level.

3. The data deletion and reorganization method based on file grouping and image popularity according to claim 1, characterized in that: The preset grouping strategy is specifically: Obtain the image file to which each data file belongs, divide the data files belonging to the same image file into the same file group, and add a grouping index map and a layer reorganization map for storing file group information to each file group in the metadata database of the registry system; The process of adding a group index map and a layer reorganization map for storing file group information to each file group includes: The group index mapping consists of multiple key-value pairs, where the key is the file group ID and the value is the file group object; wherein the file object at least includes the name of the image file to which the file group belongs; The layer reorganization map consists of multiple key-value pairs, where the key is the layer ID after the layer reorganization and the value is the set of all file groups in the layer.

4. The data deletion and reorganization method based on file grouping and image popularity according to claim 1, characterized in that: The preset selection strategy includes: In an ideal mode, multiple data layers are divided into a public layer and a private layer, the public layer stores data files shared by multiple image files, and the private layer stores data files belonging to only one image file; In the actual mode, the private layer and the public layer divided in the ideal mode are merged.

5. The data deletion and reorganization method based on file grouping and image popularity according to claim 3 is characterized in that: The process of selecting the layer with the smallest cost as the placement layer of the file group includes: Traversing the group index mapping to obtain the corresponding relationship between the file group and the image file, traversing the layer reorganization mapping to obtain the corresponding relationship between the file group and the layer; And according to the relationship between file groups, layers, and image files, calculate the storage overhead, network overhead, and depth increase corresponding to the current layer when the data group is placed in the current layer; The total cost of the current layer is weighted by using the storage cost, network cost, and depth increase. The total cost of each layer is traversed to determine the layer with the smallest cost.

Images