Storage control program, storage control method, and storage control device

Abstract

To prevent data from being deleted from an external storage in a short period of time.SOLUTION: A storage control apparatus 1 is configured to: identify chunks CK1, CK5, and CK7 that are referenced by files requested to be stored in an external storage 2 during the latest fixed period from among storage datasets stored in the external storage 2; generate chunks CK1', CK5', and CK7' having the same data as the chunks CK1, CK5, and CK7, respectively; replace the number of references corresponding to the chunks CK1, CK5, and CK7 with the chunks CK1', CK5', and CK7', respectively, and update the number of references corresponding to each of the chunks CK1, CK5, and CK7 to 0; and combine the chunks CK1', CK5', and CK7' to generate a new storage dataset ACx and store it in the external storage 2.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a storage control program, a storage control method, and a storage control device. [Background technology]

[0002] Storage services that provide storage space over a network, such as online storage services and cloud storage services, are becoming increasingly popular. Storage gateways function as a gateway between such storage services and customer computers.

[0003] For example, a storage gateway has been proposed that relays write data between a customer's computer and external storage such as cloud storage. This storage gateway accepts data write requests on a file-by-file basis from the customer's computer, deduplicates the data requested to be written, and transfers the deduplicated data to external storage on an object-by-object basis.

[0004] Furthermore, with regard to data deduplication technology, for example, a storage device has been proposed that, when merging multiple chunks containing the same data, transfers the reference count corresponding to one chunk to the other chunk.Furthermore, with regard to data storage technology for storage, for example, an information processing system has been proposed that classifies file sets that have relatively close word appearance frequencies and write times in the same cluster and stores them on a common tape medium. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2019-95925 [Patent Document 2] Japanese Patent Application Laid-Open No. 2017-45177 [Patent Document 3] Japanese Patent Application Laid-Open No. 2018-190393 Summary of the Invention [Problem to be solved by the invention]

[0006] One storage service is one that provides archive storage suitable for long-term data storage. The storage gateway described above may also use archive storage as an external storage destination for data. In this case, for example, a dataset for archive storage containing one or more objects is generated, and the dataset is stored in the archive storage as a unit.

[0007] Furthermore, defragmentation of datasets stored in archive storage is performed, for example, in the following procedure: A first dataset with many chunks with a reference count of "0" is identified from among the datasets, and the first dataset is retrieved and deleted from the archive storage. Valid chunks with a reference count other than "0" are extracted from the retrieved dataset, and a second dataset is newly generated (reconstructed) using the extracted chunks and stored in the archive storage.

[0008] However, when defragmentation processing using such a procedure is repeated, a case may occur in which a dataset newly generated by defragmentation processing is stored in archive storage, and then the dataset is deleted from archive storage in a short period of time. In this case, the number of chunks with a reference count of "0" increases early in a dataset that has been reconstructed and stored in archive storage, and the dataset is retrieved for reconstruction and deleted from archive storage.

[0009] In archive storage, a minimum storage period for a dataset is set for the purpose of long-term storage, and an additional fee may be incurred if the dataset is deleted before the minimum storage period has elapsed. As described above, if a dataset generated by defragmentation processing and stored in archive storage is deleted in a short period of time, an additional fee for the deletion may be incurred.

[0010] In one aspect, the present invention aims to provide a storage control program, a storage control method, and a storage control device that can prevent data from being deleted from an external storage device in a short period of time. [Means for solving the problem]

[0011] In one proposal, a computer obtains a plurality of chunks by eliminating duplicates from a plurality of split data sets obtained by splitting each of a plurality of files for which writing is requested, and generates a plurality of storage data sets by combining two or more chunks selected from the plurality of chunks, and upon receiving a storage request for a first file of the plurality of files, identifies one or more first storage data sets including chunks of the plurality of chunks referenced by the first file, and stores the one or more first storage data sets in external storage, and also increases the reference count associated with the chunk of the plurality of chunks referenced by the first file by the number referenced by the split data set included in the first file of the plurality of split data sets, and upon receiving a delete request for a second file of the plurality of files, decreases the reference count associated with the chunk of the plurality of chunks referenced by the second file of the plurality of split data sets by the number referenced by the split data set included in the second file of the plurality of split data sets. a storage control program for executing the following process: from a storage dataset among a plurality of storage datasets stored in the external storage, identify one or more first chunks that are referenced by a file among a plurality of files for which storage in the external storage has been requested within a recent fixed period, and acquire them from the external storage; generate one or more second chunks each having the same data as the one or more first chunks; assign one or more reference numbers corresponding to each of the one or more first chunks to one or more second chunks, respectively, and update the reference numbers corresponding to each of the one or more first chunks to 0; combine the one or more second chunks to generate a new storage dataset and store it in the external storage; and, when a second storage dataset having a reference number of 0 corresponding to all chunks in a corresponding storage dataset is detected from the storage datasets stored in the external storage among the plurality of storage datasets and the new storage dataset, delete the second storage dataset from the external storage.

[0012] Also, one proposal provides a storage control method in which a computer executes processing similar to the processing based on the above storage control program. Furthermore, in one proposal, a storage control device is provided that executes processing similar to the processing based on the above storage control program. [Effects of the Invention]

[0013] On the one hand, it can prevent data from being deleted from external storage in a short period of time. [Brief explanation of the drawings]

[0014] [Figure 1] 1 illustrates an example of a configuration and a processing example of a storage system according to a first embodiment; [Figure 2] FIG. 10 illustrates an example of a configuration of an information processing system according to a second embodiment. [Figure 3] FIG. 2 illustrates an example of the hardware configuration of a cloud storage gateway. [Figure 4] FIG. 2 is a diagram illustrating an example of the configuration of processing functions provided in a cloud storage gateway. [Figure 5] FIG. 10 is a diagram illustrating an example of the data configuration of a chunk map table. [Figure 6] FIG. 10 is a diagram illustrating an example of the data configuration of a chunk management table. [Figure 7] FIG. 10 is a diagram illustrating an example of object generation. [Figure 8] FIG. 10 is a diagram illustrating an example of the data configuration of a chunk group management table. [Figure 9] FIG. 10 is a diagram illustrating an example of the data configuration of a defragmentation determination table. [Figure 10] FIG. 1 is a first diagram illustrating an example of data storage processing in object storage and archive storage. [Figure 11] FIG. 2 is a second diagram illustrating an example of data storage processing in object storage and archive storage. [Figure 12]FIG. 3 is a third diagram illustrating an example of data storage processing in object storage and archive storage. [Figure 13] FIG. 1 is a first diagram showing a comparative example of defragmentation processing. [Figure 14] FIG. 2 is a second diagram showing a comparative example of defragmentation processing. [Figure 15] FIG. 11 is a first diagram illustrating an example of defragmentation processing in the second embodiment. [Figure 16] FIG. 10 is a second diagram illustrating an example of defragmentation processing in the second embodiment. [Figure 17] FIG. 10 is a third diagram illustrating an example of defragmentation processing in the second embodiment. [Figure 18] 10 is a first example of a flowchart illustrating a procedure for a file writing process. [Figure 19] 10 is a second example of a flowchart illustrating the procedure of a file writing process. [Figure 20] 10 is an example of a flowchart illustrating a procedure for a data storage process in archive storage. [Figure 21] 10 is an example of a flowchart illustrating a procedure for deleting data from archive storage. [Figure 22] 10 is a first example of a flowchart illustrating a procedure for defragmentation processing. [Figure 23] 10 is a second example of a flowchart illustrating the procedure of a defragmentation process. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. [First embodiment] 1 is a diagram showing an example of the configuration and processing of a storage system according to Embodiment 1. The storage system shown in FIG.

[0016] The storage control device 1 executes control processes such as storing data in the external storage 2, retrieving data from the external storage 2, and deleting data stored in the external storage 2 in response to requests from, for example, a client device (not shown). The storage control device 1 is, for example, a computer having a processor (not shown). In this case, the above control processes are realized, for example, by the processor executing a program.

[0017] The external storage 2 is a storage area connected externally to the storage control device 1. The external storage 2 is suitable for long-term storage of data, but not for short-term storage of data. For example, a minimum storage period for data is set in the external storage 2, and if data is deleted from the external storage 2 before the minimum storage period has elapsed since the data was stored, an additional fee will be incurred.

[0018] The processing of the storage control device 1 will be explained below. The storage control device 1 acquires multiple chunks by eliminating duplicates from multiple divided data sets obtained by dividing each of the multiple files requested to be written. The storage control device 1 also generates multiple storage data sets by combining two or more chunks selected from the acquired multiple chunks. In the example of FIG. 1, chunks CK1 to CK9, . . . , CK21, and CK22 are acquired from m files FL1 to FLm. Furthermore, n storage data sets AC1 to ACn are generated from chunks CK1 to CK9, . . . , CK21, and CK22.

[0019] 1, the symbols of the corresponding chunks are displayed for each divided data set included in files FL1 to FLm. For example, file FL1 is divided into four divided data sets, which correspond to chunks CK1, CK2, CK3, and CK4, respectively. File FL2 is divided into four divided data sets, which correspond to chunks CK1, CK5, CK3, and CK6, respectively. It can be seen that divided data sets with the same content corresponding to chunks CK1 and CK3 overlap between files FL1 and FL2.

[0020] Furthermore, each chunk from which duplicates have been removed is associated with a reference number. The reference number indicates how many of the divided data sets of the file stored in the external storage 2 reference the corresponding chunk. The initial value of the reference number is "0."

[0021] When the storage control device 1 receives a storage request for a first file among the multiple files, it identifies one or more stored datasets that include chunks referenced from the first file. The storage control device 1 stores the identified one or more first stored datasets in the external storage 2. At the same time, the storage control device 1 increases the reference count corresponding to the chunks referenced from the first file by the number of references from divided datasets included in the first file.

[0022] For example, when a request is made to store file FL1, the storage control device 1 identifies a storage dataset AC1 that includes chunks CK1 to CK4 referenced by file FL1, and stores the storage dataset AC1 in the external storage 2. At the same time, the storage control device 1 increases the reference counts corresponding to the chunks CK1 to CK4 by the number of references from the divided datasets included in file FL1. For example, since chunk CK1 is referenced by one divided dataset in file FL1, the reference count corresponding to chunk CK1 is increased by "1". As a result, the reference count corresponding to chunk CK1 becomes "1".

[0023] Assume that a request to store files FL1 to FLm in external storage 2 is made using this procedure, and all of the stored data sets AC1 to ACn including the chunks corresponding to files FL1 to FLm are stored in external storage 2. At this time, the reference counts corresponding to chunks CK1 to CK9 are respectively "8", "1", "3", "1", "6", "2", "4", "1", and "1". Also, the reference counts corresponding to chunks CK21 and CK22 are respectively "1" and "1".

[0024] Furthermore, when the storage control device 1 receives a request to delete a second file, it decreases the reference counts corresponding to chunks referenced by this second file by the number of references from the divided data sets included in the second file. For example, suppose that a request is made to delete files FL1 and FL2. In this case, as shown in FIG. 1, the storage control device 1 updates the reference counts corresponding to chunks CK1 to CK6 referenced by files FL1 and FL2 from "8", "1", "3", "1", "6", and "2" to "6", "0", "1", "0", "5", and "1", respectively.

[0025] Furthermore, the storage control device 1 identifies one or more chunks that are referenced by files that have been requested to be stored in the external storage 2 within a recent fixed period, from among the storage data sets AC1 to ACn stored in the external storage 2. When the storage control device 1 identifies chunks that satisfy such conditions, it acquires the identified chunks from the external storage 2.

[0026] 1, it is assumed that chunks CK1, CK5, and CK7 are identified as chunks that satisfy the above conditions. The storage control device 1 acquires chunks CK1, CK5, and CK7 by, for example, retrieving a stored data set AC1 that includes chunks CK1 and CK5 and a stored data set AC2 that includes chunk CK7 from the external storage 2.

[0027] The storage control device 1 generates new chunks CK1', CK5', and CK7' having the same data as the acquired chunks CK1, CK5, and CK7, respectively. At this time, the storage control device 1 replaces the reference counts corresponding to the original chunks CK1, CK5, and CK7 with the new chunks CK1', CK5', and CK7', respectively. In the example of FIG. 1, the reference count "6" of chunk CK1 is replaced with chunk CK1', the reference count "5" of chunk CK5 is replaced with chunk CK5', and the reference count "4" of chunk CK7 is replaced with chunk CK7'. The storage control device 1 also updates the reference counts corresponding to the original chunks CK1, CK5, and CK7 to "0."

[0028] Although not shown, the storage control device 1 updates information indicating the correspondence between a file and the original chunks CK1, CK5, and CK7 to indicate the correspondence between the file and the new chunks CK1', CK5', and CK7'. For example, if the original chunk CK1 is associated with a certain divided data set of a certain file, the chunk corresponding to this divided data set is updated to the new chunk CK1'.

[0029] Then, the storage control device 1 combines the new chunks CK1', CK5', and CK7' to generate a new storage dataset ACx, and stores this storage dataset ACx in the external storage 2. On the other hand, the storage datasets AC1 and AC2 that include the chunks CK1, CK5, and CK7 that satisfy the above conditions continue to be stored in the external storage 2, with only the reference numbers being changed as described above.

[0030] Furthermore, when a storage dataset in which the reference count corresponding to all chunks in the corresponding storage dataset is "0" is detected from among the storage datasets AC1 to ACn stored in the external storage 2 and the new storage dataset ACx, the storage control device 1 deletes this storage dataset from the external storage 2. However, at the time when the storage dataset ACx is stored in the external storage 2 as described above, a storage dataset in which the reference count corresponding to all chunks is "0" is not detected.

[0031] After this, although not shown, suppose that a request is made to delete files FL3 and FL4. In this case, the storage control device 1 reduces the reference counts corresponding to chunks referenced by files FL3 and FL4 by the number of references from the divided data sets included in files FL3 and FL4. As a result of this process, the reference counts corresponding to chunks CK3, CK6, CK8, and CK9 become "0."

[0032] As a result, the reference counts of all chunks included in the stored datasets AC1 and AC2 become "0", and the storage control device 1 deletes the stored datasets AC1 and AC2 from the external storage 2. On the other hand, the stored dataset ACx that has just been stored in the external storage 2 remains stored without being deleted.

[0033] Here, a comparative example of defragmentation processing when a request to delete files FL1 and FL2 is received will be described. In this comparative example, when a file deletion request results in a stored dataset with many chunks whose reference count is "0," the stored dataset is extracted. Then, of the chunks in the extracted stored dataset, a new stored dataset is generated from only those chunks whose reference count is not "0," and this is stored in the external storage 2. Meanwhile, the extracted stored dataset is deleted from the external storage 2.

[0034] For example, after the reference counts are updated in response to a request to delete files FL1 and FL2, stored dataset AC1 is identified as a stored dataset with many chunks whose reference counts are "0." Then, stored dataset AC1 is extracted, and a new stored dataset is generated from chunks CK1, CK3, CK5, and CK6 whose reference counts are not "0," and stored in external storage 2. Meanwhile, stored dataset AC1 is deleted from external storage 2.

[0035] Next, when a request is made to delete files FL3 and FL4, the reference counts of chunks CK3 and CK6 become "0." In this case, the newly stored stored dataset is identified as the stored dataset with the largest number of chunks whose reference count becomes "0," and is extracted and deleted from the external storage 2. In this way, in the comparative example, stored datasets that are deleted shortly after being stored in the external storage 2 are likely to occur.

[0036] In contrast to this, the storage control device 1 of this embodiment does not delete the stored data set AC1, which has many chunks with reference counts of "0" after the reference counts are updated in response to a request to delete files FL1 and FL2, but leaves it in the external storage 2. Instead, the storage control device 1 identifies chunks CK1, CK5, and CK7 that are referenced by files that have been requested to be stored in the external storage 2 within the most recent fixed period.

[0037] The storage control device 1 generates new chunks CK1', CK5', and CK7' having the same data as the identified chunks CK1, CK5, and CK7, respectively. At this time, the reference counts corresponding to the original chunks CK1, CK5, and CK7 are respectively replaced with the reference counts of the new chunks CK1', CK5', and CK7', and the reference counts corresponding to the original chunks CK1, CK5, and CK7 are updated to "0". Then, the storage control device 1 combines the new chunks CK1', CK5', and CK7' to generate a new storage dataset ACx, and stores this storage dataset ACx in the external storage 2.

[0038] The stored storage dataset ACx contains only chunks that are referenced by files that have been requested to be stored in external storage 2 within a recent fixed period, so the reference count for each chunk is unlikely to reach "0" in a short period of time. For this reason, storage dataset ACx is unlikely to be deleted early after being stored in external storage 2. On the other hand, storage dataset AC1, which contains the original chunks CK1, CK5, and CK7, will continue to be stored in external storage 2 until the reference counts for all chunks reach "0." For this reason, storage dataset AC1 is also unlikely to be deleted early after being stored in external storage 2.

[0039] In this way, the storage control device 1 of this embodiment can prevent data from being deleted from the external storage 2 in a short period of time. Second Embodiment Next, an example will be described in which an archive storage is used as the external storage 2 in FIG. 1 and a cloud storage gateway is used as the storage control device 1 in FIG.

[0040] Fig. 2 is a diagram showing an example of the configuration of an information processing system according to the second embodiment. The information processing system shown in Fig. 2 includes a cloud storage gateway 100, a NAS (Network Attached Storage) client 210, and a cloud storage system 220. The cloud storage gateway 100 is connected to the NAS client 210 via a network 231, and is also connected to the cloud storage system 220 via a network 232. The network 231 is, for example, a LAN (Local Area Network), and the network 232 is, for example, a WAN (Wide Area Network).

[0041] The cloud storage system 220 provides cloud storage services via a network 232. The cloud storage system 220 provides object storage 221 and archive storage 222 as cloud storage areas. In the object storage 221, data is managed on an object-by-object basis. For example, the object storage 221 is realized as a distributed storage system. On the other hand, the archive storage 222 is a storage area suitable for long-term storage of data.

[0042] The cloud storage system 220 accepts a request to store an object in the object storage 221 from a service client (here, the cloud storage gateway 100), and stores the object in the object storage 221. The cloud storage system 220 also accepts a designation of an object to be stored in the archive storage 222 from among the objects stored in the object storage 221, and stores the designated object in the archive storage 222. Data storage in the archive storage 222 is performed in units of archives containing one or more objects. For example, when multiple objects are designated to be stored in the archive storage 222, an archive containing these multiple objects is created, and the archive is stored in the archive storage 222. Note that objects stored in the archive storage 222 may be deleted from the object storage 221.

[0043] For both the object storage 221 and the archive storage 222, a fee is charged when data is stored or read. As mentioned above, the archive storage 222 is suitable for long-term data storage. For this reason, for example, the archive storage 222 has a minimum archive storage period, and an additional fee is charged if an archive is deleted before the minimum storage period has elapsed.

[0044] Although not shown, the cloud storage system 220 includes a server device that executes processes such as accepting access requests to the object storage 221 and the archive storage 222 and controlling access to each storage. These processes may be executed in a distributed manner by multiple server devices.

[0045] The NAS client 210 recognizes the cloud storage gateway 100 as a NAS server that provides a storage area managed by a file system. This storage area is a storage area provided by the cloud storage system 220. The NAS client 210 then requests the cloud storage gateway 100 to read and write data on a file-by-file basis, for example, according to the NFS (Network File System) protocol or the CIFS (Common Internet File System) protocol. In other words, the NAS client 210 can use the storage area of ​​the cloud storage system 220 as a large-capacity virtual network file system thanks to the NAS server function of the cloud storage gateway 100.

[0046] The NAS client 210 executes, for example, backup software for data backup. In this case, the NAS client 210 backs up files stored in the NAS client 210 or files stored in a server (e.g., a business server) connected to the NAS client 210 to a storage area provided by the NAS server.

[0047] The cloud storage gateway 100 is an example of the storage control device 1 shown in Fig. 1. The cloud storage gateway 100 relays data transferred between the NAS client 210 and the cloud storage system 220.

[0048] For example, the cloud storage gateway 100 receives a file write request from the NAS client 210 using its NAS server function, and internally caches the file for which writing is requested. The cloud storage gateway 100 divides the file for which writing is requested into chunk units, and stores the actual data in the chunks (chunk data) in the object storage 221. At this time, a predetermined number of chunk data are grouped to generate an object, and the generated object is transferred to the object storage 221.

[0049] Furthermore, when the cloud storage gateway 100 caches a file from the NAS client 210, it divides the file into chunks and performs "deduplicate" to prevent duplicate storage of chunk data with the same content. Furthermore, chunk data may be stored in a compressed state. For example, cloud storage services may charge according to the amount of data stored. By performing deduplication and data compression, the amount of data stored in the storage area of ​​the cloud storage system 220 can be reduced, thereby suppressing service usage costs.

[0050] Furthermore, the cloud storage gateway 100 accepts a request to store data in the archive storage 222 from the NAS client 210. This storage request is accepted, for example, using a GUI (Graphical User Interface) or an API (Application Programming Interface) provided by the cloud storage gateway 100. This storage request is accepted on a file-by-file basis. When the cloud storage gateway 100 accepts the specification of a file to be stored in the archive storage 222, it identifies an object that includes chunk data not stored in the archive storage 222 from among the chunk data contained in the file. The cloud storage gateway 100 requests the cloud storage system 220 to store the identified object in the archive storage 222. In response to this request, the identified object is read from the object storage 221 and stored in the archive storage 222.

[0051] 3 is a diagram showing an example of the hardware configuration of a cloud storage gateway 100. The cloud storage gateway 100 is realized as a computer as shown in FIG.

[0052] The cloud storage gateway 100 includes a processor 101, a RAM (Random Access Memory) 102, a HDD (Hard Disk Drive) 103, a GPU (Graphics Processing Unit) 104, an input interface (I / F) 105, a reading device 106, and a communication interface (I / F) 107.

[0053] The processor 101 performs overall control of the entire cloud storage gateway 100. The processor 101 is, for example, a central processing unit (CPU), a micro processing unit (MPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a programmable logic device (PLD). The processor 101 may also be a combination of two or more elements of a CPU, an MPU, a DSP, an ASIC, or a PLD.

[0054] The RAM 102 is used as the main storage device of the cloud storage gateway 100. The RAM 102 temporarily stores at least a part of the OS (Operating System) program and application programs to be executed by the processor 101. The RAM 102 also stores various data required for processing by the processor 101.

[0055] The HDD 103 is used as an auxiliary storage device for the cloud storage gateway 100. The HDD 103 stores an OS program, application programs, and various data. Note that other types of non-volatile storage devices, such as a solid state drive (SSD), can also be used as the auxiliary storage device.

[0056] A display device 104a is connected to the GPU 104. The GPU 104 displays an image on the display device 104a in accordance with an instruction from the processor 101. The display device 104a may be a liquid crystal display or an organic EL (Electroluminescence) display.

[0057] An input device 105a is connected to the input interface 105. The input interface 105 transmits a signal output from the input device 105a to the processor 101. The input device 105a includes a keyboard and a pointing device. The pointing device includes a mouse, a touch panel, a tablet, a touch pad, a trackball, and the like.

[0058] A portable recording medium 106a is detachably attached to the reading device 106. The reading device 106 reads data recorded on the portable recording medium 106a and transmits the data to the processor 101. The portable recording medium 106a may be an optical disk, a semiconductor memory, or the like.

[0059] The communication interface 107 transmits and receives data to and from other devices via a network 107a. The above hardware configuration can realize the processing functions of the cloud storage gateway 100. The NAS client 210 can also be realized as a computer having the same hardware configuration as that shown in FIG.

[0060] 4 is a diagram showing an example of the configuration of processing functions of a cloud storage gateway 100. The cloud storage gateway 100 includes a storage unit 110, a file input / output unit 120, a deduplication processing unit 130, and a cloud communication unit 140.

[0061] The storage unit 110 is a storage area secured in a storage device provided in the cloud storage gateway 100, such as the RAM 102 or the HDD 103. The processing of the file input / output unit 120, the deduplication processing unit 130, and the cloud communication unit 140 is realized, for example, by the processor 101 executing a predetermined program.

[0062] The storage unit 110 stores a directory table 111, a chunk map table 112, a chunk management table 113, a chunk group management table 114, and a defragmentation judgment table 115. In addition, a part of the storage area of ​​the storage unit 110 is used as a data cache .

[0063] The directory table 111 is management information for expressing the directory structure in a file system. Records corresponding to directories (folders) in the directory structure or files within a directory are registered in the directory table 111. An inode number for identifying the directory or file is registered in each record. Furthermore, for example, the inode number of the parent directory is registered in each record, thereby expressing the relationship between directories and between directories and files.

[0064] The chunk map table 112 is management information for managing the correspondence between files and deduplicated chunks. The chunk management table 113 is management information for managing the correspondence between chunks and objects and chunk groups, the number of references to chunks, hash keys corresponding to chunks, etc. A chunk group is a group of chunks included in one or more objects (in this embodiment, for example, a predetermined number of objects greater than or equal to two).

[0065] The chunk group management table 114 is management information for managing the number of valid chunks indicating valid chunks and the number of invalid chunks indicating invalid chunks, for each chunk group stored in the archive storage 222. The defragmentation determination table 115 is management information for managing the number of valid chunks indicating valid chunks and the number of invalid chunks indicating invalid chunks, for each archive group. The chunk group management table 114 and defragmentation determination table 115 are used to determine whether or not defragmentation is required for objects stored in the archive storage 222.

[0066] The data cache 116 is a storage area for caching deduplicated chunks. Chunk data corresponding to a file requested to be written by the NAS client 210 is deduplicated and temporarily stored in the data cache 116, and then incorporated into an object and stored in the object storage 221. When the capacity of the data cache 116 decreases due to the storage of chunks, chunks that have already been stored in the object storage 221 and are infrequently accessed by the NAS client 210 are deleted from the data cache 116.

[0067] The file input / output unit 120 executes interface processing as a NAS server. For example, the file input / output unit 120 receives a request from the NAS client 210 to read or write a file or a request to store a file in the archive storage 222, requests the deduplication processor 130 to perform processing according to the request, and responds to the NAS client 210.

[0068] The cloud communication unit 140 executes communication processing with the cloud storage system 220 in response to a request from the deduplication processor 130. For example, the deduplication processor 130 executes uploading of objects to the object storage 221 and downloading of objects from the object storage 221. The cloud communication unit 140 also transmits to the cloud storage system 220 a request to store an object in the archive storage 222 and a request to retrieve an archive from the archive storage 222.

[0069] The deduplication processor 130 executes processing for storing actual data of a file in a state in which duplicates have been eliminated. The deduplication processor 130 includes a duplication determination module 131, a chunk management module 132, and a defragmentation module 133.

[0070] The duplication determination unit 131 divides the actual data of the file for which writing is requested into chunk units, and stores the divided actual data in the data cache 116 while eliminating duplicates. The chunk management unit 132 generates an object by grouping together chunks stored in the data cache 116 by the duplication determination unit 131 to an appropriate size. The chunk management unit 132 stores the generated object in the object storage 221 via the cloud communication unit 140. In this embodiment, as an example, an object is generated from a predetermined number of chunks.

[0071] The defragmentation processing unit 133 monitors chunks (invalid chunks) that are no longer referenced due to a request to delete a file stored in the archive storage 222, and based on the monitoring results, defragments the objects stored in the archive storage 222. In this embodiment, defragmentation is a process for deleting invalid chunks that have occurred from the archive storage 222, thereby reducing the used capacity of the archive storage 222.

[0072] Next, the management information used in the deduplication process will be explained with reference to FIGS. Fig. 5 is a diagram showing an example of the data configuration of a chunk map table. The chunk map table 112 is management information for managing the correspondence between files and deduplicated chunks. As shown in Fig. 5, records having the following items are registered in the chunk map table 112: file number, offset, size, and chunk number. Each record is associated with one chunk generated by dividing the actual data of the file.

[0073] The file number indicates the identification number of the file. The offset indicates the offset amount from the beginning of the file to the beginning of the chunk. The size indicates the size of the chunk. The offset and size values ​​identify the area of ​​the chunk in the file.

[0074] The chunk number indicates the identification number of a deduplicated chunk corresponding to a chunk in a file. If the data content is the same between a first chunk in a file and a second chunk in the same file or another file, the same chunk number is registered in the record corresponding to the first chunk and the record corresponding to the second chunk. For example, in FIG. 5, the same chunk number "ck1" is registered in the record of the chunk identified by file number "f1" and offset "o1" and the record of the chunk identified by file number "f2" and offset "o14." This indicates that the data content is the same between the former chunk and the latter chunk, and that this data is stored in the data cache 116 or the cloud storage system 220 as the chunk with chunk number "ck1."

[0075] Fig. 6 is a diagram showing an example of the data configuration of a chunk management table. The chunk management table 113 is management information for managing the correspondence between deduplicated chunks and objects and chunk groups, the number of references to chunks, and the hash keys corresponding to chunks. As shown in Fig. 6, the chunk management table 113 registers records having the following fields: chunk number, object number, offset, size, chunk group number, number of references (1), hash key, number of references (2), archive number, and archive storage date and time. Each record is associated with one deduplicated chunk.

[0076] The object number indicates the identification number of the object to which the chunk belongs. The offset indicates the offset amount from the beginning of the object to the beginning of the chunk. The size indicates the size of the chunk. The offset and size values ​​identify the area of ​​the chunk in the object.

[0077] The chunk group number indicates the identification number of the chunk group to which the chunk belongs. A chunk group includes chunks contained in a predetermined number of objects. In this embodiment, the chunk group is the unit of data transfer between the cloud storage system 220.

[0078] The reference count (1) indicates how many of the chunks generated by dividing the file reference the chunk indicated by the chunk number. In other words, the reference count (1) indicates how many chunks in the file overlap the chunk indicated by the chunk number. For example, if the reference count (1) corresponding to a certain chunk number is "2," then there will be two records in the chunk map table 112 that have the same chunk number value registered.

[0079] In the example of Fig. 5, the file with file number "f1" is divided into two chunks, and the file with file number "f2" is divided into four chunks. In the example of Fig. 6, the data of the two chunks contained in the former file and the data of the first two chunks of the chunks contained in the latter file are stored in the cloud storage system 220 as chunks belonging to the object with object number "ob1".

[0080] The hash key is a hash value calculated based on the chunk data, and is used to search for a chunk that has the same data as the chunk in the file that is being requested to be written.

[0081] 7 is a diagram showing an example of object generation, and a method for generating an object will be described with reference to this FIG. Table 113a shown in Fig. 7 is obtained by extracting from chunk management table 113 the chunk number and object number items of records corresponding to chunks belonging to the object with object number "ob11". Similarly, table 113b shown in Fig. 7 is obtained by extracting from chunk management table 113 the chunk number and object number items of records corresponding to chunks belonging to the object with object number "ob12". Furthermore, table 113c shown in Fig. 7 is obtained by extracting from chunk management table 113 the chunk number and object number items of records corresponding to chunks belonging to the object with object number "ob13".

[0082] When a request to write a new file or update an existing file is made by the NAS client 210, the duplication determination unit 131 divides the actual data of the file into chunk units. In the example of Fig. 7, the actual data of file f11 is divided into eight chunks, and the actual data of file f12 is divided into five chunks.

[0083] For simplicity's sake, it is assumed here that the data in these chunks are all different (non-overlapping). For this reason, the chunk data obtained by dividing file f11 is assigned individual chunk numbers "ck11" to "ck18," and the chunk data obtained by dividing file f12 is assigned individual chunk numbers "ck19" to "ck23." The chunk data (chunks ck11 to ck23) corresponding to chunk numbers "ck11" to "ck23," respectively, are stored individually in the data cache 116.

[0084] Each chunk is assigned an object number for the object, and the object number is registered in the chunk management table 113. In this embodiment, when the number of chunks assigned to the same object number reaches a predetermined number, the object number is counted up, and the next chunk is assigned the object number after the count up. In this way, a predetermined number of chunks are assigned to the same object.

[0085] The state of an object whose number of chunks has not yet reached a predetermined number will be called "active" and ready to accept the next chunk. An active object is an incomplete object that is not ready to be stored in the object storage 221. The state of an object whose number of chunks has reached a predetermined number will be called "inactive" and not ready to accept the next chunk. An inactive object becomes an object that is ready to be stored in the object storage 221, and is transferred to the object storage 221 at a predetermined time. In practice, when a predetermined number of inactive objects have occurred, these objects are transferred together as a chunk group to the object storage 221.

[0086] In the example of Figure 7, chunks ck11 to ck15 are first assigned to the object (object ob11) with object number "ob11." At this stage, the number of chunks included in object ob11 reaches a predetermined number (five in the example of Figure 7), and object ob11 becomes inactive. Then, a new object number "ob12" is assigned to the next chunk ck16.

[0087] After this, chunks ck16 to ck20 are assigned to the object (object ob12) with object number "ob12", and at this stage object ob12 becomes inactive. Then, the next chunk ck21 is assigned a new object number "ob13". In the example of FIG. 7, chunks ck21 to ck23 are assigned to the object (object ob13) with object number "ob13", but at this stage object ob13 is active. In this case, the next chunk (not shown) to be created will be assigned object number "ob13".

[0088] By following the above procedure, when an object is created when a file is written, a new object is created each time a predetermined number of chunks are newly created through deduplication. Object numbers are assigned to the objects in the order in which they are created. Chunks with consecutive chunk numbers are assigned to each object.

[0089] 7 above, a case where there is no data duplication has been described. For example, if a chunk containing data with the same content as any of chunks ck11 to ck23 exists among the chunks in a file for which writing is subsequently requested, the data of that chunk will not be newly stored in the data cache 116, nor will it be transferred to the cloud storage system 220. In other words, no actual data will be written to this chunk, and only metadata for associating the file with the stored chunks will be written to the chunk map table 112. In this way, a "deduplication process" is performed to prevent duplicate data from being stored.

[0090] In this embodiment, an object is deactivated when a predetermined number of chunks are allocated to the object. However, as another method, for example, an object may be deactivated when the total size of the chunks allocated to the object exceeds a predetermined size.

[0091] The description will be continued below with reference to FIG. The reference count (2) indicates how many chunks, among the chunks generated by dividing a file stored in archive storage 222, reference the chunk indicated by the chunk number. In other words, the reference count (2) indicates the number of references from files stored in archive storage 222, out of the reference count (1).

[0092] The archive number indicates the identification number of the archive that the chunk is included in. The archive number is registered when the object that includes the chunk is stored in the archive storage 222.

[0093] The archive storage date and time indicates the date and time when a file referencing a chunk was stored in archive storage 222. A chunk can be referenced by multiple files, but the archive storage date and time registered is the date and time when the file referencing the chunk was most recently stored in archive storage 222. Therefore, when the reference count (2) is incremented, the archive storage date and time is also updated.

[0094] Next, the management information used to determine whether or not defragmentation is required will be described with reference to FIGS. 8 is a diagram showing an example of the data configuration of the chunk group management table. In the chunk group management table 114, a record corresponding to that chunk group is created each time the chunk group is stored in the archive storage 222. In each record, the chunk group number, the number of valid chunks, and the number of invalid chunks are registered.

[0095] The chunk group number indicates the identification number of the chunk group. The number of valid chunks indicates the number of chunks included in the chunk group whose reference count (2) is 1 or more. The number of invalid chunks indicates the number of chunks included in the chunk group whose reference count (2) is 0.

[0096] Fig. 9 is a diagram showing an example of the data configuration of a defragmentation decision table. In the defragmentation decision table 115, a record is created for each archive group that includes a predetermined number of archives. An archive group contains a predetermined number of archives in the order in which the archives were created. In the example of Fig. 9, the archive group contains 10 archives.

[0097] Each record in the defragmentation decision table 115 registers an archive number, the number of valid chunks, and the number of invalid chunks. The archive number indicates the identification number of the archive included in the archive group. The number of valid chunks indicates the number of chunks included in each archive in the archive group whose reference count (2) is 1 or more. The number of invalid chunks indicates the number of chunks included in each archive in the archive group whose reference count (2) is 0.

[0098] Next, an example of data storage processing in the object storage 221 and archive storage 222 will be described with reference to FIGS. 10 is a first diagram showing an example of data storage processing in object storage and archive storage. In the example of Fig. 10, it is assumed that the NAS client 210 requests the cloud storage gateway 100 to write files F1 and F2 in that order.

[0099] Let's say that file F1 is split into chunks A to D using variable-length chunking, and that file F1 is split into chunks A, E, C, and F. Here, it is assumed that the data in chunks that are assigned the same alphabetical character has the same content. For example, chunk A split from files F1 and F2 both has the same data content. In other words, the data in chunk A is duplicated between files F1 and F2.

[0100] When a request is made to write such files F1 and F2, deduplication processing causes chunks A to F to be stored one by one in the data cache 116. Furthermore, the reference counts (here, reference count (1)) corresponding to chunks A to F are "2", "1", "2", "1", "1", and "1", respectively.

[0101] 10, for example, three chunks are assigned to an object. In this case, object O1 is generated from chunks A to C, and object O2 is generated from chunks D to F. Then, objects O1 and O2 are transferred from the cloud storage gateway 100 to the object storage 221 by a PUT command.

[0102] Further, suppose that the NAS client 210 subsequently requests the cloud storage gateway 100 to store files F1 and F2 in the archive storage 222. In this case, objects O1 and O2 containing chunks referenced by files F1 and F2 are identified. Then, an archive A1 containing these objects O1 and O2 is generated, and archive A1 is stored in the archive storage 222. Furthermore, the reference count (2) of chunks A to F contained in archive A1 is set to the same value as the reference count (1).

[0103] Fig. 11 is a second diagram showing an example of data storage processing in the object storage and archive storage. In the example of Fig. 11, after the processing of Fig. 10, the NAS client 210 requests the cloud storage gateway 100 to write files F3 and F4 in order.

[0104] Assume that file F3 has been divided into chunks A, C, E, G, and F, and file F4 has been divided into chunks A, E, G, H, and I through variable-length chunking. Of these, chunks A, C, and E are already stored in the data cache 116, so only the reference counts (1) for chunks A, C, and E are incremented. That is, the reference counts (1) for chunks A, C, and E are updated to "4," "3," and "3," respectively. Meanwhile, chunks G through I are newly stored in the data cache 116, and the reference counts (1) corresponding to chunks G through I become "2," "1," and "1," respectively. Furthermore, object O3 is generated from chunks G through I, and object O3 is transferred to the object storage 221.

[0105] Thereafter, it is assumed that the NAS client 210 requests the cloud storage gateway 100 to store files F3 and F4 in the archive storage 222. In this case, among the chunks referenced by files F3 and F4, object O3 is identified, which includes chunks G, H, and I that are not stored in the archive storage 222. Then, archive A2 is generated, which includes this object O3, and archive A2 is stored in the archive storage 222. Furthermore, the reference count (2) of chunks G to I included in archive A2 is set to the same value as the reference count (1).

[0106] FIG. 12 is a third diagram illustrating an example of data storage processing in the object storage and archive storage. 10 and 11, it is assumed that the NAS client 210 requests the cloud storage gateway 100 to write up to the m-th file Fm. In addition, it is assumed that chunks A to M are generated accordingly, and multiple objects including chunks A to M are transferred to the object storage 221.

[0107] Furthermore, suppose that the NAS client 210 requests the cloud storage gateway 100 to store all files up to file Fm in the archive storage 222. As a result, archives from the first archive A1 to the nth archive An are stored in the archive storage 222. At this time, the reference counts (1) of chunks A to M are "8", "1", "3", "1", "6", "2", "4", "1", "1", "1", and "1", respectively. Also, suppose that the reference counts (2) of chunks A to M are all the same as the reference count (1).

[0108] Note that, since the data cache 116 has a capacity limit, chunks (objects) that have been transferred to the cloud storage system 220 are deleted from the data cache 116. Here, it is assumed that chunks A to M have been deleted from the data cache 116. However, management information for chunks A to M, such as the reference counts (1) and (2), remains in the memory unit 110 of the cloud storage gateway 100. In addition, of the objects that have been transferred to the object storage 221, objects that have been stored in the archive storage 222 may also be deleted from the object storage 221. Here, it is assumed that all objects generated based on files F1 to Fm have been deleted from the object storage 221.

[0109] Next, a case where defragmentation processing is performed on data stored in archive storage 222 from the state shown in Fig. 12 will be described. First, a comparative example of defragmentation processing will be described using Fig. 13 and Fig. 14, and then the defragmentation processing in the second embodiment will be described using Fig. 15 to Fig. 17.

[0110] Fig. 13 is a first diagram showing a comparative example of defragmentation processing, and Fig. 14 is a second diagram showing a comparative example of defragmentation processing. Services that provide object storage 221 and archive storage 222 generally do not provide a command for deleting only part of the data area within an object. For this reason, in the comparative example of defragmentation processing shown in Figures 13 and 14, the following procedure is executed: an object including invalid chunks is acquired, the invalid chunks are removed, the object is reconstructed, and the reconstructed object is then stored as an archive in archive storage 222. Note that in Figures 13 and 14, only reference count (1) is used as the reference count, and reference count (2) is not used.

[0111] 13, it is assumed that the NAS client 210 requests the cloud storage gateway 100 to delete files F1 and F2 from the archive storage 222. For example, when the period during which files F1 and F2 need to be stored on the user side (NAS client 210 side) has expired, the deletion of files F1 and F2 from the archive storage 222 is requested.

[0112] In this case, the reference count (1) of each chunk included in files F1 and F2 is decremented, and objects containing a certain number of chunks whose reference count (1) has become "0" after the decrement are identified as targets for reconstruction. In the example of FIG. 13, the reference count (1) of chunks B and D becomes "0" and they become targets for reconstruction. Therefore, archive A1 containing objects O1 and O2 which contain chunks B and D is extracted from archive storage 222, and objects O1 and O2 are transferred to cloud storage gateway 100 via object storage 221, for example. In addition, the extracted archive A1 is deleted from archive storage 222.

[0113] The cloud storage gateway 100 excludes chunks B and D with a reference count (1) of "0" from the acquired objects O1 and O2, and reconstructs the objects using the remaining chunks A, C, E, and F. In the example of FIG. 13, object Ox is generated using chunks A and E with a large reference count (1), and object Oy is generated using chunks C and F with a small reference count (1). Then, archive Ax containing object Ox and archive Ay containing object Oy are stored in the archive storage 222. By separately generating an object that aggregates chunks with a large reference count (1) and an object that aggregates chunks with a small reference count (1) in this way, the reference count (1) of the chunks in the latter object becomes "0" early, increasing the possibility that these objects can be deleted all at once.

[0114] Next, as shown in FIG. 14, assume that the NAS client 210 requests the cloud storage gateway 100 to delete files F3 and F4 from the archive storage 222. In this case, the reference count (1) of each chunk included in files F3 and F4 is decremented, and objects containing a certain number of chunks whose reference count (1) has become "0" after the decrement are identified as targets for reconstruction. In the example of FIG. 14, the reference count (1) of chunks C, F, H, and I becomes "0", and they become targets for reconstruction. Therefore, archives A2 and Ay containing objects O3 and Oy, which include chunks C, F, H, and I, are extracted from the archive storage 222, and objects O3 and Oy are transferred to the cloud storage gateway 100, for example, via the object storage 221. In addition, the extracted archives A2 and Ay are deleted from the archive storage 222.

[0115] The cloud storage gateway 100 removes chunks C, F, H, and I whose reference count (1) is "0" from the acquired objects O3 and Oy, and reconstructs object Oz using the remaining chunk G. Then, archive Az containing object Oz is stored in archive storage 222.

[0116] Through the above processing, invalid chunks whose reference count (1) has become "0" can be deleted from archive storage 222, and an archive containing only the remaining valid chunks can be left in archive storage 222. This makes it possible to reduce the amount of space used by archive storage 222.

[0117] However, when a request to delete files F3 and F4 is made in Fig. 14, archive Ay, which was just stored in archive storage 222, has already been deleted. As mentioned above, a minimum storage period for archives is set in archive storage 222, and an additional fee is incurred if an archive is deleted before the minimum storage period has elapsed. In the process of Fig. 14, archive Ay is stored, and then deleted before the minimum storage period has elapsed, resulting in an additional fee.

[0118] As described above, the comparative example of the defragmentation process has a problem in that there is a possibility that archives will be deleted before the shortest storage period set for the archive storage 222 has elapsed, which may increase the fee cost. Therefore, in this embodiment, the object and archive reconstruction procedure is changed so that archives with short storage periods are less likely to be deleted in the defragmentation process.

[0119] An example of defragmentation processing in this embodiment will be described below with reference to Figures 15 to 17. Note that in Figures 15 to 17, for simplicity of explanation, defragmentation targets are selected on an object-by-object basis, but in reality, defragmentation targets are selected on a chunk group-by-chunk group basis, each chunk group including multiple objects.

[0120] Fig. 15 is a first diagram showing an example of defragmentation processing in the second embodiment, and Fig. 16 is a second diagram showing an example of defragmentation processing in the second embodiment. 15, it is assumed that the NAS client 210 requests the cloud storage gateway 100 to delete the files F1 and F2 from the archive storage 222 in the state of FIG.

[0121] In this case, the cloud storage gateway 100 decrements the reference counts (1) and (2) of each chunk contained in files F1 and F2. After the decrement, the cloud storage gateway 100 selects, as a defragmentation target, an object (actually a chunk group) that contains a chunk that satisfies at least one of the following conditions (1) and (2):

[0122] Condition (1): The file is referenced from a file that has been requested to be stored in the archive storage 222 within a recent certain period of time. Condition (2): The number of references (2) exceeds a predetermined threshold.

[0123] The cloud storage gateway 100 extracts chunks that satisfy at least one of the above conditions (1) and (2) from the object selected as the defragmentation target, and generates new chunks containing the same data content as each extracted chunk. At this time, the cloud storage gateway 100 replaces the reference counts (1) and (2) of the original chunks with those of the generated chunks. The reference counts (1) and (2) of the original chunks become "0." The cloud storage gateway 100 generates a new object using the generated new chunks, and requests the cloud storage system 220 to store a new archive containing the generated object in the archive storage 222.

[0124] 15, for example, it is assumed that objects O1 to O3 are identified as objects containing chunks that satisfy at least one of the above conditions (1) and (2). For example, chunk G included in object O3 is referenced by file Fm, which was requested to be stored in archive storage 222 latest, and therefore satisfies condition (1). Furthermore, chunk A included in object O1 and chunk E included in object O2 have large reference counts (2), and therefore satisfy condition (2).

[0125] In this case, the cloud storage gateway 100 retrieves the archive A1 containing the objects O1 and O2 and the archive A2 containing the object O3 from the archive storage 222. However, the cloud storage gateway 100 does not delete these archives A1 and A2 at this point. The cloud storage gateway 100 acquires the objects O1 to O3 contained in the archives A1 and A2, for example, via the object storage 221. As a result, the objects O1 to O3 are selected as targets for defragmentation.

[0126] Next, as shown in FIG. 16, the cloud storage gateway 100 selects chunks that satisfy at least one of conditions (1) and (2) from the acquired objects O1 to O3. In the example of FIG. 16, it is assumed that chunks A, E, and G are selected. The cloud storage gateway 100 generates new chunks A', E', and G' that contain the same data content as the selected chunks A, E, and G, respectively. The cloud storage gateway 100 replaces the reference counts of the original chunks A, E, and G with "6," "5," and "4," respectively, for the generated chunks A', E', and G', and updates the reference counts of the original chunks A, E, and G to "0." This replacement of reference counts is performed for both reference counts (1) and (2).

[0127] Furthermore, the management information indicating the files corresponding to the original chunks A, E, and G is reassigned to the new chunks A', E', and G', respectively. That is, in the chunk map table 112, the chunk number of the original chunk associated with the file number is updated to the chunk number of the new chunk.

[0128] The cloud storage gateway 100 generates a new object Ow using the generated chunks A', E', and G', and requests the cloud storage system 220 to store the generated object Ow in the archive storage 222. As a result, an archive Aw containing the object Ow is generated and stored in the archive storage 222.

[0129] Fig. 17 is a third diagram showing an example of defragmentation processing in the second embodiment. In Fig. 17, it is assumed that the NAS client 210 requests the cloud storage gateway 100 to delete files F3 and F4 from the archive storage 222 in the state shown in Fig. 16.

[0130] In this case, the cloud storage gateway 100 decrements the reference counts (1) and (2) of each chunk contained in files F3 and F4. In the example of FIG. 17, this decrement causes the reference counts (2) of chunks C, F, H, and I to become "0." As a result, the reference counts (2) of all chunks contained in archives A1 and A2 become "0." Therefore, the cloud storage gateway 100 requests the cloud storage system 220 to delete archives A1 and A2. Because a sufficient amount of time has passed since archives A1 and A2 were stored in the archive storage 222, archives A1 and A2 can be deleted from the archive storage 222 without consuming additional space.

[0131] 13, an object with many chunks whose reference count (1) is "0" was simply selected as the target for reconstruction, regardless of when the file referencing the chunk was stored in archive storage 222 or the "number" of references. As a result, after a new archive containing the reconstructed object (archive Ay containing object Oy in FIG. 13) was stored in archive storage 222, the archive was deleted in a short time, as shown in FIG. 14.

[0132] In contrast, the cloud storage gateway 100 of this embodiment does not select as a defragmentation target an object that has many chunks with a reference count (1) of "0". Instead, the cloud storage gateway 100 selects as a defragmentation target an object that includes chunks that satisfy at least one of the above conditions (1) and (2). The cloud storage gateway 100 generates a new chunk that includes data with the same content as the chunk that satisfies at least one of conditions (1) and (2), assigns the reference counts (1) and (2) of the original chunk to the new chunk, and sets the reference counts (1) and (2) of the original chunk to "0". The cloud storage gateway 100 generates a new object using the new chunk, archives the generated object, and stores it in the archive storage 222.

[0133] An archive stored in this manner contains only chunks that satisfy at least one of conditions (1) and (2). A chunk that satisfies condition (1) is referenced by a file that is likely to be requested for deletion some time in the future, so the reference count (2) of this chunk is unlikely to immediately become "0." Furthermore, a chunk that satisfies condition (2) also has a large current reference count (2), so the reference count (2) is unlikely to immediately become "0." For this reason, it is likely that a newly stored archive will take a long time for the reference count (2) of all chunks contained in the archive to become "0," and it is unlikely to become a target for deletion from archive storage 222 in a short period of time. Therefore, additional costs are unlikely to be incurred due to deletion before the minimum storage period has elapsed.

[0134] On the other hand, the original chunks corresponding to the newly generated chunks have their reference counts updated to "0" and are left in archive storage 222. In an archive that contains such original chunks, they continue to be stored in archive storage 222 until the reference counts (2) of all chunks in the archive become "0."

[0135] In this way, the archive is deleted only when the reference count (2) of all chunks in the archive becomes "0." In other words, the process of deleting some chunks in the object contained in the archive and reconstructing the object, as in the comparative example above, is not performed. This makes it less likely that the archive will be deleted before the minimum storage period has elapsed.

[0136] Here, in the processing of this embodiment, the data storage capacity in the archive storage 222 and the number of archive inputs and outputs to and from the archive storage 222 may be greater than in the case of the above comparative example. However, rather than increasing costs due to an increase in storage capacity and the number of inputs and outputs, reducing the number of times additional costs are incurred due to the deletion of archives before the minimum storage period has elapsed has a greater overall cost reduction effect.

[0137] Furthermore, with the processing of this embodiment, as defragmentation progresses, the number of archives that are difficult to delete increases in the archive storage 222. As a result, the number of archive input / output operations between the archive storage 222 decreases over time. Therefore, over a relatively long period of time, the number of archive input / output operations between the archive storage 222 and the archive storage 222 is less than in the comparative example described above, and therefore, it is possible to obtain a cost reduction effect by reducing the number of input / output operations.

[0138] As will be explained in a later flowchart, in actual processing, the chunk group to be defragmented is selected from among chunk groups that contain many chunks with a reference count (2) of "0." This increases the number of chunks with a reference count (2) of "0" in the chunk group selected as the defragmentation target, making it possible to quickly reduce the reference count (2) of all chunks in that chunk group to "0."

[0139] Next, the processing of the cloud storage gateway 100 will be explained using a flowchart. 18 and 19 are examples of flowcharts showing the procedure of file writing processing.

[0140] [Step S11] The file input / output unit 120 receives a file write request and file data from the NAS client 210. The duplication determination unit 131 of the deduplication processor 130 acquires the file data requested to be written and adds a record indicating the directory information of that file to the directory table 111. At this time, a file number is assigned to the file. The duplication determination unit 131 also divides the file data into variable-length chunks.

[0141] [Step S12] The duplication determination unit 131 selects one chunk to be processed, starting from the beginning of the file, and calculates a hash key based on the data of the selected chunk.

[0142] [Step S13] The duplication determination unit 131 adds a record to the chunk map table 112 and registers the following information in this record: The file number field registers the file number of the file requested to be written, and the offset and size fields register information about the chunk to be processed.

[0143] [Step S14] The duplication determination unit 131 refers to the chunk management table 113 and determines whether a record exists in which the hash key calculated in step S13 is registered. This determines whether a chunk with the same content as the chunk selected in step S12 has already been stored (whether there is a duplicate). If a corresponding record is found, the process proceeds to step S15; if a corresponding record does not exist, the process proceeds to step S21 in FIG. 19.

[0144] [Step S15] The duplication determination unit 131 acquires the chunk number from the record searched for in the chunk management table 113 in step S14, and registers the acquired chunk number in the record added to the chunk map table 112 in step S13.

[0145] [Step S16] The duplication determination unit 131 references the record in the chunk management table 113 that includes the chunk number acquired in step S15, and counts up the reference count (1) registered in this record.

[0146] [Step S17] The duplication determination unit 131 determines whether all chunks divided in step S11 have been processed. If there are unprocessed chunks, the process proceeds to step S12, where one unprocessed chunk is selected from the beginning and the process continues. On the other hand, if all chunks have been processed, the duplication determination unit 131 notifies the file input / output unit 120 that the file writing has been completed. Upon receiving the notification, the file input / output unit 120 sends response information indicating the completion of the file writing to the NAS client 210.

[0147] The description will be continued below with reference to FIG. [Step S21] The duplication determination unit 131 calculates a new chunk number for the chunk selected in step S12. This chunk number is the maximum chunk number registered in the chunk management table 113 plus "1". The duplication determination unit 131 adds a new record to the chunk management table 113. The duplication determination unit 131 registers the calculated new chunk number, the chunk size, and the hash key calculated in step S12 in the added record, and registers "1" as the reference count (1).

[0148] [Step S22] The duplication determining unit 131 stores the data of the chunk selected in step S12 in the data cache 117. [Step S23] The duplication determination unit 131 registers the new chunk number calculated in step S21 in the record added to the chunk map table 112 in step S13.

[0149] [Step S24] An object number is assigned to the chunk selected in step S12. This object number indicates the object to which the chunk belongs. The object number assigned is the object number of the object currently being created. The chunk management unit 132 registers the assigned object number in the record added to the chunk management table 113 in step S21.

[0150] The chunk management unit 132 also registers, in the same record, a chunk group number indicating the chunk group to which the object currently being created belongs. A chunk group contains a predetermined number of objects in the order in which they were created. For this reason, the chunk group number is incremented each time a predetermined number of objects are created.

[0151] When a new chunk group number is registered, the chunk management unit 132 adds a new record including that chunk group number to the chunk group management table 114. In this record, the initial value "0" is registered as the number of valid chunks and the number of invalid chunks.

[0152] [Step S25] The chunk management unit 132 determines whether the number of chunks included in the object currently being created has reached a predetermined threshold (defined as M). If the number of chunks has reached threshold M, the process proceeds to step S26. On the other hand, if the number of chunks has not reached threshold M, the object currently being created is in an active state and is therefore not uploaded to the object storage 221. In this case, steps S26 and S27 are skipped.

[0153] [Step S26] The object currently being created becomes inactive and contains M chunks, and the chunk management unit 132 requests the cloud communication unit 140 to upload this object to the object storage 221. The cloud communication unit 140 uploads the object to the object storage 221 using a PUT command.

[0154] [Step S27] The chunk management unit 132 increments the object number indicating the object currently being created. As a result, the next time step S24 is executed, the object number of a new object will be assigned to the chunk.

[0155] Thereafter, the process proceeds to step S17 in FIG. Fig. 20 is an example of a flowchart showing the procedure for data storage processing in archive storage. The processing in Fig. 20 starts when file input / output unit 120 receives a request from NAS client 210 to store a file in archive storage 222. This storage request specifies one or more files to be stored.

[0156] [Step S31] The chunk management unit 132 identifies, from among the objects corresponding to chunks included in the file specified as the storage target (objects including chunk data), objects that are not stored in the archive storage 222. In this process, if an archive number is not registered in the chunk management table 113 for a chunk number associated with the specified file, it is determined that the object including the chunk indicated by that chunk number is not stored in the archive storage 222.

[0157] [Step S32] The chunk management unit 132 requests the cloud communication unit 140 to store an archive containing the object identified in step S31 in the archive storage 222. The cloud communication unit 140 specifies the object number of the identified object and requests the cloud storage system 220 to store the object in the archive storage 222. As a result, the archive containing the identified object is stored from the object storage 221 to the archive storage 222.

[0158] [Step S33] The chunk management unit 132 identifies records corresponding to each chunk included in the object identified in step S31 from the chunk management table 113. The chunk management unit 132 registers, for each identified record, an archive number indicating the archive stored in the archive storage 222.

[0159] [Step S34] The chunk management unit 132 executes the following process for each of the objects identified in step S31. The chunk management unit 132 identifies a record corresponding to the chunk group that includes the object in question from the chunk group management table 114. The chunk management unit 132 adds the number of chunks included in the object in question (the above-mentioned M) to the number of valid chunks registered in the identified record.

[0160] [Step S35] The chunk management unit 132 identifies a record corresponding to the archive group to which the archive stored in the archive storage 222 in step S32 belongs, from the defragmentation determination table 115. The chunk management unit 132 adds the number of chunks included in the archive (the total number of chunks added in step S34) to the number of valid chunks registered in the identified record.

[0161] [Step S36] The chunk management unit 132 identifies records in the chunk management table 113 that include chunk numbers associated with each file requested to be stored in the archive storage 222. For each identified record, the chunk management unit 132 registers the current date and time as the archive storage time. If a date and time has already been registered, the date and time is updated to the current date and time.

[0162] [Step S37] The chunk management unit 132 performs the following process for each chunk number associated with each file requested to be stored in the archive storage 222. The chunk management unit 132 identifies a record including the corresponding chunk number from the chunk management table 113, and increments the reference count (2) registered in the identified record.

[0163] Fig. 21 is an example of a flowchart showing the steps of a process for deleting data from archive storage. The process of Fig. 21 starts when the file input / output unit 120 receives a request from the NAS client 210 to delete a file from the archive storage 222. In this deletion request, one or more files are specified as the files to be deleted.

[0164] [Step S41] The chunk management unit 132 refers to the chunk management table 113 and subtracts the reference counts (1) and (2) corresponding to each chunk included in the file specified to be deleted. Specifically, the chunk management unit 132 performs the following process for each chunk number associated with each specified file: The chunk management unit 132 identifies a record including the relevant chunk number from the chunk management table 113, and decrements (subtracts "1") the reference counts (1) and (2) registered in the identified record.

[0165] [Step S42] The chunk management unit 132 determines whether the processing in step S41 has resulted in the generation of a chunk whose reference count (2) is 0. If a corresponding chunk has occurred, the processing proceeds to step S43; if a corresponding chunk has not occurred, the data deletion processing ends.

[0166] [Step S43] The chunk management unit 132 updates the number of valid chunks and the number of invalid chunks registered in the corresponding records in the chunk management table 113 and the defragmentation determination table 115. Specifically, the chunk management unit 132 performs the following process for each chunk whose reference count (2) has become "0."

[0167] The chunk management unit 132 identifies a record corresponding to the chunk group to which the relevant chunk belongs from the chunk management table 113. The chunk management unit 132 decrements the number of valid chunks registered in the identified record and increments the number of invalid chunks. The chunk management unit 132 also identifies a record corresponding to the archive group to which the relevant chunk belongs from the defragmentation determination table 115. The chunk management unit 132 decrements the number of valid chunks registered in the identified record and increments the number of invalid chunks.

[0168] [Step S44] The chunk management unit 132 determines whether there is an archive whose valid chunk count is "0" (an archive whose reference count (2) of all chunks it belongs to is "0"). If there is a matching archive, the process proceeds to step S45; if there is no matching archive, the process proceeds to step S46.

[0169] [Step S45] The chunk management unit 132 requests the cloud communication unit 140 to delete archives whose valid chunk count is "0" from the archive storage 222. The cloud communication unit 140 specifies the archive number of the relevant archive and requests the cloud storage system 220 to delete it from the archive storage 222. As a result, the relevant archive is deleted from the archive storage 222.

[0170] The processing of steps S44 and S45 may be performed on an archive group basis rather than on an archive basis. In this case, in step S44, it is determined whether there is an archive group in which the number of valid chunks is "0" in the defragmentation determination table 115. If such an archive group is found, then in step S45, each archive included in that archive group is deleted from the archive storage 222.

[0171] [Step S46] The chunk management unit 132 references the defragmentation determination table 115 and determines whether there are any archive groups whose percentage of invalid chunks exceeds a predetermined threshold. The percentage of invalid chunks is calculated as the ratio of the number of invalid chunks to the total number of valid chunks and invalid chunks. If a corresponding archive group is found, processing proceeds to step S47; if no corresponding archive group is found, the data deletion processing ends.

[0172] [Step S47] The chunk management section 132 instructs the defragmentation processing section 133 to start defragmentation processing, thereby starting the defragmentation processing shown in FIGS.

[0173] 22 and 23 are examples of flowcharts showing the procedure of the defragmentation process. [Step S51] The defragmentation processing unit 133 waits for a certain period of time to elapse after the instruction to start defragmentation is given in step S47 of Fig. 21. Then, after the certain period of time has elapsed, the defragmentation processing unit 133 executes the processing of step S52. By starting defragmentation a certain period of time after the instruction to start, more invalid chunks with a reference count (2) of "0" are generated, and it is expected that more archives that can be deleted in bulk will be generated.

[0174] [Step S52] The defragmentation processing unit 133 refers to the defragmentation determination table 115 and identifies archive groups whose percentage of invalid chunks exceeds a predetermined threshold. As in step S46 of Fig. 21, the percentage of invalid chunks is calculated as the ratio of the number of invalid chunks to the total number of valid chunks and invalid chunks.

[0175] [Step S53] The defragmentation processor 133 selects one chunk group to be processed from among the chunk groups included in the archive group identified in step S52. [Step S54] The defragmentation processing unit 133 determines whether the selected chunk group includes a chunk that satisfies at least one of the above-mentioned conditions (1) and (2). A chunk satisfies condition (1) if the archive storage time registered in the corresponding record in the chunk management table 113 is within the most recent fixed period. A chunk satisfies condition (2) if the reference count (2) corresponding to the chunk exceeds a predetermined threshold. The threshold for condition (2) is set, for example, to the median (the N / 2th value in ascending order) based on the reference count (2) of all chunks (assuming that there are N chunks) whose reference count (2) is 2 or greater.

[0176] If the selected chunk group includes one or more applicable chunks, the process proceeds to step S55; if the selected chunk group does not include any applicable chunks, the process proceeds to step S56.

[0177] [Step S55] The defragmentation processor 133 designates the selected chunk group as a target for defragmentation. [Step S56] The defragmentation processing unit 133 determines whether all of the chunk groups included in the archive group identified in step S52 have been selected as processing targets. If there are unselected chunk groups, the process proceeds to step S53, where one unselected chunk group is selected. On the other hand, if all chunk groups have been selected, the process proceeds to step S57.

[0178] [Step S57] The defragmentation processing unit 133 requests the cloud communication unit 140 to download the archive containing the chunk group designated as the defragmentation target from the archive storage 222. The cloud communication unit 140 specifies the archive number of the relevant archive and requests the cloud storage system 220 to retrieve it from the archive storage 222. As a result, the relevant archive is downloaded to the cloud storage gateway 100.

[0179] [Step S58] The defragmentation processing unit 133 extracts chunks that satisfy at least one of conditions (1) and (2) from the chunks included in the downloaded archive. The defragmentation processing unit 133 sorts the extracted chunks using the archive storage date and time in the chunk management table 113 as a first key and the reference count (2) in the chunk management table 113 as a second key. That is, the extracted chunks are sorted in descending order of the most recent archive storage date and time, and chunks with the same archive storage date and time are sorted in descending order of the reference count (2).

[0180] The description will be continued below with reference to FIG. [Step S61] The defragmentation unit 133 selects one chunk from the beginning of the chunks sorted in step S58 as a processing target.

[0181] [Step S62] The defragmentation processing unit 133 determines whether either of the following conditions is met: there is no existing chunk group being created during defragmentation processing, or there is an existing chunk group being created but the size of that chunk group exceeds a specified size. If either of these conditions is met, processing proceeds to step S63; if neither condition is met, processing proceeds to step S64.

[0182] [Step S63] The defragmentation processor 133 creates a new chunk group. A new chunk group number is assigned to the created chunk group. In addition, a record corresponding to the new chunk group is added to the chunk group management table 114, and the chunk group number is registered.

[0183] [Step S64] The defragmentation processor 133 adds a new chunk containing the same data content as the selected chunk to the chunk group. If step S63 is executed, the new chunk is added to the chunk group newly created in step S63. On the other hand, if the determination in step S62 is No, the new chunk is added to the existing chunk group being created.

[0184] In this process, the defragmentation processor 133 assigns a new chunk number to the new chunk, adds a record to the chunk management table 113, and registers the chunk number in that record. The defragmentation processor 133 also registers the chunk group number of the chunk group to which the chunk is to be added in the added record, and registers a new object number that corresponds one-to-one with the chunk group number as the object number. Note that values ​​indicating the starting position of the chunk in the chunk group are registered as the offset and size.

[0185] [Step S65] The defragmentation processing unit 133 changes the reference counts (1) and (2) from the original chunk (the chunk selected in step S61) to the new chunk. Specifically, the defragmentation processing unit 133 copies the values ​​of the reference counts (1) and (2) corresponding to the original chunk to the reference counts (1) and (2) corresponding to the new chunk in the chunk management table 113, respectively, and updates the values ​​of the reference counts (1) and (2) corresponding to the original chunk to "0."

[0186] The defragmentation processor 133 also copies the value of the hash key corresponding to the original chunk to the hash key item corresponding to the new chunk in the chunk management table 113. Furthermore, the defragmentation processor 133 updates all of the chunk numbers of the original chunks registered in the chunk map table 112 to the chunk numbers of the new chunks. As a result, the management information indicating the correspondence between chunks and files is updated, and the original chunks are no longer referenced by the files.

[0187] [Step S66] The defragmentation processing unit 133 determines whether all of the chunks sorted in step S58 have been selected for processing. If there are any unselected chunks, the process proceeds to step S61, where the first chunk among the unselected chunks is selected. On the other hand, if all applicable chunks have been selected, the process proceeds to step S67.

[0188] [Step S67] The defragmentation processing unit 133 requests the cloud communication unit 140 to store an archive including the chunk groups created in the processes of steps S61 to S66 in the archive storage 222. The cloud communication unit 140 uploads each of the created chunk groups as an individual object to the object storage 221. Thereafter, the cloud communication unit 140 requests that an archive including each uploaded object be stored in the archive storage 222. As a result, the relevant objects are stored in the archive storage 222.

[0189] [Step S68] The defragmentation processing unit 133 registers the archive numbers of all chunks included in the archive stored in the archive storage 222, among the records in the chunk management table 113. The defragmentation processing unit 133 also registers the current date and time in the archive storage date and time field of those records.

[0190] Furthermore, the defragmentation processing unit 133 adds the number of chunks included in the chunk group to the number of valid chunks in the record corresponding to each chunk group included in the archive among the records in the chunk group management table 114. Also, the defragmentation processing unit 133 adds the number of chunks included in the archive to the number of valid chunks associated with the archive number of the archive in the defragmentation determination table 115.

[0191] The defragmentation processing unit 133 also executes the following process for each of the original chunks whose reference counts (1) and (2) have been updated to "0" in step S65. The defragmentation processing unit 133 decrements the number of valid chunks and increments the number of invalid chunks in the record in the chunk group management table 114 that corresponds to the chunk group to which the chunk belongs. Furthermore, the defragmentation processing unit 133 decrements the number of valid chunks and increments the number of invalid chunks in the record in the defragmentation determination table 115 that corresponds to the archive to which the chunk belongs.

[0192] [Step S69] The chunk management unit 132 determines whether there is an archive whose valid chunk count is "0" (an archive whose reference count (2) of all chunks it belongs to is "0"). If there is a matching archive, the process proceeds to step S70; if there is no matching archive, the defragmentation process ends.

[0193] [Step S70] The chunk management unit 132 requests the cloud communication unit 140 to delete archives whose valid chunk count is "0" from the archive storage 222. The cloud communication unit 140 specifies the archive number of the relevant archive and requests the cloud storage system 220 to delete it from the archive storage 222. As a result, the relevant archive is deleted from the archive storage 222.

[0194] As with steps S44 and S45, the processing of steps S69 and S70 may be executed in units of archive groups, rather than in units of archives. According to the above process, during the defragmentation process, the cloud storage gateway 100 stores archives that are unlikely to be deleted early in the archive storage 222. This reduces the probability of an archive being deleted from the archive storage 222 before the specified minimum storage period has elapsed. As a result, it is possible to increase the possibility of reducing the usage costs of the archive storage 222.

[0195] 22, defragmentation targets may be specified on an object-by-object basis rather than on a chunk group basis. However, since a large number of objects are created in actual processing, if defragmentation targets are specified on an object-by-object basis, defragmentation processing using small defragmentation target data may be executed frequently, in which case processing efficiency will be low. In contrast, processing efficiency can be improved by specifying defragmentation targets on a chunk group basis that includes multiple objects.

[0196] Similarly, in step S46 of Fig. 21 and step S52 of Fig. 22, the percentage of invalid chunks may be calculated on an archive basis rather than on an archive group basis. However, calculating the percentage of invalid chunks on an archive group basis can improve processing efficiency.

[0197] The processing functions of the devices (e.g., the storage control device 1, the cloud storage gateway 100) shown in each of the above embodiments can be realized by a computer. In this case, a program describing the processing contents of the functions that each device should have is provided, and the processing functions are realized on the computer by executing the program on the computer. The program describing the processing contents can be recorded on a computer-readable recording medium. Examples of computer-readable recording media include magnetic storage devices, optical discs, and semiconductor memories. Examples of magnetic storage devices include hard disk drives (HDDs) and magnetic tapes. Examples of optical discs include CDs (Compact Discs), DVDs (Digital Versatile Discs), and Blu-ray Discs (BD, registered trademark).

[0198] When distributing a program, for example, the program is recorded on a portable recording medium such as a DVD or CD and sold. Alternatively, the program can be stored in a storage device of a server computer and transferred from the server computer to other computers via a network.

[0199] A computer that executes a program stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. The computer then reads the program from its own storage device and executes processing in accordance with the program. Note that the computer can also read the program directly from a portable recording medium and execute processing in accordance with that program. The computer can also execute processing in accordance with the program received each time a program is transferred from a server computer connected via a network. [Explanation of symbols]

[0200] 1 Storage control device 2. External Storage AC1,AC2,ACn,ACx Stored Data Set CK1~CK9,CK21,CK22,CK1',CK5',CK7' chunks FL1~FL4,FLm files

Claims

1. On the computer, Obtaining a plurality of chunks from a plurality of divided data sets obtained by dividing each of the plurality of files for which writing is requested, while eliminating duplicates, and generating a plurality of storage data sets by combining two or more chunks selected from the plurality of chunks, when a storage request for a first file of the plurality of files is received, one or more first storage datasets including chunks of the plurality of chunks referenced by the first file are identified, and the one or more first storage datasets are stored in an external storage, and a reference count associated with the chunks of the plurality of chunks referenced by the first file is increased by the number of references from split datasets included in the first file among the plurality of split datasets; When a request to delete a second file among the plurality of files is received, the reference count corresponding to a chunk among the plurality of chunks referenced by the second file is decreased by the number of chunks referenced by a split data set included in the second file among the plurality of split data sets; Among the plurality of stored data sets stored in the external storage, one or more first chunks that are referenced by files among the plurality of files for which storage in the external storage has been requested within a recent fixed period are identified and acquired from the external storage; generating one or more second chunks each having the same data as the one or more first chunks, respectively reassigning the reference numbers corresponding to the one or more first chunks to the one or more second chunks, respectively, and updating the reference numbers corresponding to the one or more first chunks to 0, and aggregating the one or more second chunks to generate a new storage dataset and storing the new dataset in the external storage; When a second storage dataset in which the reference count corresponding to all chunks in a corresponding storage dataset is 0 is detected from among the storage datasets stored in the external storage among the plurality of storage datasets, the second storage dataset is deleted from the external storage. Storage control program.

2. As the one or more first chunks, chunks whose reference count is greater than a predetermined threshold are further identified from among the plurality of stored datasets stored in the external storage. The storage control program according to claim 1.

3. In identifying the one or more first chunks, Identifying one or more third storage datasets from among the plurality of storage datasets stored in the external storage, in which the proportion of chunks with the reference count of 0 exceeds a predetermined proportion; identifying the one or more first chunks from within the one or more third archived datasets; 3. The storage control program according to claim 1.

4. The computer Obtaining a plurality of chunks from a plurality of divided data sets obtained by dividing each of the plurality of files for which writing is requested, while eliminating duplicates, and generating a plurality of storage data sets by combining two or more chunks selected from the plurality of chunks, when a storage request for a first file of the plurality of files is received, one or more first storage datasets including chunks of the plurality of chunks referenced by the first file are identified, and the one or more first storage datasets are stored in an external storage, and a reference count associated with the chunks of the plurality of chunks referenced by the first file is increased by the number of references from split datasets included in the first file among the plurality of split datasets; When a request to delete a second file among the plurality of files is received, the reference count corresponding to a chunk among the plurality of chunks referenced by the second file is decreased by the number of chunks referenced by a split data set included in the second file among the plurality of split data sets; Among the plurality of stored data sets stored in the external storage, one or more first chunks that are referenced by files among the plurality of files for which storage in the external storage has been requested within a recent fixed period are identified and acquired from the external storage; generating one or more second chunks each having the same data as the one or more first chunks, respectively reassigning the reference numbers corresponding to the one or more first chunks to the one or more second chunks, respectively, and updating the reference numbers corresponding to the one or more first chunks to 0, and aggregating the one or more second chunks to generate a new storage dataset and storing the new dataset in the external storage; When a second storage dataset in which the reference count corresponding to all chunks in a corresponding storage dataset is 0 is detected from among the storage datasets stored in the external storage among the plurality of storage datasets, the second storage dataset is deleted from the external storage. Storage control methods.

5. Obtaining a plurality of chunks from a plurality of divided data sets obtained by dividing each of the plurality of files for which writing is requested, while eliminating duplicates, and generating a plurality of storage data sets by combining two or more chunks selected from the plurality of chunks, when a storage request for a first file of the plurality of files is received, one or more first storage datasets including chunks of the plurality of chunks referenced by the first file are identified, and the one or more first storage datasets are stored in an external storage, and a reference count associated with the chunks of the plurality of chunks referenced by the first file is increased by the number of references from split datasets included in the first file among the plurality of split datasets; When a request to delete a second file among the plurality of files is received, the reference count corresponding to a chunk among the plurality of chunks referenced by the second file is decreased by the number of chunks referenced by a split data set included in the second file among the plurality of split data sets; Among the plurality of stored data sets stored in the external storage, one or more first chunks that are referenced by files among the plurality of files for which storage in the external storage has been requested within a recent fixed period are identified and acquired from the external storage; generating one or more second chunks each having the same data as the one or more first chunks, respectively reassigning the reference numbers corresponding to the one or more first chunks to the one or more second chunks, respectively, and updating the reference numbers corresponding to the one or more first chunks to 0, and aggregating the one or more second chunks to generate a new storage dataset and storing the new dataset in the external storage; a processing unit that, when a second storage dataset in which the reference count corresponding to all chunks in a corresponding storage dataset is 0 is detected from among the storage datasets stored in the external storage among the plurality of storage datasets, deletes the second storage dataset from the external storage; A storage control device having:

Images