File transfer system, file transfer method, file transfer program

Abstract

To efficiently transfer a file.SOLUTION: A file update system transfers an object file to be updated in a first computer to a second computer by every part obtained by dividing the object file, and comprises: an update recording unit which records an update position of the object file in the first computer as an offset flag; an update determination unit which determines presence / absence of update by every part in the object file by referring to the offset flag; and a transfer unit which transfers a part which is determined that there is the update by the update determination unit to the second computer, wherein the transfer unit transmits a re-update part updated by re-update to the second computer regardless of whether the part is already transferred when the re-update in which the object file is updated after the transfer unit starts transfer of any part included in the object file occurs.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a file transfer system, a file transfer method, and a file transfer program. [Background technology]

[0002]

[0003] Patent Literature 1 discloses a file storage system that includes a first file system provided to an application, a first storage system in which files are stored by the first file system, and a processor, and that can use a second storage system, the file storage system including: state management information that stores states of the files; a state information management unit that manages the state management information; and a file virtualization unit that manages files stored in the first storage system and the second storage system, the processor calling the first file system based on a file operation request from the application, the first file system processing the file operation request, the state information management unit updating the file state management information based on input information or operation content to the first file system related to the operation request, and the file virtualization unit managing the file between the first storage system and the second storage system based on the state management information. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-157381 Summary of the Invention [Problem to be solved by the invention]

[0004] The invention described in Patent Document 1 leaves room for improvement in the file transfer method. [Means for solving the problem]

[0005] A file updating system according to a first aspect of the present invention is a file updating system that transfers a target file to be updated in a first computer to a second computer in parts obtained by dividing the target file, wherein the part is a unit of data transfer, and the target file before the update has been transferred to the second computer in advance, and the first computer comprises: an update recording unit that records an update position of the target file in the first computer as an offset flag; an update determination unit that references the offset flag and determines whether or not each part of the target file has been updated; and a transfer unit that transfers the part that the update determination unit determines to have been updated to the second computer, and wherein when a re-update of the target file occurs after the transfer unit has started to transfer any of the parts included in the target file, the transfer unit: Until the re-update no longer occurs, The re-updated part, which is the part updated by the re-updating, is transmitted to the second computer regardless of whether it has already been transferred or not. A file updating method according to a second aspect of the present invention is a file updating method in which a target file to be updated in a first computer is transferred to a second computer by dividing the target file into parts, the part being a unit of data transfer, and the target file before the update has been transferred to the second computer in advance, the method comprising the steps of: an update recording process in which the first computer records an update position of the target file in the first computer as an offset flag; an update determination process in which the first computer references the offset flag and determines whether or not each part in the target file has been updated; and a transfer process in which the part determined to have been updated by the update determination process is transferred to the second computer, and in the transfer process, if a re-update of the target file occurs after the transfer of any of the parts included in the target file has started by the transfer process, Until the re-update no longer occurs,The re-updated part, which is the part updated by the re-updating, is transmitted to the second computer regardless of whether it has already been transferred or not. A file transfer program according to a third aspect of the present invention is a file transfer program for causing a first computer to transfer a target file to be updated on the first computer to a second computer in parts into which the target file is divided, wherein the part is a unit of data transfer, and the target file before the update has been transferred to the second computer in advance, and causes the first computer to function as an update recording unit that records an update position of the target file on the first computer as an offset flag, an update determination unit that references the offset flag and determines whether or not each part of the target file has been updated, and a transfer unit that transfers the part determined by the update determination unit to have been updated to the second computer, and wherein when a re-update of the target file occurs after the transfer unit has started to transfer any of the parts included in the target file, the transfer unit: Until the re-update no longer occurs, The re-updated part, which is the part updated by the re-updating, is transmitted to the second computer regardless of whether it has already been transferred or not. [Effects of the Invention]

[0006] According to the present invention, files can be transferred efficiently. [Brief explanation of the drawings]

[0007] [Figure 1] File transfer system configuration diagram [Figure 2] Computer configuration diagram [Figure 3] Object storage configuration diagram [Figure 4] Conceptual diagram explaining status flags [Figure 5] Diagram explaining flag data [Figure 6] Event and status flag transition diagram for parts and offsets [Figure 7] File status flag transition diagram [Figure 8]Part and offset status flag transition diagram [Figure 9] Transition diagram of offset and part status flags when the MPU transfer mode is other than "Complete" [Figure 10] Schematic showing the MPU for user files [Figure 11] MPU operation concept diagram [Figure 12] MPU operation concept diagram [Figure 13] MPU operation concept diagram [Figure 14] MPU operation concept diagram [Figure 15] MPU operation concept diagram [Figure 16] Flowchart showing replicator processing [Figure 17] Flowchart showing replicator processing [Figure 18] A flowchart showing the part transfer process shown in step S309 of FIG. 17. [Figure 19] 17. A flowchart showing the part duplication process shown in step S308 of FIG. [Figure 20] 17. Flowchart showing the connection confirmation process shown in step S312 of FIG. [Figure 21] A flowchart showing the retry process shown in step S315 of FIG. 17. [Figure 22] 17. Flowchart showing the differential transfer process shown in step S316 of FIG. [Figure 23] A flowchart showing the completion process shown in step S317 of FIG. [Figure 24] A flowchart showing the subsequent processing shown in step S313 of FIG. 17. [Figure 25] A flowchart showing the subsequent processing shown in step S313 of FIG. 17. [Figure 26] A flowchart showing the subsequent processing shown in step S313 of FIG. 17. [Figure 27] Figure showing the user interface for setting the transfer mode DETAILED DESCRIPTION OF THE INVENTION

[0008] -First embodiment- A first embodiment of a file transfer system will be described below with reference to FIGS.

[0009] FIG. 1 is a configuration diagram of a file transfer system 100. The file transfer system includes one data center 103 and multiple bases 101. The data center 103 and the bases 101 are connected by a wide area network 102. Each base 101 has one or more computers 111. The data center 103 has an object storage 130. In FIG. 1, each computer 111 is distinguished by being assigned a branch number, but since the computers 111 have a common functional configuration, the configuration and operation of one computer 111 will be described below. In this embodiment, the transfer of a file from the computer 111 to the object storage 130 will be described.

[0010] However, each computer 111 may additionally have configurations and functions other than those described below, and the additional configurations and functions that each computer 111 has may differ from one another. Also, the file transfer system does not need to include multiple computers 111, and the computers 111 and object storage 130 may be locally connected.

[0011] 2 is a configuration diagram of the computer 111. The computer 111 includes a processing unit 201 and a peripheral device 202. The processing unit 201 includes a main memory device 211 and a CPU 212. The peripheral device 202 includes an auxiliary memory device 220 and a network card 214. The main memory device 211, the CPU 212, the auxiliary memory device 213, and the network interface card 214 are connected via a bus.

[0012] The main memory device 211 stores an exclusive management program 232, a hash calculation program 233, an IO hook program 234, and a replicator 236. The exclusive management program 232 restricts writing to the user file 221 based on an operation instruction received from the replicator 236. The hash calculation program 233 calculates a hash value for the entire user file 221 or for a part of the user file 221. Various known methods can be used to calculate the hash value, such as SHA-256 or MD5.

[0013] The IO hook program 234 detects a write process to the user file 221 and writes the detected write data to the file flag 223 and offset flag 225 of the flag data 222. However, the content that the IO hook program 234 writes to the offset flag 225 changes according to instructions from the replicator 236. Hereinafter, the IO hook program 234 is also referred to as an "update recording unit."

[0014] The replicator 236 transmits the user file 221 stored in the computer 111 to the object storage 130. In this embodiment, "transmission" and "transfer" are used interchangeably. However, the replicator 236 does not necessarily transmit the entire user file 221, but divides the user file 221 and transmits only the necessary parts. Hereinafter, the transfer of the user file 221 by the replicator 236 is also referred to as MPU (Multi Part Upload). There are multiple transfer modes in the MPU, and the transfer mode to be used is set in advance. The replicator 236 uses the exclusive management program 232, hash processing program 233, and IO hook program 234 as necessary. The replicator 236 also writes and reads flag data 222. The detailed operation of the replicator 236 will be described later.

[0015] The auxiliary storage device 213 stores a user file 221 and flag data 222. The user file 221 is a file to be transferred to the object storage 130. There may be multiple user files 221, but the following mainly describes a case where one user file 221 is the target of transfer. The flag data 222 is made up of a file flag 223, a part flag 224, and an offset flag 225. The flag data 222 is written by the IO hook program 234 and the replicator 236. The flag data 222 will be described in detail later.

[0016] The computer 111 may further include a storage medium interface (not shown), read a program from a storage medium 99-1, and store the program in a ROM (not shown) and the main storage device 211. The computer 111 may also receive a program as a data signal from the outside via a carrier wave 98-1, and store the received program in a ROM (not shown) and the main storage device 211.

[0017] 3 is a configuration diagram of the object storage 130. The object storage 130 includes a processing device 301 and a peripheral device 302. The processing device 301 includes a main memory device 311 and a CPU 312. The main memory device 311 stores an object system program 331 and a hash calculation program 332. The peripheral device 302 includes an auxiliary memory device 313 and a network card 314. The auxiliary memory device 313 includes an object system 340. The object system 340 includes a bucket 341. The bucket 341 stores an object 342. The object 342 includes a user file 351 and a metadata file 352. The main memory device 311, the CPU 312, the auxiliary memory device 313, and the network card 314 are connected by a bus.

[0018] The user file 351 in Fig. 3 corresponds to the user file 221 in Fig. 2. The object system program 331 operates in cooperation with the replicator 236 of the computer 111 to update the user file 351. Immediately after the processing by the replicator 236 is completed successfully, the user file 351 in Fig. 3 is the same as the user file 221 in Fig. 2. The metadata file 352 contains data being processed when the object system program 331 updates the metadata file 352.

[0019] Specifically, when the object system program 331 receives an instruction to replicate a part from the replicator 236, it replicates the instructed part, calculates a hash value of the replicated part using the hash calculation program 332, and transmits the calculated hash value to the computer 111. Furthermore, when a part is transmitted from the replicator 236, the object system program 331 calculates a hash value of the received part using the hash calculation program 332, and transmits the calculated hash value to the computer 111. Note that hereinafter, the object storage 130 is also referred to as the "second computer."

[0020] The object storage 130 may further include a storage medium interface (not shown), read a program from a storage medium 99-2, and store it in a ROM (not shown) and the main storage device 311. Furthermore, the object storage 130 may receive a program as a data signal from the outside via a carrier wave 98-2, and store the received program in a ROM (not shown) and the main storage device 311.

[0021] FIG. 4 is a conceptual diagram explaining the status flag. The user file 221 is divided into multiple parts. A part is a unit of data transfer. However, the unit of data transfer here is not limited to the maximum data size in one communication, for example, the maximum size of an IP packet, and any size may be set for management purposes. In this embodiment, writing to a file is recorded in byte units, which are conveniently called "offsets." Since an offset corresponds to a write area, it may be one byte or two or more bytes. In other words, an offset may be included in one part or may span multiple parts.

[0022] As shown in Fig. 4, in this embodiment, status flags are attached to files, parts, and offsets to determine whether transfer is necessary, and are managed accordingly. In this embodiment, the status flags for files, parts, and offsets are managed as file flags 223, part flags 224, and offset flags 225. However, the three types of flags may be managed together in one place. That is, although Fig. 4 shows a file as if it has a status flag attached, in reality, the file and the status flag are not integrated. The same applies to the relationship between parts and status flags, and the same applies to the relationship between offsets and status flags.

[0023] When writing is performed in an area of ​​the user file 221, the written area is recorded as an offset, and the status flag of that offset is set to "Dirty." Also, the status flag of a part including an offset with a status flag of "Dirty" and the file to which writing has been performed is set to "Dirty." The detailed transition of the status flag will be described later.

[0024] The status flag includes three main states and two temporary states, for a total of five states. The three main states are "Clean", "Dirty", and "More Dirty". The two temporary states are "Copied" and "Transferred". A file can be in one of the three main states, and parts and offsets can be in one of five states.

[0025] The bottom of Figure 4 shows the priority of status flags. The highest priority is "More Dirty," the second highest priority is "Dirty," and the lowest priority is "Clean," "Transferred," and "Copied." This priority is taken into account when determining the status flags of parts and files, and is set to the highest priority value within the range. That is, if a part contains multiple offsets, the status flag of that part is set to the status flag with the highest priority among the multiple status flags. Also, if a file contains multiple parts, the status flag of that file is set to the status flag with the highest priority among the multiple status flags.

[0026] Fig. 5 is a diagram illustrating the flag data 222. As described above, the flag data 222 is composed of a file flag 223, a part flag 224, and an offset flag 225. The file flags 223 shown in the upper part of Fig. 5 are a set of status flags for each user file 221. In the example shown in Fig. 5, the status flag for "File001" is set to "Dirty," and the status flag for "File002" is set to "Clean."

[0027] The part flag 224 shown in the middle of Figure 5 is a collection of status flags for each file and for each part. The size of a part may be the same for all files, or may differ for each file. The size of a part is set in advance, for example. In the example shown in Figure 5, all parts are 5 MB, and the first part of "File001" is "Dirty" and the second part is "Clean." Both parts of "File002" are "Clean." In other words, "File001" has at least one part that is "Dirty," so it is "Dirty" on a file-by-file basis, and "File002" has all parts that are "Clean," so it is "Clean" on a file-by-file basis.

[0028] The offset flag 225 shown in the bottom of Figure 5 is a collection of status flags for each file and each offset. For convenience of drawing, only the offsets included in the first part, the range of 0 to 5MB, are listed for "File001." Here, it shows that writing was performed in "0x0011 to 0x0013" and the status was set to "Dirty." The offset flag 225 can record offsets at any granularity. For example, the entire area of ​​"File002" is "Clean," so it can be listed in one record.

[0029] Figure 6 shows the transition diagram of events and status flags for parts and offsets. "Update" in the table means that writing is being performed by the computer 111. "Transfer processing in progress" means that the MPU is in the middle of processing. "X" in the table means that the corresponding transition is not expected.

[0030] If the state is "Clean" and an update is made outside of transfer processing, it will transition to "Dirty". If the state is "Clean" and an update is made during transfer processing, it will transition to "More Dirty". If the state is "Clean" and an update is made during transfer processing or the transfer is completed, it will transition to "Copied". If the state is "Dirty" and an update is made outside of transfer processing, it will remain in "Dirty". If the state is "Dirty" and an update is made during transfer processing, it will transition to "More Dirty". If the state is "Dirty" and an update is made during transfer processing or the transfer is completed, it will transition to "Transferred". If the state is "More Dirty" and an update is made during transfer processing, it will remain in "More Dirty". If the state is "More Dirty" and it is recognized as an additional transfer target, it will transition to "Dirty".

[0031] If the status is "Copied" and an update is made outside of transfer processing, the status will transition to "Dirty". If the status is "Copied" and an update is made during transfer processing, the status will transition to "More Dirty". If the status is "Copied" and the transfer processing is confirmed, the status will transition to "Clean". If the status is "Copied" and the transfer processing is canceled, the status will transition to "Clean". If the status is "Transferred" and an update is made outside of transfer processing, the status will transition to "Dirty". If the status is "Transferred" and an update is made during transfer processing, the status will transition to "More Dirty". If the status is "Transferred" and the transfer processing is confirmed, the status will transition to "Clean". If the status is "Transferred" and the transfer processing is canceled, the status will transition to "Dirty".

[0032] Figure 7 is a transition diagram of the status flag for a file. As mentioned above, the file status flag can have three states: "Clean," "Dirty," and "More Dirty." The initial state and final state are "Clean." "Clean" means that no transfer has been made, or that a transfer has been made. "Dirty" means that an update has been made outside of the transfer process and that the file has not yet been transferred. "More Dirty" means that an update has been made during the transfer process and that the file will need to be resent next time.

[0033] If the status flag is in the "Clean" state and the file is updated outside of the transfer process, the IO hook program 234 transitions the status flag of the file to "Dirty." If the status flag is in the "Dirty" state and the transfer of the file is completed, the replicator 236 sets the status flag of the file to "Clean." If the status flag is in the "Dirty" state and the file is updated outside of the transfer process, the IO hook program 234 maintains the status flag of the file at "Dirty."

[0034] If the status flag of a file is in the "Dirty" state and the file is updated during the transfer process, the IO hook program 234 transitions the status flag of the file to "More Dirty." If the status flag of a file is in the "More Dirty" state and the file is recognized as one that needs to be resent, the replicator 236 transitions the status flag of the file to "Dirty." If the status flag of a file is in the "More Dirty" state and the file is updated during the transfer process, the IO hook program 234 maintains the status flag of the file at "More Dirty."

[0035] Figure 8 is a transition diagram of the status flags for offsets and parts. Strictly speaking, Figure 8 is a transition diagram of the status flags when the MPU transfer mode is "Complete." As mentioned above, the file status flag can take five states: "Clean," "Copied," "Transferred," "Dirty," and "More Dirty." In Figure 8, the arrows of the status transitions are numbered (1) to (9), but this is merely for convenience of explanation, and the size of the numbers has no relation to the order of the transitions or the priority of the transitions.

[0036] When the status flag is in the "Clean" state and the offset and part are updated outside of the transfer process, the IO hook program 234 transitions the status flag of the offset and part to "Dirty" as shown in (1). When the status flag is in the "Dirty" state and the offset and part are updated outside of the transfer process, the IO hook program 234 maintains the status flag of the offset and part at "Dirty" as shown in (2). When the status flag is in the "Clean" or "Dirty" state and the MPU of the file to which the offset and part belong is running and the offset and part are updated before or during the transfer, the IO hook program 234 transitions the status flag of the offset and part to "More Dirty" as shown in (3).

[0037] If the offset and part are updated during the transfer process of the offset and part while the status flag is in the "More Dirty" state, the IO hook program 234 maintains the status flag of the offset and part at "More Dirty" as shown in (4). If the replication process or transfer process of the offset and part is completed while the status flag is in the "Clean" or "Dirty" state, the replicator 236 updates the status flag of the file to "Copied" or "Transferred" as shown in (5). If the status flag is in the "Clean" or "Transferred" state and an update occurs after the offset and part are sent while the MPU of the file to which the offset and part belong is running, the IO hook program 234 transitions the status flag of the offset and part to "More Dirty" as shown in (6).

[0038] When the status flag is in the "More Dirty" state and the offset and part are recognized as offset and part that need to be resent, the replicator 236 transitions the status flag of the offset and part to "Dirty" as shown in (7). When the status flag is in the "Copied" or "Transferred" state and the MPU of the file to which the offset and part belong is completed, the replicator 236 transitions the status flag of the offset and part to "Clean" as shown in (8). When the status flag is in the "Copied" or "Transferred" state and the MPU of the file to which the offset and part belong is aborted, the replicator 236 transitions the status flag of the offset and part to "Clean" or "Dirty" as shown in (9).

[0039] Figure 9 is a transition diagram of the status flags for offsets and parts when the MPU transfer mode is other than "Complete." The difference between Figure 8 and Figure 9 is the addition of (10). Other than that, there are no differences between the two. When the status flag is "Copied" or "Transferred" and the offset and part are updated outside of the transfer process, the IO hook program 234 transitions the status flags for the offset and part to "Dirty," as shown in (10).

[0040] 10 is a schematic diagram showing the MPU of the user file 221. First, as shown in (1), the user U updates the user file 221 stored in the computer 111. This update is, for example, editing the file using a mouse or keyboard. As shown in (3), writing is performed in the user file 221. The IO hook program 234 extracts IO information associated with this writing as shown in (2), and rewrites the flags of the file, part, and offset in the flag data 222 where the writing has been performed.

[0041] The replicator 236 reads the flag from the flag data 222 at regular intervals as shown in (5), and transfers the updated file to the object storage 130 using the MPU as shown in (6). When the transfer is complete, the replicator 236 writes a flag to the flag data 222 upon completion of the transfer as shown in (7).

[0042] 11 to 15 are conceptual diagrams of the operation of the MPU. As shown in FIG. 11, the MPU is started with an updated user file 221 as the target file, which is the file to be transferred. At this time, the status flag of the target file is set to "Dirty." When the MPU is started, a file lock is first applied to the target file, and after a copy of the flag data 222 is obtained, the lock is released, for example. Then, it is determined whether or not each part, which is the unit of transfer, needs to be transferred to the object storage 130. In other words, the part flag 224 does not need to be included in the flag data 222, and the part flag 224 may be created at this timing based on the offset flag 225.

[0043] In the target file, parts that have not been written since the previous MPU have a status flag of "Clean." The replicator 236 instructs the object storage 130 to replicate the part using the transferred user file 351. When the replication is complete, the status flag of the part is updated to "Copied."

[0044] The explanation continues with Figure 12. The status flag of the part written after the previous MPU is "Dirty." The replicator 236 transfers that part to the object storage 130, and when the transfer is complete, the status flag of that part is updated to "Transferred." Next, we will explain Case 1 and Case 2 separately. Case 1 is the case where the target file was not updated during the MPU. In this case, when the duplication and transfer of all parts is completed, the replicator 236 issues a command to the object storage 130 to combine and confirm. The object system program 331 of the object storage 130 receives this command, combines the duplicated parts and the transferred parts, and confirms the MPU. The replicator 236 also updates the status flag of the target file from "Dirty" to "Clean."

[0045] We will continue the explanation by going to Figure 13. Case 2 is when the target file is updated during MPU. In this case, the response differs depending on the value of the transfer mode that is set in advance. If the transfer mode is "retry" or "differential transfer", the current MPU ends, but if it is "complete", the MPU continues as explained below to reflect the additional updates.

[0046] The explanation continues with Fig. 14. Fig. 14 and Fig. 15 show the process when the target file is updated during MPU and the transfer mode is "Completed." In this case, a file lock is applied to the target file again, and the lock is released after a copy of the flag data 222 is obtained again. Then, it is determined for each part whether additional transfer to the object storage 130 is necessary. For parts that require additional transfer, the status flag is updated from "Copied" or "Transferred" to "More Dirty."

[0047] The explanation continues with Figure 15. When a part that requires additional transfer is transferred to the object storage 130, the status flag of that part is updated from "More Dirty" to "Transferred." If the target file is further updated before the MPU of the target file is completed, the process is repeated from the file lock shown at the top of Figure 14. When there are no further updates to the target file after the file lock and the transfer and duplication of all parts are completed, the replicator 236 issues a command to the object storage 130 to combine and commit. Upon receiving this command, the object system program 331 of the object storage 130 combines the copied part and the transferred part in response to the command and commits the MPU. The replicator 236 also updates the status flag of the target file from "More Dirty" to "Clean."

[0048] 16 and 17 are flowcharts showing the processing of the replicator 236. This processing is stored in the computer 111 and is executed every fixed time, for example, every five minutes, for each updated user file 221 in order. Whether or not a user file 221 has been updated can be determined by referring to the file flag 223.

[0049] First, in step S300, replicator 236 determines whether the status flag of the target file is "Dirty." If replicator 236 determines that the status flag of the target file is "Dirty," it proceeds to step S301. If replicator 236 determines that the status flag of the target file is not "Dirty," i.e., is "Clean," it ends the processing shown in FIG. 16. Note that a file's status flag can also be "More Dirty," but "More Dirty" is a status flag that can only be set for a short period of time while an MPU is being executed. Furthermore, since the previous MPU has already completed when the processing shown in FIG. 16 is executed, the status flag will not be "More Dirty" at the time of the determination in step S300.

[0050] In step S301, the replicator 236 starts transferring multiple parts of the target file and proceeds to step S302. In step S302, the replicator 236 acquires exclusive access to the target file. In the following step S303, the replicator 236 starts detecting updates during the transfer process. In step S303, if the IO hook program 234 detects an update to the offset in the target file after this, the offset is transitioned to "More Dirty." In the following step S304, the replicator 236 acquires the offset flag 225 of the target file, and in the following step S305, the replicator 236 releases exclusive access to the target file.

[0051] In the following step S306, the replicator 236 generates part flags 224 based on the offset flags 225 acquired in step S304 and identifies the parts to be updated. The status flag for each part in the part flags 224 adopts the value of the status flag with the highest priority among the status flags for the offsets in the predetermined address range of the part. When the processing of step S306 is completed, the replicator 236 proceeds to FIG. 17 via circled A.

[0052] In FIG. 17, processing starts from circled A, and step S307 is executed first. In step S307, the replicator 236 determines the target part, which is the part to be processed. Any unprocessed part constituting the target file is selected as the target part. In the following step S307A, the replicator 236 determines whether the status flag of the target part is "Clean" or "Dirty." If the replicator 236 determines that the status flag of the target part is "Clean," it proceeds to step S308, and if the replicator 236 determines that the status flag of the target part is "Dirty," it proceeds to step S309. Details of steps S308 and S309 will be described later. When the replicator 236 has completed execution of step S308 or step S309, it proceeds to step S310.

[0053] In step S310, the replicator 236 determines whether the status flag of the target file is "More Dirty." If the replicator 236 determines that the status flag of the target file is "More Dirty," the process proceeds to step S314. If the replicator 236 determines that the status flag of the target file is not "More Dirty," the process proceeds to step S311. In step S311, the replicator 236 determines whether all parts of the target file have been processed, i.e., whether a part was determined to be the target part in step S307. If the replicator 236 determines that all parts of the target file have been processed, the process proceeds to step S312. If the replicator 236 determines that at least one part of the target file has not been processed, the process returns to step S307. In step S312, the replicator 236 executes a merge confirmation process, which will be described later. In the following step S313, the replicator 236 executes a post-stage process, which will be described later, and then ends the process shown in FIG. 17.

[0054] In step S314, the replicator 236 determines the set transfer mode. If the replicator 236 determines that the transfer mode is "retry," it proceeds to step S315; if the replicator 236 determines that the transfer mode is "differential transfer," it proceeds to step S316; and if the replicator 236 determines that the transfer mode is "completion," it proceeds to step S317. In step S315, the replicator 236 executes a retry process, which will be described later, and ends the process shown in Figure 17. In step S316, the replicator 236 executes a differential transfer process, which will be described later, and ends the process shown in Figure 17. In step S317, the replicator 236 executes a completion process, which will be described later, and returns to step S302 in Figure 16 via circled B.

[0055] Fig. 18 is a flowchart showing the part transfer process shown in step S309 in Fig. 17. Before this process starts, the part to be transferred is determined in advance.

[0056] In step S320, the replicator 236 calculates a hash value of the part to be transferred using the hash calculation program 233. In the following step S321, the replicator 236 transmits the part to be transferred and the hash value calculated in step S320 to the object storage 130. In the following step S322, the replicator 236 receives the comparison result of the hash value of the part to be transferred from the object storage 130.

[0057] In other words, in this step, the replicator 236 determines the content of the comparison result received in step S322. If the replicator 236 determines that the hash values ​​are the same, it proceeds to step S325, and if the replicator 236 determines that the hash values ​​are different, it proceeds to step S324. In step S324, the replicator 236 performs a retry process, i.e., resends the part to be transferred, and returns to step S322.

[0058] In step S325, the replicator 236 acquires exclusive access to the file containing the part to be transferred, and proceeds to step S326. In step S326, the replicator 236 determines whether the status flag of the part to be transferred is "Dirty." If the replicator 236 determines that the status flag of the part to be transferred is "Dirty," the replicator 236 proceeds to step S327. If the replicator 236 determines that the status flag is not "Dirty," i.e., is "More Dirty," the replicator 236 proceeds to step S328. In step S327, the replicator 236 updates the status flag of the offset included in the part to be transferred from "Dirty" to "Transferred," and proceeds to step S328. In step S327, the replicator 236 releases exclusive access to the file containing the part to be transferred, and ends the processing shown in FIG. 18. The file is locked in step S325 to prevent writing from occurring during the processing of steps S326 to S327.

[0059] Fig. 19 is a flowchart showing the part duplication process shown in step S308 of Fig. 17. Before this process starts, the parts to be duplicated are determined in advance. Note that the replicator 236 can also be called a "duplication instruction unit" because it instructs the object storage 130 to duplicate parts, as will be explained below.

[0060] In step S330, the replicator 236 calculates a hash value of the part to be replicated. In the following step S331, the replicator 236 sends a replication instruction for the part to be replicated and the hash value calculated in step S330 to the object storage 130. In the following step S332, the replicator 236 receives the comparison result of the hash value of the part to be replicated from the object storage 130.

[0061] In the following step S333, the replicator 236 determines whether the hash values ​​are the same. In other words, in this step, the replicator 236 determines the content of the comparison result received in step S332. If the replicator 236 determines that the hash values ​​are the same, it proceeds to step S335, and if it determines that the hash values ​​are different, it proceeds to step S334. In step S334, the replicator 236 performs a retry process, i.e., resends the replication instruction, and returns to step S332.

[0062] In step S335, the replicator 236 acquires exclusive access to the file containing the part to be copied, and proceeds to step S336. In step S336, the replicator 236 determines whether the status flag of the part to be copied is "Clean." If the replicator 236 determines that the status flag of the part to be copied is "Clean," the replicator 236 proceeds to step S337. If the replicator 236 determines that the status flag is not "Clean," i.e., that the status flag is "More Dirty," the replicator 236 proceeds to step S338. In step S337, the replicator 236 updates the status flag of the offset included in the part to be copied from "Clean" to "Copied," and proceeds to step S338. In step S337, the replicator 236 releases exclusive access to the file containing the part to be copied, and ends the processing shown in FIG. 19. The file is locked in step S325 to prevent writing from occurring during the processing of steps S336 to S337.

[0063] Figure 20 is a flowchart showing the merge confirmation process shown in step S312 of Figure 17. First, in step S340, the replicator 236 merges the parts to generate a file. In the following step S341, the replicator 236 confirms the transfer of multiple parts. The process of this step is a counterpart to the process of S301 in Figure 16. In the following step S342, the replicator 236 receives a hash value of the object from the computer 111. In the following step S343, the replicator 236 calculates a hash value of the file obtained by merging in step S340. In the following step S344, the replicator 236 compares the hash value received in step S342 with the hash value calculated in step S343.

[0064] In the following step S345, the replicator 236 determines whether the hash values ​​are the same, in other words, the result of the comparison in step S344. If the replicator 236 determines that the two hash values ​​are the same, it proceeds to step S346, and if it determines that the two hash values ​​are different, it proceeds to step S347. In step S346, the replicator 236 sends part information to the object storage 130 as necessary, and ends the processing shown in Figure 20. In step S347, the replicator 236 enables the confirmation abnormality flag for multiple part transfer. In the following step S348, the replicator 236 deletes the object, and ends the processing shown in Figure 20.

[0065] 21 is a flowchart showing the retry processing shown in step S315 of FIG. 17. First, in step S350, the replicator 236 acquires exclusive access to the target file. In the following step S351, the replicator 236 ends detection of updates during the transfer process and proceeds to step S352. Step S351 is a pair of processing with step S303, and if the IO hook program 234 detects an update to the offset in the target file after this, the offset is transitioned from "More Dirty" to "Dirty." In the following step S352, the replicator 236 updates the status flag of the target file from "More Dirty" to "Dirty" and proceeds to step S353.

[0066] In step S353, replicator 236 determines a target offset. The target offset is determined to be any unprocessed offset in the target file. In the following step S353A, replicator 236 determines the value of the status flag of the target offset. If replicator 236 determines that the status flag of the target offset is "More Dirty," it proceeds to S354; if it determines that the status flag is "Transferred," it proceeds to S355; if it determines that the status flag is "Copied," it proceeds to S356; if it determines that the status flag is "Clean" or "Dirty," it proceeds to step S358.

[0067] In step S354, the replicator 236 updates the status flag of the target offset from "More Dirty" to "Dirty" and proceeds to step S358. In step S355, the replicator 236 updates the status flag of the target offset from "Transferred" to "Dirty" and proceeds to step S358. In step S356, the replicator 236 updates the status flag of the target offset from "Copied" to "Dirty" and proceeds to step S358.

[0068] In step S358, replicator 236 determines whether all offsets included in the target file have been processed. If replicator 236 determines that all offsets included in the target file have been processed, it proceeds to step S359, and if it determines that there are unprocessed offsets, it returns to step S353.

[0069] In step S359, the replicator 236 releases the exclusive access to the target file. In the following step S360, the replicator 236 enables the abort flag for the multiple part transfer. In the following step S361, the replicator 236 executes the post-stage processing described below, and ends the processing shown in Figure 21. Note that this post-stage processing includes duplication and deletion of the transferred parts.

[0070] Figure 22 is a flowchart showing the differential transfer process shown in step S316 of Figure 17. First, in step S370, the replicator 236 acquires exclusive access to the target file. In the following step S371, the replicator 236 ends detection of updates during the transfer process and proceeds to step S372. Step S371 is a pair of processing with step S303, and if the IO hook program 234 detects an update to the offset in the target file after this, the offset is transitioned from "More Dirty" to "Dirty." In the following step S372, the replicator 236 updates the status flag of the target file from "More Dirty" to "Dirty" and proceeds to step S373.

[0071] In step S373, replicator 236 determines a target offset. The target offset is determined to be any unprocessed offset in the target file. In the following step S374, replicator 236 determines whether the status flag of the target offset is "More Dirty." If replicator 236 determines that the status flag of the target offset is "More Dirty," it proceeds to step S375; if replicator 236 determines that the status flag of the target offset is not "More Dirty," it proceeds to step S376 without performing any special processing. In step S375, replicator 236 updates the status flag of the target offset from "More Dirty" to "Dirty" and proceeds to step S376.

[0072] In step S376, the replicator 236 determines whether all offsets of the target file have been processed. If the replicator 236 determines that all offsets of the target file have been processed, the process proceeds to step S377. If the replicator 236 determines that any offset of the target file has not been processed, the process returns to step S373. In step S377, the replicator 236 releases the exclusive access to the target file and ends the process shown in FIG. 22.

[0073] Figure 23 is a flowchart showing the completion process shown in step S317 of Figure 17. First, in step S380, the replicator 236 acquires exclusive access to the target file. In the following step S381, the replicator 236 updates the status flag of the target file from "More Dirty" to "Dirty." In the following step S382, the replicator 236 releases the exclusive access to the target file and ends the process shown in Figure 23.

[0074] Figures 24 to 26 are flowcharts showing the latter stage processing shown in step S313 in Figure 17. A series of processing continuing from Figure 16 is executed by determining in advance the target file, which is the file to be processed, as described above.

[0075] First, in step S390, replicator 236 determines whether the confirmation error flag of the target file is valid. If replicator 236 determines that the confirmation error flag of the target file is valid, it proceeds to step S394, and if replicator 236 determines that the confirmation error flag of the target file is not valid, it proceeds to step S391. In step S391, replicator 236 determines whether the abort flag of the target file is valid. If replicator 236 determines that the abort flag of the target file is valid, it proceeds to step S394, and if replicator 236 determines that the abort flag of the target file is not valid, it proceeds to step S392.

[0076] In step S392, the replicator 236 acquires exclusive access to the target file, and proceeds to step S400 in Fig. 25 via circled C. In step S394, the replicator 236 acquires exclusive access to the target file, and proceeds to step S420 in Fig. 26 via circled D.

[0077] 25 for further explanation. In step S400, which is executed from step S393 via the circled C, replicator 236 determines whether the status flag of the target file is "More Dirty." If replicator 236 determines that the status flag of the target file is "More Dirty," it proceeds to step S410, and if it determines that the status flag of the target file is not "More Dirty," it proceeds to step S401.

[0078] In step S401, the replicator 236 updates the status flag of the target file from "Dirty" to "Clean" and proceeds to step S402. In the following step S402, the replicator 236 determines a target offset, which is an offset to be processed. The target offset is determined to be any offset in the target file that has not been selected as the target offset in this step.

[0079] In the next step S403, the replicator 236 determines the status flag of the target offset. If the replicator 236 determines that the status flag of the target offset is "Transferred," the process proceeds to step S405. If the replicator 236 determines that the status flag of the target offset is "Copied," the process proceeds to step S404. If the replicator 236 determines that the status flag of the target offset is "Clean" or "Dirty," the process proceeds to step S406.

[0080] In step S404, the replicator 236 changes the status flag of the target offset from "Transferred" to "Clean" and proceeds to step S406. In step S405, the replicator 236 changes the status flag of the target offset from "Copied" to "Clean" and proceeds to step S406. In step S406, the replicator 236 determines whether all offsets of the target file have been processed. If the replicator 236 determines that all offsets have been processed, the replicator 236 proceeds to step S407, and if the replicator 236 determines that an unprocessed offset exists, the replicator 236 returns to step S402.

[0081] If the determination in step S400 is affirmative, the replicator 236 updates the status flag of the target file from "More Dirty" to "Dirty" in step S410, and then proceeds to step S411. In step S411, the replicator 236 determines a target offset, which is an offset to be processed. The target offset is determined to be any offset in the target file that has not been selected as the target offset in this step.

[0082] In the following step S412, replicator 236 determines whether the status flag of the target offset is “More Dirty.” If replicator 236 determines that the status flag of the target offset is “More Dirty,” it proceeds to step S413, and if replicator 236 determines that the status flag of the target offset is not “More Dirty,” it proceeds to step S414.

[0083] In step S413, replicator 236 changes the status flag of the target offset from "More Dirty" to "Dirty" and proceeds to step S414. In step S414, replicator 236 determines whether all offsets of the target file have been processed. If replicator 236 determines that all offsets have been processed, it proceeds to step S407, and if it determines that there are unprocessed offsets, it returns to step S411.

[0084] In step S407, which is executed when the determination in step S406 is affirmative or when the determination in step S414 is affirmative, the replicator 236 releases the exclusive access to the target file and terminates the processing shown in Fig. 25. The release of the exclusive access in step S407 corresponds to the acquisition of the exclusive access in step S392 in Fig. 24.

[0085] 26 for further explanation. In step S420, which is executed from step S395 via circled D, replicator 236 determines whether the status flag of the target file is "More Dirty." If replicator 236 determines that the status flag of the target file is "More Dirty," it proceeds to step S422, and if it determines that the status flag of the target file is not "More Dirty," it proceeds to step S421.

[0086] In step S422, the replicator 236 updates the status flag of the target file from "More Dirty" to "Clean" and proceeds to step S423. In step S423, the replicator 236 determines a target offset, which is an offset to be processed. The target offset is determined to be any offset in the target file that has not been selected as the target offset in this step.

[0087] In the following step S424, replicator 236 determines whether the status flag of the target offset is “More Dirty.” If replicator 236 determines that the status flag of the target offset is “More Dirty,” it proceeds to step S425, and if replicator 236 determines that the status flag of the target offset is not “More Dirty,” it proceeds to step S426.

[0088] In step S425, replicator 236 changes the status flag of the target offset from "More Dirty" to "Dirty" and proceeds to step S426. In step S426, replicator 236 determines whether all offsets in the target file have been processed. If replicator 236 determines that all offsets have been processed, it proceeds to step S421, and if it determines that there are unprocessed offsets, it returns to step S423.

[0089] In step S421, the replicator 236 releases the exclusive access to the target file and proceeds to step S427. Note that the release of the exclusive access in step S421 corresponds to the acquisition of the exclusive access in step S394 of Fig. 24. In step S427, the replicator 236 determines whether the abort flag for the target file is valid. If the replicator 236 determines that the abort flag is valid, it proceeds to step S428, and if it determines that the abort flag is invalid, it terminates the processing shown in Fig. 26. In step S428, the replicator 236 deletes the transfer part, and in the following step S429, it terminates the transfer of multiple parts and terminates the processing shown in Fig. 26.

[0090] 27 is a diagram showing a user interface for setting a transfer mode. A transfer mode setting screen 400 has a device selection field 401 for selecting a target storage device and a mode setting field 402 for setting a transfer mode. The configuration of the computer 111 shown in FIG. 2 has only one auxiliary storage device 213, but if multiple auxiliary storage devices 213 are provided or if a storage device external to the computer 111, such as a NAS (Network Attached Storage), is available, the device for which the mode is set can be changed by operating the device selection field 401.

[0091] The mode setting field 402 allows the user to select one of three modes available in the MPU, namely, "retry," "differential transfer," and "completion." When the user makes a selection in the device selection field 401 and mode setting field 402 and presses the setting button 403 shown in the lower right of the transfer mode setting screen 400, the setting is recorded in an area (not shown) of the computer 111. Note that the transfer mode setting does not have to be on a storage device basis, and may be set on a user file 221 basis, for example.

[0092] According to the first embodiment described above, the following advantageous effects can be obtained. (1) The file transfer system 100 transfers a target file to be updated in the computer 111 to the object storage 130 for each part into which the target file is divided. The file transfer system 100 includes an IO hook program 234 that records the update position of the target file in the computer 111 as an offset flag 225, a replicator 236 that references the offset flag 225 and determines whether each part of the target file has been updated (S306 in FIG. 16 ), and a replicator 236 that transfers the part determined by the replicator 236 to have been updated to the object storage 130. When a re-update occurs in which the target file is updated after the replicator 236 starts transferring any part included in the target file using the MPU, the replicator 236 transmits the re-updated part, which is the part updated by the re-updating, to the object storage 130 regardless of whether it has already been transferred. Therefore, the file transfer system 100 can efficiently transfer the target file from the computer 111 to the object storage 130 by detecting the occurrence of an update to the target file without temporarily copying the target file.

[0093] (2) The replicator 236 transitions the target file to a write-protected state before referencing the offset flag 225 (S302 in FIG. 16), and transitions the target file to a writable state after completing the offset flag reference (S305 in FIG. 16). This makes it possible to minimize the time during which writing to the target file is not possible.

[0094] (3) The file transfer system 100 includes a replicator 236 (S308 in FIG. 17, FIG. 19) that replicates, to the object storage 130, the part that the replicator 236 has determined not to have been updated, using the target file before the update stored in the object storage 130. In the object storage 130, the part transferred by the replicator 236 and the part replicated based on the instruction of the replicator 236 are integrated to form the target file after the update in the second computer (FIGS. 12, 15). Therefore, by using data that has been previously transferred, the amount of data transferred and the time required for data transfer can be reduced.

[0095] (4) The MPU by the replicator 236 is repeatedly executed at a predetermined time interval. When the transfer mode is "Complete" (S314 to S317 in FIG. 17), the replicator 236 repeatedly transmits the re-updated part to the object storage 130 without waiting for a predetermined time interval until no re-updating occurs. This allows the transfer to be completed quickly.

[0096] (5) The MPU by the replicator 236 is repeatedly executed at a predetermined time interval. When the transfer mode is "retry" or "differential transfer" (S314 → S315, S316 in FIG. 17), the replicator 236 repeats sending the re-updated part to the object storage 130 at a predetermined time interval until no re-updating occurs. This allows the execution time of each MPU that is executed periodically to be shortened.

[0097] (6) If the transfer mode is "retry," when a re-update occurs, the replicator 236 retransmits the part that has not been updated in the re-update and that has been transmitted previously (S315 in FIG. 17, FIG. 21). Therefore, it is possible to retransmit the part that has already been transmitted just in case.

[0098] (7) When the transfer mode is "differential transfer," if a re-update occurs, the replicator 236 does not re-transmit parts that have not been updated in the re-update and that have been transmitted previously (S316 in FIG. 17, FIG. 22). Therefore, processing time can be saved by re-transmitting only the data that became necessary due to the re-update.

[0099] (Variation 1) 17, the order of steps S310 and S311 may be reversed. That is, whether the target file has been updated during MPU and the status flag has been changed to "More Dirty" may be determined after the part duplication process or part transfer process has been completed for all parts of the target file. Also, the operation mode of the replicator 236 may be preset and may not support other transfer modes.

[0100] (Variation 2) In the above-described embodiment, the user file 221 that the computer 111 transmits to the object storage 130 is stored in the computer 111. However, the user file 221 does not have to be stored in the computer 111. For example, the user file 221 may be stored in another device connected to the computer 111 through a network.

[0101] In the above-described embodiment and modified examples, the functional block configurations are merely examples. Some functional configurations shown as separate functional blocks may be configured as an integrated unit, or a configuration shown in a single functional block diagram may be divided into two or more functions. Furthermore, some of the functions of each functional block may be provided by other functional blocks.

[0102] In the above-described embodiments and modifications, the program of the computer 111 is stored in a ROM (not shown), but the program may be stored in the auxiliary storage device 213. Furthermore, the computer 111 may be provided with an input / output interface (not shown), and the program may be loaded from another device as needed via the input / output interface and a medium available to the computer 111. Here, the medium refers to, for example, a storage medium detachable from the input / output interface, or a communication medium, i.e., a wired, wireless, optical, or other network, or a carrier wave or digital signal propagating through the network. Furthermore, some or all of the functions realized by the program may be realized by a hardware circuit or FPGA.

[0103] The above-described embodiments and modifications may be combined with each other. Although various embodiments and modifications have been described above, the present invention is not limited to these. Other embodiments conceivable within the scope of the technical concept of the present invention are also included within the scope of the present invention. [Explanation of symbols]

[0104] 100: File transfer system 111: Calculator 130: Object Storage 221: User files 222: Flag data 223: File flags 224: Part flag 225: Offset flag 234: IO hook program 236: Replicator 331: Object System Program

Claims

1. 1. A file transfer system that transfers a target file to be updated in a first computer to a second computer for each part into which the target file is divided, The part is a unit of data transfer, the target file before update has been transferred to the second computer in advance; The first computer an update recording unit that records an updated area of ​​the target file in the first computer as an offset flag; an update determination unit that references the offset flag and determines whether or not each part in the target file has been updated; a transfer unit that transfers the part determined by the update determination unit to have been updated to the second computer, A file transfer system in which, when a re-update occurs in which the target file is updated after the transfer unit starts transferring any of the parts contained in the target file, the transfer unit sends the re-updated part, which is the part updated by the re-update, to the second computer regardless of whether it has already been transferred or not, until the re-update no longer occurs.

2. 2. The file transfer system according to claim 1, The update determination unit transitions the target file to a write-protected state before referencing the offset flag, and transitions the target file to a writable state after completing the reference to the offset flag.

3. 2. The file transfer system according to claim 1, a copy instruction unit that copies the part determined by the update determination unit not to have been updated to the second computer by using the target file before the update stored in the second computer; A file transfer system in which, in the second computer, the part transferred by the transfer unit and the part copied based on the instructions of the copy instruction unit are integrated to form the updated target file in the second computer.

4. 2. The file transfer system according to claim 1, the determination by the update determination unit and the transfer by the transfer unit are repeatedly performed at predetermined time intervals; The transfer unit further repeats sending the re-updated part to the second computer without waiting for the predetermined time interval until the re-updating no longer occurs.

5. 2. The file transfer system according to claim 1, the determination by the update determination unit and the transfer by the transfer unit are repeatedly performed at predetermined time intervals; The transfer unit repeats transmitting the re-updated part to the second computer at the predetermined time intervals until the re-updating no longer occurs.

6. 2. The file transfer system according to claim 1, When the re-update occurs, the transfer unit further re-transmits the part that has not been updated in the re-update and that has been previously transmitted.

7. 2. The file transfer system according to claim 1, When the re-update occurs, the transfer unit does not re-transmit the part that has not been updated in the re-update and that has been previously transmitted.

8. 1. A file transfer method in which a target file to be updated in a first computer is transferred to a second computer for each part into which the first computer has divided the target file, comprising: The part is a unit of data transfer, the target file before update has been transferred to the second computer in advance; The first computer an update recording process for recording an updated area of ​​the target file in the first computer as an offset flag; an update determination process that refers to the offset flag and determines whether or not each part in the target file has been updated; a transfer process of transferring the part determined to have been updated by the update determination process to the second computer, In the transfer process, if a re-update occurs in which any of the parts contained in the target file is updated after the transfer process has begun, the re-updated part, which is the part updated by the re-update, is sent to the second computer regardless of whether it has already been transferred or not, until the re-update no longer occurs.

9. A file transfer program for causing a first computer to transfer a target file to be updated on the first computer to a second computer for each part into which the target file is divided, the program comprising: The part is a unit of data transfer, the target file before update has been transferred to the second computer in advance; The first computer an update recording unit that records an update position of the target file in the first computer as an offset flag; an update determination unit that references the offset flag and determines whether or not each part in the target file has been updated; a transfer unit that transfers the part determined by the update determination unit to have been updated to the second computer; A file transfer program in which, when a re-update occurs in which the target file is updated after the transfer unit starts transferring any of the parts contained in the target file, the transfer unit sends the re-updated part, which is the part updated by the re-update, to the second computer until the re-update no longer occurs, regardless of whether it has already been transferred.

Images