Device, method of performing file transaction, and method of performing access operation

Abstract

A method for performing a file transaction, the method comprising: providing a set of transaction instructions to perform one or more transaction operations on a device; in response to determining that the transaction instructions include one or more write transaction operations, wherein the one or more write transaction operations collectively relate to a first file group including at least one file object, the first file group having a first size, and each of the at least one file object including identification information, if the first size does not exceed available device memory: writing each of the at least one file object with an uncommitted file type; after the first file group is written, storing transaction information indicating that the transaction has been committed on device memory; in response to determining that one or more pre-existing file objects share identification information with any file object in the first file group, erasing one or more pre-existing file objects; and updating the type of each of the at least one file object in the first file group to a final type.

Description

Technical Field

[0001] This invention relates to a device, a method for performing file transactions, and a method for performing access operations. Background Technology

[0002] Computing devices include a persistent or non-volatile memory for storing information and firmware used for device operation. A hardware security module (HSM) is an example of a computing device. Persistent storage media can also exist independently of the computing device, such as in a physical authentication token (e.g., a smart card for storing information). Changes can be applied to information in a file system stored on the memory within such a device, such as erasing, writing, or overwriting transactions of information stored on the device. Devices such as hardware security modules and smart cards can store sensitive information, such as encryption keys. If a transaction on such a device is interrupted midway through completion, the file system may be left in an intermediate state, such as only some files or only a portion of files being erased, written, or overwritten. For example, if a network connection is lost while a transaction is in progress, a transaction failure may occur, leaving the file system in an intermediate state.

[0003] Devices such as HSMs and smart cards can contain small-footprint storage media, for example, with a total storage capacity of less than 1 megabyte (MB) or even less than 64 kilobytes (KB). For example, authentication tokens such as smart cards can contain a small storage space of 1 to 10 KB. Mechanisms to mitigate transaction failures suitable for use in such small-footprint devices are needed. Attached Figure Description

[0004] The apparatus and method according to a non-limiting embodiment will now be described with reference to the accompanying drawings, in which:

[0005] Figure 1(a) is a schematic diagram of a device according to an embodiment, wherein the device is a hardware security module;

[0006] Figure 1(b) is a schematic diagram of an example smart card device;

[0007] Figure 2 This is a schematic diagram of a system including a hardware security module device and a smart card device according to an embodiment;

[0008] Figure 3 The structure of a file system stored on a device according to an embodiment is shown;

[0009] Figure 4 This is a flowchart illustrating a method for performing file transactions according to an embodiment, wherein the transaction instructions include one or more write transaction operations;

[0010] Figures 5(a) to 5(c)A compression method applied to a file system as part of a method for performing file transactions is illustrated according to an embodiment;

[0011] Figure 6 This is a flowchart illustrating a method for performing file transactions according to an embodiment, wherein the transaction instructions include one or more erase transaction operations;

[0012] Figure 7 This is a flowchart illustrating a method for performing file access operations according to an embodiment. Detailed Implementation

[0013] In a first aspect, a method for performing file transactions is provided, the method comprising:

[0014] Provides a set of transaction instructions to perform one or more transaction operations on the device;

[0015] In response to determining that the transaction instruction includes one or more write transaction operations, wherein the one or more write transaction operations collectively involve a first filegroup comprising at least one file object, the first filegroup having a first size, and each of the at least one file object including identification information, if the first size does not exceed the available device memory, then:

[0016] Write each of the at least one file objects to an uncommitted file type;

[0017] After the first file group is written, transaction information indicating that the transaction has been committed is stored in the device memory;

[0018] In response to determining that one or more pre-existing file objects share identification information with any file object in the first file group, the one or more pre-existing file objects are erased;

[0019] Update the type of each file object in at least one file object in the first file group to the final type.

[0020] As mentioned above, if a file transaction on a device is interrupted midway through completion, it can leave the file system in an intermediate state, such as only some files or only a portion of files being written or overwritten. For example, updates to the smart card file system performed remotely over a secure network connection may be interrupted if the network connection is lost, or if the connected client or HSM end is removed or closed. The method described above provides improved transaction safety because a transaction is not considered committed until all content or (if updating one or more files) replacement content has been written. This allows the file system to be recovered in the event of an interruption, as transactions can be rolled forward (if committed) or rolled back (if not committed). This method enables the file system to automatically recover to a consistent state after a failure. Therefore, this method mitigates data loss in failure scenarios.

[0021] Furthermore, this method is suitable for file systems with small footprints. By writing each file object in at least one file object with an uncommitted file type, and then storing transaction information on the device indicating that the transaction instructions have been committed after the first filegroup has been written, the progress of transactions is recorded without storing a separate "transaction log." This makes the method particularly suitable for, for example, smart cards (which can have 1 to 10 KB of storage space), or other small, secure storage areas on hardware devices.

[0022] In an embodiment, the method further includes: determining whether a first size exceeds the available storage space on the device. Optionally, if the size of the file to be written does indeed exceed the available storage space on the device, a transaction with degraded transaction security can be performed, for example, making the security only for each file, rather than for each file group. In this case, the method may further include: performing a first determination of whether there is sufficient space to write the file object; writing the file object in response to determining that there is sufficient space to write the file object; storing transaction information on the device indicating that the transaction has been committed in response to determining that there is insufficient space to write the file object; determining whether any pre-existing file object shares identification information with any file object in the first file group that has been written with an uncommitted file type, and erasing the one or more pre-existing file objects in response to determining that one or more pre-existing file objects share the identification information; performing a second determination of whether there is sufficient space to write the file object; and writing the file object in response to determining in the second determination that there is sufficient space to write the file object.

[0023] Optionally, a further degraded transaction safety level can be applied, where files can be erased and rewritten first to free up space. In response to a second determination that there is insufficient space to write to a file object, the method may further include: determining whether any pre-existing file objects share identification information with that file object, and in response to determining that pre-existing file objects share the identification information, erasing the pre-existing file object; and writing data into the file object. This pull-down approach to a lower consistency level means that transaction behavior does not need to limit effective storage space when there is insufficient space to apply full consistency.

[0024] Files not included in a transaction instruction are never marked as uncommitted. The only changes to files not included in a transaction instruction will be moving them if the file system is compressed.

[0025] A flag system applied in a well-defined order can be used to indicate transaction progress, allowing interrupted transactions in the file system to be rolled forward or backward. Flags can be used to indicate committed transactions. According to embodiments, the method further includes clearing the committed flag after updating the type of each file object in at least one file object to the final type. The method may also include setting a progress flag indicating that a file transaction is in progress. Flags can be updated in a way that makes a “corrupted” file system unreadable by a malicious user, including formatting in a way that ensures that interrupted formatting can be detected. For example, during formatting, the “type” of the file system can typically be set to zero in the generic file system header, which would indicate a corrupted file system to the user. Furthermore, values ​​other than zero (e.g., hexadecimal 0xFF) can be used to indicate an unformatted and / or corrupted file system.

[0026] Alternatively, storing transaction information indicating that a transaction has been committed includes updating the type of each file object in at least one file object to a committed type.

[0027] This method can also mitigate other interruption issues, such as removing the smart card from the reader during writing, power failure, disconnection of the reader cable during writing, or software crashes in the smart card or HSM during writing.

[0028] Failures may occur where a single byte write fails. The method described above provides a mechanism so that a file is not considered committed until all of its contents or (if the file is being updated) replacement content has been written.

[0029] Interruptions to transactions can lead to file system corruption, thus making it impossible to navigate the file system, for example, because the file header is corrupted due to changes caused by the interruption. For instance, if the file length is stored in the file header, the location of subsequent files in memory will be unknown if the file header is corrupted. Furthermore, failures can occur where a single byte write fails. In this case, a corrupted file may result, for example, because only a portion of the file was written. If an existing file exists, a corrupted file may result due to a partial update being applied, or if the file is new, a corrupted file may result because only a portion of the file was written. File system-level corruption due to interruptions in header updates or file system compression used to free up space can also result in errors in the obviously readable file content.

[0030] In this embodiment, updating the file header includes storing a redundant copy of the file length. For example, the process for updating the file header may include storing a first copy of the new file length, storing information indicating the uncommitted header length for the file, and updating the file header with the new file length. This information is then changed so that it no longer indicates the header length is uncommitted after the file header has been updated. In the event of an interruption, the file header can be rolled forward based on the first copy of the new file length while the information indicating the header length is uncommitted is stored. For example, the first copy of the new file length may be stored in the file system header. In this way, writing the file header update ensures that an interruption while writing any given byte will not result in a malformed file system or a file system that appears to contain incorrect content. Furthermore, any files not included in the transaction will not be lost because the way the file header update is performed means that the file system can still be navigated if the change is interrupted. Improved usability is provided by mitigating corrupted files or corrupted file systems on the storage device. Improved security is also provided by resisting malicious damage to the stored content by attackers. This scheme mitigates damage that could be maliciously performed to cause a secure system to read incorrect data.

[0031] In this embodiment, the lower-level system will write individual bytes atomically, and the order between writes will be maintained. The order in which multiple bytes are written within a single write operation is not assumed.

[0032] In an embodiment, writing a file object with an uncommitted file type includes: searching for any free space block with a size greater than or equal to that of the file object; in response to finding two or more free space blocks, selecting a free space block with the minimum excess available space, which, when written, will not cause the remaining space block to fall below the minimum size. In response to not finding a free space block, a compression method can be performed. The compression method may include: moving a pre-existing file object of length L; storing information specifying the entry type of the entry being moved, the offset of the original start F2 of the entry data relative to the new start F1, and the length M of the data that has been successfully moved, wherein M is initially set to zero; wherein moving the pre-existing file includes: i. writing the next X bytes of the entry starting at address F1+M, where X is the minimum value selected from the following: offset sum (LM); ii. updating the value of M stored in the fragment index; iii. setting M = M+X and repeating steps i to iii until M equals L. The compression algorithm is performed in a manner that mitigates data loss in the event of an interruption. All file contents are recoverable, even if they were in two fragments during the move.

[0033] In this embodiment, the method provides a way to avoid the loss of a file being updated within a transaction when the update to a file is a redundant no-op, by reading existing file data and comparing it with the new data. If they are the same, they are removed from the list of files actually processed.

[0034] In an embodiment, erasing one or more pre-existing file objects includes: updating the entry type of the pre-existing file object to indicate an uncommitted free space type; rewriting the pre-existing file object with zeros; and updating the entry type to indicate a free space type. The method may further include: in response to determining that the newly formed free space is adjacent to one or more other free space entries, merging the adjacent entries to form a single free space entry.

[0035] In an embodiment, the method further includes: in response to determining that the transaction instruction includes one or more erase transaction operations, the one or more erase transaction operations collectively relating to a second file group to be erased from the device, the second file group including at least one file object; updating each file object in the at least one file object in the second file group to an uncommitted erase type; and after the second file group is updated to an uncommitted erase type, storing transaction information indicating that the transaction has been committed on the device; and erasing the first file group. Storing the transaction information indicating that the transaction has been committed may include setting a committed flag, and wherein the method further includes clearing the committed flag after erasing the first file group. If the transaction instruction includes one or more write transaction operations and one or more erase transaction operations, then the transaction information indicating that the transaction has been committed is stored after writing to the first file group and after updating the second file group to an uncommitted erase type.

[0036] According to another aspect, a method for performing an access operation is provided, the method comprising:

[0037] Determine whether the device stores transaction information indicating that a transaction has been committed;

[0038] Determine if any file objects of the Uncommitted file type exist;

[0039] In response to determining that the device stores transaction information indicating that a transaction has been committed and that one or more file objects with an uncommitted file type exist, the type of the one or more file objects with an uncommitted file type is updated to the final type; and

[0040] In response to determining that the device does not store transaction information indicating that a transaction has been committed and that there are one or more file objects with the uncommitted file type, the files with the uncommitted file type are erased.

[0041] In devices such as HSMs or smart cards, the file system can become vulnerable to malicious attacks if processes are interrupted midway through completion. For example, if the formatting process is interrupted, it may mean that not all data is erased. Malicious sorting and timed updates to the file system on a storage device can lead to compromised control by an attacker, potentially resulting in the leakage of secrets, bypassing access controls, or causing damage to the rest of the device. Recovery processes mitigate this attack by automatically rolling forward or backward from interrupted processes when access attempts are made.

[0042] In an embodiment, in response to determining that the device stores transaction information indicating that a transaction has been committed and that there are one or more file objects with an uncommitted file type, and in response to determining that one or more pre-existing file objects share identification information with any file with an uncommitted file type, the one or more pre-existing file objects are erased.

[0043] In one embodiment, the method further includes determining whether a flag indicates that a transaction is in progress, and wherein, if the flag indicates that a transaction is not in progress, the determination of whether the device stores transaction information indicating that the transaction has been committed and whether any file object with an uncommitted file type exists is not performed. In another embodiment, the method further includes: determining whether one or more flags indicate a valid state of the device; and returning an error in response to determining that the state of the device is invalid.

[0044] In one embodiment, the method further includes: determining whether any file objects have an uncommitted erase type, and erasing the one or more file objects with the uncommitted erase type in response to determining that the device stores transaction information indicating that a transaction has been committed and that one or more file objects have the uncommitted erase type. In another embodiment, in response to determining that the device does not store transaction information indicating that a transaction instruction has been committed and that one or more file objects have the uncommitted erase type, the method further includes restoring the one or more file objects to their final type.

[0045] This method defines new "uncommitted file" and "uncommitted erased file" types. This mitigates the loss of file readability when an interrupted file system state is read by an earlier version of the file system.

[0046] According to another aspect, a non-transitory data carrier is provided that carries processor control code, which, when executed, causes a computer to perform the methods described above. This method is a computer-implemented method. Since some methods according to the embodiments can be implemented in software, some embodiments include computer code provided on any suitable carrier medium. The carrier medium may include any storage medium, such as a CD-ROM, magnetic device, or programmable storage device, or any transient medium (e.g., any signal, such as an electrical, optical, or microwave signal). The carrier medium may include a non-transitory computer-readable storage medium.

[0047] According to another aspect, an apparatus is provided, comprising:

[0048] One or more processors are configured as follows:

[0049] A set of transaction instructions to perform one or more transaction operations on a device memory located on the device or another device;

[0050] In response to determining that the transaction instruction includes one or more write transaction operations, wherein the one or more write transaction operations collectively involve a first filegroup comprising at least one file object, the first filegroup having a first size, and each of the at least one file object including identification information, if the first size does not exceed the available device memory, then:

[0051] Write each of the at least one file objects to an uncommitted file type;

[0052] After the first file group is written, transaction information indicating that the transaction has been committed is stored in the device memory;

[0053] In response to determining that one or more pre-existing file objects share identification information with any file object in the first file group, the one or more pre-existing file objects are erased;

[0054] Update the type of each file object in at least one file object in the first file group to the final type.

[0055] According to another aspect, an apparatus is provided, comprising:

[0056] One or more processors are configured as follows:

[0057] Determine whether the device memory stores transaction information indicating that a transaction has been committed, wherein...

[0058] The device memory is located on the device or on another device;

[0059] Determine if any file objects of the Uncommitted file type exist;

[0060] In response to determining that the device stores transaction information indicating that a transaction has been committed and that one or more file objects with an uncommitted file type exist, the type of the one or more file objects with an uncommitted file type is updated to the final type; and

[0061] In response to determining that the device does not store transaction information indicating that a transaction has been committed and that there are one or more file objects with the uncommitted file type, the files with the uncommitted file type are erased.

[0062] The device may be a hardware security module. The device memory may reside on the hardware security module or in a separate device such as a smart card. In this embodiment, the device memory is persistent memory. In this embodiment, the device is a security device.

[0063] Figure 1(a) is a schematic diagram of a hardware security module (HSM) 21 device according to an embodiment. An HSM is a device that securely stores and manages cryptographic keys and performs cryptographic functions. An HSM may include both physical and non-physical security attributes that provide security. Non-physical security attributes may include the use of encryption, i.e., including software or physical components within the device for performing encryption of stored data. Physical security attributes may include tamper switches triggered by physical access, and tamper-proof membranes surrounding, for example, the physical boundaries of the device.

[0064] HSM 21 includes non-volatile or persistent memory 305. Memory 305 stores information (e.g., cryptographic keys) and programs. Non-volatile memory 305 can include any form of non-volatile device memory, such as flash memory, optical disk, or magnetic hard disk. Non-volatile memory 305 can be physically secure and tamper-proof, for example, by including a physical security element such as a membrane that covers the entire device and cannot be removed without damaging the underlying physical hardware.

[0065] HSM 21 also includes a processor or CPU 303 configured to execute programs stored in non-volatile memory 305. One or more programs may be referred to as "firmware"; however, typically these programs comprise a set of computer instructions stored in non-volatile memory 305. The computer instructions or firmware can be written in any of a variety of programming languages. Firmware includes computer instructions embodying one or more methods as described herein. Firmware may also include computer instructions embodying one or more of the following cryptographic functions: cryptographic key generation; key derivation; encryption; decryption; and digital signature functions (e.g., digital signature or verification of digital signature).

[0066] CPU 303 communicates bidirectionally with non-volatile memory 305 and with working memory or RAM 309 via wired connections. RAM 309 corresponds to the operating memory of CPU 303. Processor 303 includes logic circuitry that responds to and processes instructions in code stored in working memory 309. Specifically, when executed, a program is represented as a software product or process stored in working memory 309. Firmware may be embedded in a hardware security module during manufacturing or may be provided as a whole or in part after manufacturing. For example, firmware may be introduced as a computer program product, which may be available for download. Alternatively, existing firmware may be modified by updates or plugins. Execution of various programs by processor 303 will enable the implementation of the methods described herein.

[0067] HSM device 21 may include additional components, such as board support processor 313. Board support processor 313 is configured to communicate with multiple on-board sensors to monitor the operation of the main CPU and other hardware components of hardware security module 21. Sensors may include, but are not limited to, CPU and / or board temperature sensors, voltage and / or current sensors. In this example, HSM device 21 also includes a cryptographic coprocessor 311, which is configured to perform certain cryptographic functions in place of CPU 303.

[0068] An application key associated with the client is used to execute one or more cryptographic functions contained in the firmware on the HSM 21. When a client request in the form of a command is received at the HSM 21, the associated one or more application keys are retrieved and loaded into the RAM space of the firmware process. The hardware security module 21 can then use them to perform one or more cryptographic operations as described above. These keys can be stored in non-volatile memory 305 on the HSM device 21. Alternatively, these keys can be stored encrypted outside the HSM device 21 and loaded onto the HSM device when a specific cryptographic operation needs to be performed. For example, application-specific keys can be stored on a smart card. For enhanced security, the keys can be encrypted with an HSM key and then divided across multiple smart cards using a secret-sharing scheme that allows a number of cards to reconstruct the original key (e.g., there may be five cards with partial keys, and any three of them have enough information to reconstruct the key). The portion of the key information on each smart card is called a "share".

[0069] Figure 1(b) is a schematic diagram of an example smart card device 23 that may be included in a system according to an embodiment. The smart card device 23 includes non-volatile or persistent memory 29. Memory 29 stores information such as password keys. Non-volatile memory 29 may include any form of non-volatile device memory, such as flash memory, optical disk, or magnetic hard disk. Optionally present on some smart cards, a smart card applet processor 24 represents any processing control present between connector point 26 and memory. The applet processor 24 may be, for example, fixed circuitry or a small programmable environment. When using the smart card remotely (e.g., Figure 2 In the case of a smart card 23 in the HSM, the applet processor 24 includes logic for terminating one end of a secure communication channel with the HSM. When the smart card is locally inserted into the HSM, communication may or may not be conducted through the secure channel.

[0070] Figure 2A schematic diagram of a system according to an embodiment is shown, including an HSM device 21 and a first smart card device 23 belonging to a client. The system also includes a client device (user equipment) 25. The user equipment 25 may be a general-purpose computer. This arrangement can be used when the first smart card 23 and user equipment 25 are located in a first location and the HSM 21 is located in a second location away from the first location. The second location may be a host system, such as a cloud service provider system.

[0071] A second smart card device 27 is also shown, which is inserted locally into the HSM device 21. Such an arrangement can be used for HSMs that are placed directly in or inserted into, for example, a client PC or server directly under client control.

[0072] A second smart card 27 may be provided in place of the first smart card 23, or a second smart card 27 may be provided in addition to the first smart card 23, as shown in the figure.

[0073] This figure illustrates various ways in which HSM device 21 can perform access operations and file transactions, as described below regarding methods on device memory. Specifically, HSM device 21 can perform access operations and file transactions on a remote smart card, such as on a first smart card device memory 29. Additionally or alternatively, HSM device 21 can perform access operations and file transactions on a locally inserted smart card, such as on a second smart card device memory 30. HSM device 21 can also perform access operations and file transactions on local HSM memory 305.

[0074] The algorithm described below is executed on HSM device 21, where the smart card memory is treated as memory to be read from and written to via some implementation-specific interface. This can be a virtual mapping of the smart card device memory, allowing code to be written to it transparently in the same way that the HSM would write to local RAM. Device read and write functions provided by a lower layer can be used, which accept or return blocks of bytes and take the memory address (and, if it is a read, the length to be read) as parameters, whereby the lower layer translates these reads and writes into appropriate device-specific operations for the underlying HSM local storage or the smart card device type discussed. Those skilled in the art will understand that in practice, to support the implementation of read and write operations, there will be one or more layers of code and communication between the algorithm described herein and the device memory, such as a device driver on the HSM and a small amount of supporting code running on the smart card. More complex smart cards, such as JavaCards capable of running programs themselves, are still treated as memory to be written to and read from by the HSM, and the code associated with the algorithm described herein executes on the HSM itself. Such smart cards may have logic for establishing secure communication with the HSM, but this does not affect the performance of the algorithm described below.

[0075] It should also be noted that whenever the algorithm specifies a read from memory, if the implementation has already read that portion of memory, or previously written to that portion, this may actually be a read from a cache in the HSM memory. If the storage device is a removable storage device, such as a smart card, any such cache will become invalid when the storage device is removed.

[0076] It's also important to note that whenever the algorithm specifies a write operation, this needs to be a synchronous operation so that when the write operation reports successful completion, the specified byte has been successfully written to the actual underlying memory, not just the cache. No assumptions are made about the intermediate state of the write if any single byte is written atomically—that is, if any single byte is the byte the algorithm requires to be written, or it is not written at all but is by no means a corrupted byte (e.g., the bytes in the block being written can be written from the highest address to the lowest address instead of the other way around, or in chunks).

[0077] The first smart card 23 includes a first smart card memory 29 storing a first file system. The second smart card 27 includes a second smart card memory 30 storing a second file system. The HSM 21 includes a secure non-volatile storage area 305 storing a third file system 31. The file system stored on each of the first smart card 23, the second smart card 27, and the HSM 21 is a file system used for storing and retrieving data (e.g., password keys). The memory on smart cards 23 and 27 and the HSM 21 can have a size ranging from, for example, from several hundred bytes to approximately 64KB.

[0078] A first smart card 23 is inserted into a card reader connected to user equipment 25. The first smart card 23 uses its own application to establish a secure communication channel to HSM 21. The card reader connection to user equipment 25 allows stored content to be transferred to and from smart card 23. HSM 21 can be communicatively coupled to a computer or server device in a host system (not shown) via interface 307, which includes a communication link. For example, it can include a USB connector, or HSM device 21 can be a PCI express card directly inserted into the computer or server device. Communication between user equipment 25 and the host system can be performed via an Internet connection. A TCP network connection is used for tunneling communication between the first smart card 23 and HSM 21 via devices in user equipment 25 and the host system (not shown). The PCI Express or USB connector in interface 307 is used for communication with HSM device 21, including sending commands and receiving responses, and as a conduit for any network communication (e.g., communication with a remote smart card that communicates with HSM via a secure, encrypted, and authenticated channel). A PCI Express or USB connection will be used to connect to the host PC or server. The host server can also be a tamper-proof chassis equipped with an HSM that is not user-manageable.

[0079] A second smart card 27 is locally inserted into the HSM 21. The HSM may have a separate serial connector for locally attached smart card readers, which can be an external reader that is inserted into a socket on the HSM by the user, or it can be embedded in a surrounding enclosure around the HSM instead of being inserted by the user. The locally attached reader can support less complex smart card types (most of which are dumb memories) as well as smart cards running their own applications to establish a secure communication channel to the HSM. Less complex smart cards are not used remotely because they cannot form such a secure channel.

[0080] In the example, a rewrite transaction is performed to overwrite files stored in the file system on the first smart card 23. For example, changing the passphrase on the first smart card 23 involves replacing an existing file stored on the first smart card 23 with a new file encrypted with the new passphrase. This can be performed using a rewrite transaction, as will be described in this document.

[0081] Execution of various programs by the processor 303 on the HSM 21 will result in the implementation of the methods described herein to perform transactions on the first smart card 23 file system, the HSM local memory 305, and / or the second smart card file system 30. Implementation code corresponding to the various methods described herein is included in the HSM firmware on the HSM 21.

[0082] For example, an update to the file system of the first smart card 23 is initiated by the HSM 21 and performed remotely (i.e., via a network connection). The network connection may fail during the update process, or the connected client or HSM end may be removed or closed midway. Such a connection failure can result in a corrupted smart card file system, which may therefore hinder the retrieval or readability of the information stored therein.

[0083] This article describes a file system structure in persistent storage media (e.g., physical authentication tokens or HSM memory). This file system is particularly suitable for storing and retrieving data such as cryptographic keys, and can be implemented on a small secure storage area on a physical token such as a smart card or an HSM (Hardware Security Module). This file system is also suitable for media with storage capacities ranging from a few hundred bytes to approximately 64KB.

[0084] However, it will be immediately apparent to those skilled in the art that the file systems and methods disclosed herein are suitable for use on any device having persistent storage media and storage capacity of any size. Therefore, the size of the storage device or its intended use does not limit the ability to use the file systems or methods disclosed herein.

[0085] The methods described herein aim to mitigate potential problems involved in updating such a file system (e.g., a smart card file system). The file system structure and the methods for updating said file system structure mitigate problems when updates performed remotely (e.g., via a network connection) are interrupted, for example, if the network connection is lost, or alternatively, if the connection is removed or closed at the HSM. Interruptions can also occur during updates to the local smart card or HSM memory. For example, this document discloses how to mitigate problems such as removing the smart card from the reader during a write operation, power failure, disconnection of the reader cable during a write operation, or software crashes in the smart card or HSM during a write operation.

[0086] The file system structure and corresponding methods disclosed herein can also address other problems, such as mitigating data corruption in the file system during connection loss and maintaining the security of sensitive information. For example, corruption can be mitigated by ensuring that intermediate states of file transactions cannot be read and subsequently by rolling forward or backward through repair operations. In some embodiments, data security can be maintained by updating flags in the file system header to intentionally indicate a corrupted or unformatted file system.

[0087] The methods disclosed in this paper provide improved availability while reducing file and file system storage size overhead, enabling support for storage devices with small capacities (e.g., less than 64KB). Improved availability is achieved through enhanced transaction safety for changes involving single files or groups of files. Transaction safety for file groups provides the ability to update the file group atomically and to roll forward or backward the entire group in the event of a failure. Transaction safety for updates to individual existing files is also relevant to avoid intermediate states such as partial updates or two copies of a file. Similarly, transaction safety allows updates to roll forward or backward in the event of a failure.

[0088] When sufficient free space is available, the exposed method supports changes across multiple files as a single atomic transaction. Users can optionally configure this implementation so that the transactional method automatically degrades to a lower consistency level when insufficient storage is available for higher consistency. At the lower consistency level, file header updates are still performed by storing a redundant copy of the file length, thus mitigating corrupted files and file systems. Furthermore, the header for a new entry is not visible until it has been fully written; that is, the header is written into free space, and then the size of the free space entry is safely adjusted via the redundant length so that the new entry is visible as an atomic operation. In this way, any file header update is written such that an interruption while writing any given byte cannot result in a malformed file system or a file system that appears to contain erroneous content.

[0089] Furthermore, a file is not committed until all of its contents (or, if a file is being updated, its contents are replaced) have been written. At lower consistency levels, consistency is at the file level rather than across filegroups, where no single file will be committed until all contents / replacements have been written. Additionally, at lower consistency levels, files not included in transaction instructions are not lost in the event of a failure. This is because the way file headers are updated means that if changes are interrupted, the filesystem can still be navigated, i.e., a redundant copy of the file length is stored. Furthermore, files not included in transaction instructions are never marked as uncommitted at any time. The only changes to files not included in transaction instructions will be moved if the filesystem is compressed. In this case, the transaction-safe compression algorithm ensures no data loss in the event of an interruption, as the file contents are recoverable even if they are in two fragments during the move.

[0090] Improved availability can also be achieved through enhanced fault safety, which prevents file or file system corruption or loss in the event of a write interruption to the underlying storage. Fault safety specifies that any single-byte write can fail, but the file system can be recovered. Fault safety improvements are provided for any writes to both the smart card and the HSM storage. For example, improved fault safety is provided during file system compression, header updates, or file system formatting.

[0091] The method disclosed herein can also reduce the risk of residual magnetism in previous data (e.g., residual magnetism in physical hard disk drive storage) by ensuring that all erased files and formatted storage are completely zeroed out. Furthermore, it should be understood that those skilled in the art can similarly apply a scheme to replace zeroing with a more robust formatting-safe operation (e.g., a residual magnetism-resistant erase operation suitable for the specific physical storage used).

[0092] Figure 3 This is a schematic diagram of a file system structure that can be stored on a device according to an embodiment. The diagram illustrates a 1024-byte file system, including a header, a first file named / a / 1 with content "Example 1", free space gaps, a second file named / b / 2 with content "Example 2", and free space blocks for remaining storage. In this example, the text is encoded in ASCII, and the 2-byte length encoding is little-endian. However, this is merely illustrative; the actual encoding may be implementation-defined.

[0093] Locking reads and writes is the responsibility of the file system user; the file system itself is not reentrant on a particular device. It is assumed that the stored content will not change except for any modifications made by the algorithm itself during its execution period. Multiple reads can actually occur simultaneously without interfering with each other, but must be mutually exclusive with any writes. Writes are mutually exclusive with any other writes.

[0094] Alternatively, implementations can specify their own locking structures for reads and writes, for example, to ensure that parallel write operations do not collide with each other. This can be done as follows:

[0095] Typically, all reads acquire a "shared" lock on the specific storage device on which the operation is being performed (it can operate concurrently with any other "shared" lock, but waits if a currently held "exclusive" lock exists). All write operations acquire an "exclusive" lock, which prevents any further reads or writes and waits until all "shared" and "exclusive" locks are released before proceeding. Exclusive locks are typically held on the storage device throughout the implementation via the following interfaces: Tknfs_Transact; Tknfs_CreateFile; Tknfs_EraseFile; Tknfs_Format; Tkfs_UpdateFile. Shared locks are typically held on the storage device throughout the implementation via the following interfaces: Tknfs_ListFiles; Tknfs_ReadFile.

[0096] Typically, it is assumed that the underlying storage system writes individual bytes atomically and maintains the order between writes. Furthermore, the order in which multiple bytes are written within a single write operation is not assumed. Using a storage system that lacks this characteristic will reduce fault tolerance.

[0097] The general structure of the file system and the file formats used are described in detail below. The following terms will be used in this specification:

[0098] –STORAGE_SIZE is the total size of the storage space used by the file system;

[0099] –LEN_SIZE is the number of bytes used to specify the size of a file system entry (LEN_SIZE is typically 2 bytes (16 bits), and can be used to specify a file system entry with a size of up to 65535 bits).

[0100] – When using sizeof(TYPE) to indicate size, it represents the number of bytes used to store the field or header called TYPE.

[0101] A file system consists of a single file system header, referred to here as FS_Header, followed by one or more contiguous blocks of variable size, referred to here as Entry_Blocks or entries. An entry can be a file or simply free space. Files can be in various states reflecting transactions and file system compression operations. These states are defined in detail below.

[0102] The formats of FS_Header and Entry_Block are defined in the following sections:

[0103] FS_Header format

[0104] For the purposes of this specification, the file system header (FS_Header) begins at address '0' in the underlying storage medium.

[0105] The file system header can be prefixed with one or more implementation-defined identifier bytes to distinguish it from other file system types. In some examples, the file system header may not be at the beginning of the physical memory address space; in this case, the implementation will apply appropriate translations to convert the local file system address to the actual physical address.

[0106] The file system header typically includes the following fields in the order listed:

[0107]

[0108]

[0109] The FS_Type byte typically has the following structure, consisting of 8 bits:

[0110]

[0111] The values ​​0x00 and 0xFF (i.e., hexadecimal values ​​of 0 and 255 respectively) are intentionally considered never to be valid FS_Type values ​​for the file system. This can help identify unformatted storage and also has security advantages: for example, during the formatting process, the value of FS_Type can be set to 0x00, causing entries to be considered corrupted. Once formatting is complete, FS_Type will be set to FSTYPE_FORMATTED (0x01) to indicate that the file system has a valid format. This can then prevent malicious attempts to interrupt the formatting / erase process and subsequently read entries.

[0112] The following type codes and corresponding bit flags (hexadecimal values ​​in the "Value" column) are defined. Therefore, the bit flags below correspond to the flags available in "Bit Offsets" 5 and 6 of the table above describing the FS_Type structure.

[0113]

[0114] More specifically, if only the FSTYPE_DIRTY flag is set, if the transaction is interrupted during the transaction, the progress of the transaction will be "rolled back" to restore the file system to its original state when it restarts.

[0115] If both the FSTYPE_DIRTY and FSTYPE_COMMITTED flags are set, the transaction will continue its progress and complete when the interrupted transaction restarts; this is known as "roll-forward".

[0116] It will be understood that the above values ​​are for illustrative purposes and for consistency within this disclosure; however, different example implementations may use other suitable sets of values.

[0117] 'Entry_Block' format

[0118] In the address space following the file system header (FS_Header), the file system typically includes one or more Entry_Block file system entries (e.g., data files).

[0119] File system entries can include any of the following: a single padding byte, a free space block, a file, etc.

[0120]

[0121]

[0122] Regarding the name "Entry_Len" above, it will be understood that the current file system is particularly well-suited for use in very small storage areas (e.g., less than 64KB), so 2 bytes is usually sufficient. However, it will generally be understood that any suitable number of bytes can be used, i.e., to define longer entries. Therefore, Entry_Len can be extended to additional bytes if needed, and the block length will be updated accordingly when stored in other locations. For example, a third byte, len_mid, could be used, in which case the total length of the block, including the header length, could be calculated as: len_lo + (256 * len_mid) + (65536 * len_hi).

[0123] The Entry_Type byte can be defined as follows:

[0124]

[0125] Below are examples of type codes corresponding to different types of entries. The table below also lists the remedial actions associated with each type code. Remedial actions are performed when an access operation is executed on the device, and will be discussed below regarding... Figure 7 The repair action will be described in more detail. It will be understood that the values ​​(i.e., the bit flags given in hexadecimal) are for illustrative purposes, and the implementation may use other suitable values.

[0126]

[0127]

[0128]

[0129] A collection of interfaces is provided for reading from and writing to the file system. Generally, an "interface" will be understood as a code function that implements any algorithm described in detail below, such as code stored in the HSM firmware. These provide interfaces for code that needs to read / write files in the file system to utilize the functionality described below; that is, they provide a library of functions for use by other code. Various details can be customized according to the needs of the user of the file system. General aspects of the interfaces, such as those for formatting the file system, implementing file system transactions, performing updates to files, enumerating file system contents, and reading files, are described here.

[0130] In this embodiment, the file system itself does not implement any access control over files and does not enforce locking for concurrent access to the same storage device (alternatively, the scheme specified above for implementing "shared" and "exclusive" locks held on the storage device via an interface can be used). These are the responsibility of the file system user. The algorithm / interface is configured to assume that no changes are made to the storage while any other code or device is being executed.

[0131] Consistent with the example structures of file system entries and file system headers described above, this document describes the following example algorithms and methods for interfacing with file systems. These algorithms and methods are intended to provide improved fault and transaction safety for storage media, particularly those with small footprints (e.g., approximately or less than 64KB). The methods can be implemented on devices such as those described above (e.g., HSM device 21) and are associated with file systems, for example, stored in local HSM memory 305, local smart card memory 30, or remote smart card memory 29.

[0132] Specifically, Figure 4 This is a flowchart illustrating a method for performing file transactions according to an embodiment, wherein the transaction instructions include one or more write transaction operations. Figures 5(a) to 5(c) A compression method applied to a file system as part of a method for performing file transactions is illustrated according to an embodiment. Figure 6 This is a flowchart illustrating a method for performing a transaction according to an embodiment, wherein the transaction instruction includes one or more erase transaction operations. Figure 7 This is a flowchart illustrating a method for performing an access operation according to an embodiment.

[0133] Enumerating file systems

[0134] Before performing file transactions, you can execute the method to enumerate the file system as described below. For example, you can execute the method to enumerate the file system before performing file transactions. Figure 4 and Figure 6 These steps are performed before each of the methods described in the document.

[0135] To understand the contents of the file system, a function is provided to "traverse" the file system by enumerating file system entries. The complete address space of the file system is read to understand the location, type, and / or status of files / entries within it. The enumeration results can then be cached to allow fast access to file system records without having to repeat the enumeration. For example, in the case where the file system of a smart card is initiated by the Hardware Security Module (HSM) communicating with it, the enumeration results can be stored in a cache within the HSM.

[0136] This disclosure refers to the enumeration function as the Tknfs_EnumEntries() function, which can then be configured to pass details of each file system entry to a user-specified callback function.

[0137] Typically, the Tknfs_EnumEntries() function is executed as follows:

[0138] 1. Set the address pointer to sizeof(FS_Header), that is, point to the space immediately following the general file system header;

[0139] 2. If the current address pointer is at or past the end of the file system storage space, then stop;

[0140] 3. The byte at the current address pointer is the Entry_Type value: if it is ETYPE_PADBYTE, skip one byte forward and repeat from (2);

[0141] 4. Otherwise (i.e., if the current byte is not ETYPE_PADBYTE), read the header of the current entry (depending on the Entry_Type value, which can be a file or a free space block) at the current address pointer;

[0142] 5. Inspect the entry header for a valid Entry_Type, and (unless the entry type is ETYPE_PADBYTE) check the length bytes from Entry_Len (or, if the ETYPE_UNCOMMITTED_LENGTH flag is set in Entry_Type, read the length from FS_Len2 in the filesystem header instead). If the Entry_Type value is not recognizable, or the length points beyond the end of storage: report an error and terminate the algorithm.

[0143] 6. If the entry is a file, then read Entry_NameLen;

[0144] 7. Then, read the next N bytes, where N is determined by the number of bytes specified in Entry_NameLen, where the N bytes include the file name Entry_Name;

[0145] 8. Pass the memory address pointer of Entry_Block, along with the value of its Entry_Type, and the applicable Entry_Len and Entry_Name, to the callback routine.

[0146] 9. Increment the address pointer by the length read in (5) (i.e., Entry_Len) and repeat from (2).

[0147] The method of enumerating the file system uses the embedded file length stored in the file header, rather than a separate table. This makes the file system and enumeration method particularly suitable for small-sized devices, as there is no need to store a separate table.

[0148] Transactions on the file system

[0149] Changes made to the file system are typically referred to as transactions. A transaction can include one or more operations selected from the following groups: for example, write, overwrite, and erase.

[0150] Here, rewriting is described as a specific type of write transaction operation. Therefore, when the term "write transaction operation" is used in the following description, it should be understood that this can include a rewrite transaction operation. However, it will be understood that "rewrite" as described below can include a write operation followed by erasing a (pre-existing) copy of the file to be rewritten. In this way, there is always a copy of the pre-existing file or the new file, allowing the file system to be safely rolled forward or backward to its expected final or initial state. In some examples (e.g., where there is not enough space to maintain redundant copies of file entries), the file to be rewritten may be erased first.

[0151] When a transaction is initiated (e.g., by an HSM), transaction instructions (by HSM 21) are provided, comprising one or more transaction objects as described below. Each transaction object specifies a transaction operation and, in the case of a write or rewrite, the corresponding information to be written. The data structures used for the transaction objects are described below. The object that collectively forms the transaction instructions is called a TKNFS_FILE_CONTENT object and is used to store information about file changes. The TKNFS_FILE_CONTENT object relates to transaction operations that write files to the device (i.e., when the TKNFS_OPERATION field specifies the type of the transaction operation as TKNFS_CREATE or TKNFS_OVERWRITE as described below) or transaction operations that erase files from the device (i.e., when TKNFS_ERASE is specified).

[0152] The alternative structures suitable for describing objects will be obvious to those skilled in the art. However, an example structure for a TKNFS_FILE_CONTENT object is shown below, comprising three parts:

[0153]

[0154] TKNFS_FILENAME corresponds to the identification information of the file to be written to or erased.

[0155] The procedure for executing a sequence of changes applied to the file system when executing a set of one or more transactions is performed as follows:

[0156] i. Set the FSTYPE_DIRTY flag in FS_Type to indicate that a transaction is in progress.

[0157] ii. Transaction changes are written as uncommitted.

[0158] →(Up to this point, after the transaction was interrupted, the repair action will cause the file system to be rolled back to its original state.)

[0159] iii. Set the FSTYPE_COMMITTED flag in FS_Type to indicate that the transaction has been committed (i.e., fully written).

[0160] →(From this point forward, after the transaction was interrupted, the repair action will cause the file system to be rolled forward to the expected final state.)

[0161] iv. Changes to transactions are rolled forward.

[0162] v. Reset the FSTYPE_DIRTY and FSTYPE_COMMITTED flags in FS_Type.

[0163] It will be understood that the FSTYPE_DIRTY flag set in i does not provide an indication of the progress of the write / erase operation; rather, it indicates that the transaction as a whole has begun but not yet completed. Including the FSTYPE_DIRTY flag allows the repair action to avoid performing subsequent checks (e.g., whether the transaction has been committed) before the file transaction has been interrupted. Therefore, if the FSTYPE_DIRTY flag is not set when the access operation begins, no further checks are required. In some alternative embodiments, the feature of the FSTYPE_DIRTY flag is not included.

[0164] Use the committed flag to provide a single check whether a transaction has been committed. Setting the committed flag corresponds to storing transaction information on the device that indicates the transaction has been committed. Alternatively, for example, this can be indicated by updating individual files to the committed type.

[0165] After step ii and before step iii, since the FSTYPE_DIRTY flag is in the generic filesystem header but the FSTYPE_COMMITTED flag is not set, the transaction will not be considered committed. Therefore, any recovery of the filesystem in this state will roll back the transaction.

[0166] Following step iii, any recovery of the file system in this state will roll forward the transaction, indicated by the FSTYPE_COMMITTED type in the generic file system header. Advantageously, in step v, both flags are reset atomically, i.e., as a single 1-byte write operation.

[0167] A write to the file system is not performed until all changes to the transaction are known. This can be achieved by writing to or updating the entire set of files using the APIs used by the file system implementation. More generally, a sequence of file transactions can be surrounded by BEGIN / COMMIT transaction wrappers, where all tracking of changes is done in main memory, and a write to the file system occurs only when the transaction is COMMITed, at which point all files to be updated will already be known.

[0168] The specific interfaces that can be used to interact with file system content are detailed below:

[0169] Tknfs_Transact:

[0170] This interface creates, rewrites, and / or erases one or more files as a single transaction. A single transaction instruction can include a transaction operation with each of the write, rewrite, and erase functions, so all three types of operations can be performed simultaneously and with transaction safety. Each of the write, rewrite, and erase operations will either roll forward (to the expected final state) or roll back (to the initial file system state) during a recovery action following a break in the file transaction.

[0171] Pass an array of TKNFS_FILE_CONTENT objects (as described in Table 5 above) to specify the parameters for each file. This array corresponds to the transaction instructions. Each object specifies the filename and flags to indicate any of the following:

[0172] – New file (TKNFS_CREATE); or

[0173] - A file that can overwrite an existing file with the same filename (TKNFS_OVERWRITE); or

[0174] - Files with that filename (TKNFS_ERASE) should be erased.

[0175] The file content included in the TKNFS_DATA (see Table 5 above) of a new or rewritten file is typically provided as opaque byte blocks. A specific user implementation (i.e., an implementation that can be further defined by the user of the file system) may optionally implement additional structures within these opaque byte blocks of the new / rewritten file.

[0176] Now regarding Figure 4 Describes a method for performing file transactions on a device. Figure 4 This is a flowchart illustrating a method for performing file transactions on a device according to an embodiment.

[0177] The HSM provides a set of transaction instructions for performing one or more transaction operations on the device. Transaction instructions can be generated at the HSM device 21 in response to a command received from the user. Each transaction instruction includes one or more transaction objects as described in Table 5 above, each specifying a transaction operation to be performed. These operations correspond to the operations specified in the user command. Various interfaces can be provided to generate transaction objects corresponding to the requests in the received user command, as will be apparent to those skilled in the art.

[0178] In S101, before making any other changes to the storage, update FS_Type to set the FSTYPE_DIRTY flag to indicate that a transaction is in progress.

[0179] Optionally, all file parameters are validated before any changes are made to the file system. This validation may include checking that there is enough space for all content. In other words, this includes determining whether the size of all files to be written exceeds the available storage on the device. This step may also include checking that no attempts are made to replace existing files unless the TKFNS_OVERWRITE flag is set.

[0180] In S102, the first transaction operation specified by the first transaction object in the transaction instruction is processed. The steps executed in response to determining in S102 that the transaction operation is a write operation are part of the complete process. Figure 4 As shown in the image. For clarity, in... Figure 6 The remaining steps of the method, which are more relevant to the "erase" transaction operation, are shown separately and in full (some of which are in...). Figure 4(Represented by dashed boxes). Write transaction operation objects have the structure shown in Table 5 above, where TKNFS_CREATE or TKNFS_OVERWRITE is specified. Each write transaction operation object in the transaction instruction corresponds to a file object. The "write" transaction operation objects in the transaction instruction collectively involve a first filegroup comprising at least one file object. The first filegroup includes all file objects to be written to the device, wherein the file objects in the first filegroup collectively have a first size, and wherein each file object includes identification information, i.e., a filename.

[0181] In S103, a search is performed in the file system for free space blocks with sufficient space to write the file object corresponding to the write transaction. To write (and rewrite) the file, Tknfs_EnumEntries() is first called to find appropriate records containing free space. In some examples, a full “traversal” of the file system may not be necessary; that is, the results of the previous file system enumeration can be stored in a cache for fast access. A best-fit algorithm is performed, in which the entire device storage space is searched. A free space block with the minimum amount of redundant space is selected; preferably, a free space block with a size that does not require writing (unusable) padding bytes is selected.

[0182] Optionally, any new file (i.e., any file in the file object group with the TKNFS_CREATE or TKNFS_OVERWRITE flags) is written to the gap between two existing files only if the file size exactly matches the gap size, or if the remaining space (between the file size match and the gap size) is at least large enough for a free space entry. In this way, padding bytes are avoided. Padding bytes are only written if there is no other available space. This has the advantage of ensuring that unusable and incompressible padding bytes are impossible to generate. Writing new files in this way makes it less likely that small, unusable spaces that cannot be safely compressed will appear.

[0183] If no such free space block exists, then the entire device is compressed in S104 (see below for details). Figures 5(a) to 5(c)(A detailed description of the compression algorithm follows.) In summary, compression involves moving files "backward" (i.e., towards an earlier address space) (starting at the beginning of a storage area) so that all gaps between existing entries are removed. For example, gaps between file entries can include free space or padding bytes. After the existing entries have been moved, a new free space record is created, containing all available storage in a single block. For example, if all existing files are moved backward to the "front" of the storage space, a single free space block will be created at the "end" of the storage space. File system compression occurs if a gap of appropriate size cannot be found between two existing files. Compression is generally preferred when writing new files in gaps larger than the size of the new file (but not in cases where the gap is too small for both the new file and the free space entry), as this avoids writing padding bytes.

[0184] After compression is complete, the search for the appropriate record containing free space is repeated in S105 (i.e., enumeration). The repetition of compression / enumeration ensures that file creation will always succeed if physically possible, i.e., if there is physically enough free space available.

[0185] If there is still not enough space after compression, a degradation process can be followed in S106. This is explained in more detail below. This may include erasing existing file entries to be rewritten before writing new files. This results in a degradation of transaction safety behavior. After erasing the files to be rewritten, the search for free space is repeated, and compression is performed again if necessary.

[0186] When allocation is successful, in S107, the file data specified in the TKNFS_FILE_CONTENT file object is written, and the header at the beginning of the free space record is overwritten to reflect the new file. The Entry_Type byte of the new file is written as ETYPE_UNCOMMITTED_FILE. If the allocated free space block is longer than required, a new free space record is created at the end of the new file. Optionally, if there is not enough space for the free space record, a padding byte block is written instead.

[0187] In S107, a new file object is created using ETYPE_UNCOMMITTED_FILE Entry_Type. If the TKNFS_OVERWRITE flag is set in the file object, the file object's name will be shared with the name of the original / pre-existing file in the file system. Original files with the same name are not modified during this stage.

[0188] In response to the fact that all new file entries have been written and the Entry_Type of all files to be erased has been updated to ETYPE_UNCOMMITTED_ERASE (as will be discussed below), Figure 6 As described in any transaction erasure process shown, the transaction is marked as committed by setting the FS_Type to the FSTYPE_COMMITTED flag in S109. This is an example of the information indicating that a transaction has been committed. If further transaction operations are to be processed, the method returns to S102 and processes the next transaction object.

[0189] In the case of rewriting the file, the original file with the same filename is then erased in S110 according to the secure erase algorithm (which will be described in further detail below).

[0190] In step S111, after the FSTYPE_COMMITTED flag is set in the generic file system header, the Entry_Type on the newly written file is changed from ETYPE_UNCOMMITTED_FILE to ETYPE_FILE. This is also referred to as the final type.

[0191] Finally, in S113, the FS_Type in the general file system header is updated to clear both the FSTYPE_COMMITTED and FSTYPE_DIRTY flags.

[0192] In this way, changes to the file system are made in such a way that if any I / O (e.g., network connection) fails, the file system can be restored to its original or expected state whenever possible.

[0193] File system recovery (below is about...) occurs when any read or write operation is attempted on the file system. Figure 7 (Detailed Description) This occurs automatically. If a write transaction was previously interrupted, when the file system is read again, it will be rolled back to the original state where there are no new files and the original version of the files has been rewritten, or rolled forward to the new state with all new or changed files. An exception to this is when there is insufficient space to make changes in a fully transaction-safe manner and the degradation process in S106 is used, in which case there may be a subset of new files, and the files to be rewritten may have old or new content or may not exist.

[0194] The filegroup is written to (or rewritten) as a single atomic transaction. Atomic transactions guarantee that updates do not occur only partially, thus ensuring that file transactions, as defined by transaction instructions, are executed to completion, or do not occur at all. In practice, in the event of a transaction interruption, the recovery / repair system rolls the filesystem forward or backward to its initial or expected final state.

[0195] Now regarding Figure 6 The method for performing file transactions on the device is further described. Figure 6 This is a flowchart illustrating a method for performing a file transaction on a device according to an embodiment. The step performed in response to determining in S102 that the transaction operation is an erase transaction operation is... Figure 6 As shown in the image.

[0196] For clarity, Figure 4 Steps S101 and S102 shown are also... Figure 6 As shown in the image. Then... Figure 6 The steps executed in response to determining that a transaction is an eraser are shown. Therefore, if the transaction is an eraser, step S108 is executed instead of S103. The eraser transaction has the structure shown in Table 5 above, where TKNFS_ERASE is specified. The eraser transaction corresponds to a file object. The eraser transaction operation objects in the transaction instruction collectively involve a second filegroup comprising at least one file object. The second filegroup includes all file objects to be erased from the device, wherein each file object includes identification information, i.e., a filename.

[0197] In S108, the type of the file to be erased is updated to ETYPE_UNCOMMITTED_ERASE Entry_Type. This is the second uncommitted type used for the file to be erased.

[0198] As mentioned earlier, in response to all new file entries being written and all Entry_Type entries for files to be erased being updated to ETYPE_UNCOMMITTED_ERASE, by... Figure 4 In step S109, setting FS_Type to the FSTYPE_COMMITTED flag marks the transaction instruction as committed. This step also... Figure 6 As shown in the image.

[0199] After the FSTYPE_COMMITTED flag is set, in S112, each ETYPE_UNCOMMITTED_ERASE file is erased sequentially according to the secure erase algorithm (described below).

[0200] Finally, update the FS_Type in the generic file system header to clear both the FSTYPE_COMMITTED and FSTYPE_DIRTY flags, as previously described in S113.

[0201] Although for the sake of simplicity, it has already been Figure 4 In order to complete write transactions and in Figure 6The method is illustrated with the complete steps of an erase transaction, but those skilled in the art will understand how to perform the method when the transaction instruction includes both erasing transaction operation objects and writing transaction operation objects. Specifically, it is clear from the above description that the commit flag is not set until all new file entries have been written as uncommitted and the entry type of all files to be erased has been set to uncommitted erase. Furthermore, it is clear that the commit flag and the transaction in progress flag are not cleared until all new files have been updated to final and all uncommitted erased files have been erased.

[0202] Any erased file is typically rewritten with zeros before being converted to free space. The space freed by this operation is merged with any free space before or after the file being erased. The callback to Tknfs_EnumEntries() records the first free space record found before the given file and the first non-free space record found after it. All storage between these two addresses is converted to a single free space record, thus aggregating adjacent blocks of free space.

[0203] If a file transaction is interrupted, when the file system is read again, it will be rolled back to the original state where the file existed, or rolled forward to the expected state where the file was removed. This will be explained below regarding... Figure 7 Describe it.

[0204] Optionally, whenever an Entry_Block is "erased," the following steps are taken to ensure that all previous data is safely removed without leaving any portion of the original data on storage:

[0205] i. Set the Entry_Type of Entry_Block to ETYPE_UNCOMMITTED_FREESPACE (this informs the file system recovery code that the original data may not have been completely overwritten with zeros);

[0206] ii. Rewrite all storage after the Entry_Len field in Entry_Block with zeros;

[0207] iii. If there is no ETYPE_FREESPACE entry before or after Entry_Block:

[0208] Then update Entry_Block's Entry_Type to ETYPE_FREESPACE;

[0209] Otherwise, if there is an ETYPE_FREESPACE entry before Entry_Block, but no ETYPE_FREESPACE entry after it:

[0210] The size of the preceding ETYPE_FREESPACE entry is then adjusted using the security header update algorithm (see “Update Entry Header Algorithm” below) to include both the entry being erased and the entry preceding it.

[0211] Otherwise, if there is an ETYPE_FREESPACE entry only after Entry_Block (i.e., not before it):

[0212] The size of the Entry_Block being erased is then adjusted using the security header update algorithm to include both the entry being erased and the entries following it, and then the type of the Entry_Block is changed to ETYPE_FREESPACE.

[0213] Otherwise, if there are ETYPE_FREESPACE entries before and after Entry_Block:

[0214] The size of the preceding ETYPE_FREESPACE entry is then adjusted using the security header update algorithm to include all three blocks.

[0215] `ETYPE_UNCOMMITTED_FREESPACE` ensures that free space previously used to store files is zeroed out during file system recovery (if it wasn't zeroed out due to interruption caused by changes). For example, if a file is logically erased but the original storage wasn't zeroed out, the data can still be recovered. This is handled during recovery when the file system is next mounted, ensuring no data leakage after erasure.

[0216] It will be understood that the erase operation can be applied to any of the following: erasing ETYPE_FILE as part of a Tknfs_Transact transaction; erasing any other Entry_Block, such as erasing an existing copy of the rewritten file when a transaction is committed; and erasing ETYPE_UNCOMMITTED_FILE when an uncommitted transaction is rolled back during a repair operation. Therefore, this procedure can be performed in S112 and S111 of the above method.

[0217] The following will describe what can be done. Figure 4 This is an example of a degraded write process with lower transaction safety executed in S106. In some cases, the space on the token device may not be sufficient for transactional changes to be performed in a fully transaction-safe manner. Therefore, the implementation can apply a degraded level of security. In such an example, the integrity of the file system is still preserved, ensuring that files that should be unaffected by changes are always retained without any corruption.

[0218] If there is not enough space to write to the next file in the group of that file object, the transaction algorithm is modified in S106 to downgrade the transaction safety to two lower levels according to the following procedure:

[0219] i. Attempt to make more storage available by "committing" the current state of the transaction:

[0220] a. Set the FSTYPE_COMMITTED flag in the FS_Type field of the general file system header;

[0221] b. Delete any original (i.e., pre-existing) file entries that have a duplicate filename identifier (if overridden) with any ETYPE_UNCOMMITTED_FILE entry;

[0222] c. Promote any ETYPE_UNCOMMITTED_FILE entries to ETYPE_FILE and update the ETYPE_UNCOMMITTED_ERASE file to completely erase them (wherein, the erasure is preferably performed according to a secure erasure algorithm).

[0223] Therefore, after a point in the transaction where there isn't enough space to write to the next file, effective transaction safety is downgraded to per file rather than per filegroup. In this scenario, the atomicity of the transaction as a whole may be lost.

[0224] ii. If the steps in i do not reclaim enough available storage (including optionally compressing the file system according to the compression algorithm), then optionally perform the following steps:

[0225] a. The next file to be written in the group of the file object rewrites the original (i.e., pre-existing) file with the same identifier, including erasing the original file according to the secure erase algorithm;

[0226] b. Write to the new file to be written.

[0227] Then, the file loses transaction safety in ii, although the integrity of other files will be preserved.

[0228] iii. After the completion of either of the intermediate security-degraded transactions in i or ii, and before continuing any further writes, reset the FSTYPE_COMMITTED flag in the generic file system header.

[0229] In the two degraded transaction safety levels described in i and ii above, the implementation can also pre-calculate whether there is enough space to complete the transaction before any changes are made to the storage. If there is still not enough available storage, the entire transaction is considered to have failed.

[0230] Tknfs_CreateFile

[0231] This interface is a special case of Tknfs_Transact() used for writing or rewriting a single file. The behavior of this function appears to be that Tknfs_Transact() is called with a single TKNFS_FILE_CONTENT object that has the TKNFS_CREATE or TKNFS_OVERWRITE flags set.

[0232] Tknfs_EraseFile

[0233] This interface is a special case of Tknfs_Transact() used to erase a single file. Its behavior appears to be that Tknfs_Transact() is invoked with a single TKNFS_FILE_CONTENT file object containing the filename and the TKNFS ERASE flag.

[0234] The various additional methods that can be executed on the file system will now be described.

[0235] Tknfs_Format

[0236] This interface first passes any device-specific "low-level formatting" request to initialize the device, zeroing out the storage, then writes the file system header and the beginning of the storage space, followed by a single free space record spanning the remaining storage. Performing the formatting operation ensures that all existing data is zeroed out, not just the header is updated.

[0237] The flag "FSTYPE_FORMATTED" is set to zero at the start of the formatting process. If the formatting operation is interrupted before completion, the file system will not be recognized as a properly formatted file system when it is read again, and therefore will have to be reformatted. This allows malicious users attempting to interrupt the formatting process to lose access to the file system's contents.

[0238] The Tknfs_Format function executes as follows:

[0239] i. Set FS_Type to 0 to indicate that the file system is not formatted correctly (in addition, this indicates that the system is invalid, making the file system unreadable during the formatting operation).

[0240] ii. Zero out the entire storage space of the file system (the choice of zero is arbitrary; alternatively, the file system can be filled with 1s).

[0241] iii. Create an ETYPE_FREESPACE entry at the address sizeof(FS_Header) that spans the entire remaining address space, i.e., Entry_Len will be STORAGE_SIZE - sizeof(FS_Header).

[0242] iv. Set FS_Type to FSTYPE_FORMATTED to indicate that the file system has a valid format.

[0243] If formatting is interrupted, the file system becomes unreadable. Therefore, interrupted formatting is detectable, and the file system is considered unformatted. The storage will have to be reformatted manually before it becomes readable.

[0244] Alternatively, as part of the recovery process, the unformatted storage can be automatically formatted upon the first attempt to read / write to it. As another alternative, an explicit formatting operation may be required first as part of the recovery process for the unformatted storage. The aim is to ensure that even if formatting is interrupted, all old data is completely erased, and not merely rendered "invisible" due to updates to the file system header.

[0245] The formatting operation essentially creates blocks of the full length of the file system. If a valid file system previously existed, the formatting operation will transform it into zeroed-out free space as a single transaction.

[0246] Tknfs_ListFiles

[0247] This interface lists all files on the device by name. This can be achieved using Tknfs_EnumEntries() as described earlier.

[0248] Tknfs_ReadFile

[0249] This interface reads the contents of a file with a user-specified filename, or returns an error if the file does not exist. This can be achieved using Tknfs_EnumEntries(), which searches for a file entry with the specified name and then reads the Entry_Content from storage at the offset from the Entry_Block address pointer.

[0250] Tknfs_UpdateFile

[0251] This function selectively updates a portion of an existing file entry from the provided buffer; given the filename (along with flags), and the byte offset and length within the file. The byte offset describes the point in the file where the selective update begins, while the length describes the size of the portion of the file to be updated. If the call is interrupted, when the filesystem is read again, it will be rolled back to the file's original state or rolled forward to the file's new expected state. Alternatively, the file may not exist if there is insufficient space on the device memory to perform a transaction-safe operation. File non-existence may result from an attempt to update the file during a lower level of consistency; for example, if the implementation drops to a lower level of transactional consistency and the original file is forcibly deleted before a new (updated) file is written because there is insufficient space for two copies.

[0252] The Tknfs_UpdateFile function executes as follows:

[0253] This implementation holds a lock on the storage for the duration of both read and write operations. This means that updates are performed atomically, reading the current file and writing it to the updated file as a rewrite operation, without allowing any other code or device to interfere with the write (if interference were allowed, some other executing thread could modify the file being updated, potentially modifying a portion of the file other than the one intended to be updated). The update interface is described as follows:

[0254] i. Read the existing file into main memory (HSM's RAM), for example, using Tknfs_ReadFile();

[0255] ii. Virtually apply the update operation in main memory so that the entire new file is stored in main memory (i.e., allow roll-forward in the event of an interrupt);

[0256] iii. This change can be achieved by calling Tknfs_CreateFile() with the TKNFS_OVERWRITE flag.

[0257] Because the provided buffer (as created in step ii) is created from outside the file system and contains the entire updated version of the file, at any given time, there is at least one copy of either the original file or the updated file (or both). Therefore, transaction safety is guaranteed, as either a rollback or a rollforward is always possible.

[0258] Update entry header

[0259] The headers of individual entries can be modified in a variety of different scenarios. The following is just a specific example of updating the header, where the order of the update operations is chosen so that the update as a whole is atomic.

[0260] When an existing Entry_Block is split into two blocks, the header is updated. This occurs, for example, when an ETYPE_FREESPACE entry is split into a new file entry and a smaller ETYPE_FREESPACE block. Therefore, a header update occurs, for example, in S107 of the method described above. The header is updated as an initial step before writing the file content.

[0261] In this example, a new Entry_Block header will first be written at a higher address, making it "invisible" when navigating the file system, and will only be visible when the size of an existing Entry_Block is adjusted (i.e., its size is reduced).

[0262] Generally, whenever a "visible" file header (i.e., the file header required by the currently navigating file system, not the file header written within an existing Entry_Block) is modified, it must be ensured that no intermediate state would be irreparable. Entry_Len is typically at least two bytes long, therefore it cannot be reliably updated atomically at the I / O level. This is because only one byte of the length might be written, and no assumptions are made about simultaneous byte writes. Therefore, the following procedure is followed:

[0263] i. Before the Entry_Block length (Entry_Len) value is updated, the safe update of Entry_Len is achieved by initially storing the new length in FS_Len2 in the file system header;

[0264] ii. Then set the Entry_Type bit flag ETYPE_UNCOMMITTED_LENGTH, which indicates that the implementation should look at the length in FS_Len2 instead of Entry_Len;

[0265] iii. Update the Entry_Len position with the new length. If this fails before all bytes of the length are written, the correct length can still be read from the FS_Len2 position;

[0266] iv. Update Entry_Type again so that the ETYPE_UNCOMMITTED_LENGTH flag is no longer set.

[0267] Setting the "uncommitted" length flag in step ii is atomic because Entry_Type is a single byte, so it can be reliably updated in a single write operation (depending on file system prerequisites).

[0268] In practice, only the sizes of the ETYPE_UNCOMMITTED_FREESPACE, ETYPE_FREESPACE, and ETYPE_FRAGMENT entries need to be adjusted. Therefore, the recovery code can optionally force the ETYPE_UNCOMMITTED_LENGTH flag to be set only on these three entries, otherwise an error will be reported.

[0269] If a transaction is interrupted, and the ETYPE_UNCOMMITTED_LENGTH flag is present in the Entry_Type of any entry, then Entry_Len is updated with the length read from FS_Len2, and the ETYPE_UNCOMMITTED_LENGTH flag is subsequently reset. This allows for forward roll-forward file header updates.

[0270] Compression algorithm

[0271] As described above, for example with S104, a compression process can be performed on the file system. An example compression process will now be described.

[0272] The FS_FragmentIndex from the file system header described in Table 1a above is used during the compression process. This stores additional trace information about the contents of an Entry_Block of type ETYPE_FRAGMENT when the file system is compressed. The FS_FragmentIndex is stored in the generic file system header to ensure that there is always space on the device to store the fragment information. This ensures that there is always enough space for the compression process. An example structure of FragmentIndex is shown in Table 6 below.

[0273]

[0274]

[0275] When creating or updating files, if there isn't enough space to write the new file as a single contiguous chunk, the code can compress the file system (moving files to lower addresses to remove gaps of free space between them). The fault safety of the compression algorithm is achieved in the following ways:

[0276] Figures 5(a) to 5(c) The following examples illustrate a series of states of the file system 200 during a compression process according to the following algorithm.

[0277] Figure 5(a) shows the initial state of the example file system 200. The integers of the file system 200 are as follows: beginning / start of the file system's address space 202, general file system header 204, a first file named file A 206a, a first free space block 208a, a second file named file B 206b, a second free space block 208b, and a third file named file C 206c. The length / size of the first free space block 208a is indicated by 'Off'. Each block (file and free space) contains a header containing information about the block length. The length / size of the second file, which is a file moved during compression, is indicated by 'L'.

[0278] F1 represents the address where the file data (i.e., file B) is moved, and F2 represents the address from which the file data is moved. Therefore, the identity F2 = F1 + Off holds true, where Off is the original start of the file data relative to the new start of the file data.

[0279] Figure 5(b) illustrates the state of file system 200 during compression. M represents the length of the file being moved (file B) that has been moved to the new location so far. Block 210a represents the portion of file B that has been moved to position F1 so far, and block 210b represents the remaining portion of file B that has not yet been moved. During compression, the header of the first free space block 208a (at F1) is changed such that the total length 212 stored covers the entire space spanned from F1 to the end of the original file B (corresponding to the header in the second free space block 208b). Additionally, the type of intermediate block 212 is changed to 'ETYPE_FRAGMENT' in the header at F1.

[0280] The intermediate state of compression is tracked in the FS_FragmentIndex in the file system header, which is initialized to all zeros before moving each file. The Entry_Type of the moved file is stored in FS_FragCode, and the offset (Off) of the original start of the file data relative to the new start of the file data is stored in FS_FragOffset. Depending on whether the ETYPE_FRAGMOVED1 flag is set, the length (M) of the data that has been successfully moved so far is alternately stored in FS_FragMoved0 and FS_FragMoved1. Alternating the positions of the length storage allows it to be updated atomically.

[0281] Following step 3(i) below, a maximum of MIN(Off, LM) bytes of file B are written at a time. In the example shown in the figure, in the first step, this length corresponds to the length Off of block 210a, because... Figures 5(a) to 5(c)In the example, the file length (L) of B is greater than the size of the free space block 208a in which it is moved.

[0282] In the event of an interruption, file data can be recovered by reading addresses from F1 to F1+M-1 (corresponding to the portion of the file that has been moved) and from F2+M to F2+L-1 (corresponding to the remaining portion of the original file that has not yet been moved). Inclusion counting applies here. This provides improved fault safety for the compressed file system, where any single byte write may fail, but the file system can be recovered.

[0283] Figure 5(c) shows the final state of the file after compression of file B. Therefore, a new free space block 214a is created after file B, and its length 214b is equal to the sum of the lengths of the original free space blocks 208a and 208b.

[0284] To maintain a valid write order for FS_FragmentIndex, follow these steps:

[0285] 1. Read the entire existing file that has been moved to main memory (RAM of HSM), and initialize M to 0 in FS_FragMoved0;

[0286] 2. Update the header of the lowest address free space block to ETYPE_FRAGMENT, and its own length (updated as usual with ETYPE_UNCOMMITTED_LENGTH as an intermediate state) points to the next header after the file is moved (that is, the fragment will represent the entire address range containing both the original file and the free space in which it was moved).

[0287] 3. The following loop continues until M == L:

[0288] i. Write the next MIN(Off, LM) bytes to the file starting at address F1+M;

[0289] ii. Update M in FS_FragmentIndex to reflect the data copied atomically, as follows:

[0290] a. If the previous M value was stored in FS_FragMoved0, then write the new M value to FS_FragMoved1, and then set the ETYPE_FRAGMENT Entry_Type flag to ETYPE_FRAGMOVED1.

[0291] b. If the previous M value is stored in FS_FragMoved1, then write the new M value to FS_FragMoved0 and clear the ETYPE_FRAGMOVED1 flag on ETYPE_FRAGMENT Entry_Type.

[0292] iii. Optionally: If you want to perform storage-specific anti-residual erase operations instead of rewriting or zeroing out, then apply these anti-residual erase operations to the storage portion from which file bytes have just been copied at this time.

[0293] 4. When the file move is complete:

[0294] i. Write a new ETYPE_FREESPACE entry at F1+L;

[0295] Optionally: Rewrite the remaining space in ETYPE_FRAGMENT after the ETYPE_FREESPACE block header with zeros (i.e., F1+L+1+LEN_SIZE to F2+L-1, including end values);

[0296] ii. Adjust the size of ETYPE_FRAGMENT to the actual file length (using ETYPE_UNCOMMITTED_LENGTH as an intermediate state);

[0297] iii. Update the Entry_Type of ETYPE_FRAGMENT to the original Entry_Type of the entry read from FS_FragCode.

[0298] According to step 3(i) above, writing at most Off bytes at a time means that the entire file data always exists on the token. The benefit of doing this is that the beginning of data that has not yet been moved will never be overwritten by data that has already been moved.

[0299] Specifically, if free space 208a has a length of 40 and File_B has a length of 60, then File_B is larger than the free space block it is to move into. This means that if all of File_B is written starting at the beginning of free space 208a, it will also overwrite the beginning of the original File_B data. This means that if the write is interrupted after some original File_B data has been overwritten but before updating the fragment index to indicate how far, the beginning of the File_B data will be corrupted because the recovery operation cannot know how much File_B has been moved so far. Therefore, File_B is segmented into chunks so that in the event that compression is interrupted and needs to be recovered, the move will never overwrite data that would be considered part of the file. If there is enough free space to move the entire file at once, then the algorithm will do this: when M is initially 0 and the offset is greater than or equal to L, the minimum value of the offset sum (LM) will be exactly L.

[0300] Corresponding to Figure 5(c), block 214a is a new ETYPE_FREESPACE entry that begins at F1+L. The size of the block corresponding to segment 212 in Figure 5(b) is adjusted to the correct file length (L) in step 4(ii).

[0301] Optionally, in step 4(i), the remaining space in ETYPE_FRAGMENT is rewritten to address the possibility that remnants of files moved during compression and subsequently erased may remain in storage. By including this step, the compression algorithm ensures that the file is zeroed out of the memory it was moved from.

[0302] Step 3(iii) may optionally be included to address the possibility that remnants of files that were moved during compression and subsequently zeroed out or overwritten may remain on storage. Including this step ensures that any applicable storage-specific remanent magnetization resistance measures are provided to the storage from which the files were moved.

[0303] File system check and recovery

[0304] When an access operation is performed on the device, a file system check and recovery process is executed first. Typically, a file system check is performed automatically before any read or write operation is performed on the file system. This is to determine if a transaction interruption has occurred that left the file system in an intermediate state.

[0305] Figure 7 A method for performing an access operation on a device according to an embodiment is shown.

[0306] The checks were performed as follows:

[0307] i. FS_Type is read and checked to be a valid value; if FS_Type is not a valid value, an error is returned (as described above, during formatting, FS_Type is actively zeroed to indicate that the file system is invalid—once formatting is complete, this is updated to indicate that the file system is valid).

[0308] ii. If the FSTYPE_DIRTY flag is not set, the check can now be terminated. Optionally, the implementation can continue performing integrity checks to verify that all entries in the file system contain a valid Entry_Type value, and a valid embedding length in Entry_Len and Entry_NameLen. Therefore, in S701, it is determined whether the FSTYPE_DIRTY flag is set.

[0309] iii. Examine the header of each file entry to identify any Entry_Type of ETYPE_FRAGMENT, ETYPE_UNCOMMITTED_FREESPACE, ETYPE_UNCOMMITTED_FILE, or ETYPE_UNCOMMITTED_ERASE, and any ETYPE_UNCOMMITTED_LENGTH flags. Therefore, in S702, it is determined whether any file entry with the Uncommitted File type (ETYPE_UNCOMMITTED_FILE) exists. It is also determined whether any file entry with the types ETYPE_FRAGMENT, ETYPE_UNCOMMITTED_FREESPACE, ETYPE_UNCOMMITTED_FILE, ETYPE_UNCOMMITTED_ERASE, or ETYPE_UNCOMMITTED_LENGTH exists.

[0310] The repair function is performed in the following ways:

[0311] i. If the ETYPE_UNCOMMITTED_LENGTH flag exists in the Entry_Type of any entry, then update Entry_Len with the length read from FS_Len2, and then reset the ETYPE_UNCOMMITTED_LENGTH flag.

[0312] ii. If ETYPE_FRAGMENT is found, and M (the number of data moved) in FS_FragMoved0 (or FS_FragMoved1 if the ETYPE_FRAGMOVED1 flag is set) is non-zero, then the file compression process is complete.

[0313] iii. If FSTYPE_COMMITTED is set in FS_Type, the commit process will proceed. forward roll The transaction is complete. Therefore, in S703, it is determined whether the FSTYPE_COMMITTED flag is set in FS_Type. The transaction roll-forward is performed in S704 as follows:

[0314] a. Erase any ETYPE_FILE entries whose filenames match the filenames of ETYPE_UNCOMMITTED_FILE entries according to the secure erase algorithm;

[0315] b. Update any ETYPE_UNCOMMITTED_FILE Entry_Type entries to ETYPE_FILE;

[0316] c. Erase any ETYPE_UNCOMMITTED_ERASE files according to the secure erase algorithm.

[0317] iv. If FSTYPE_COMMITTED is not set in FS_Type, the transaction will be rolled back in S705 as follows:

[0318] a. Any ETYPE_UNCOMMITTED_FILE entries will be erased according to the secure erase algorithm;

[0319] b. Any ETYPE_UNCOMMITTED_ERASE entries will be restored to ETYPE_FILE.

[0320] v. Erase any ETYPE_UNCOMMITTED_FREESPACE entries according to the secure erase algorithm.

[0321] In the compression process, in step ii above, F1 is the start of the data portion of the ETYPE_FRAGMENT file, L (original file length) is the data length of the fragment block minus Off, and F2 is F1 + Off.

[0322] Recovery from the fault during the recovery period is also provided by the methods described above.

[0323] In the above embodiments, various example methods have been described. However, these are presented only by way of example, and those skilled in the art will understand that various omissions, substitutions, and changes can be made to the methods.

[0324] For example, in the description above, the file system includes a file system header, as shown in Table 1a. The file system header stores various flags. However, the file system header can be omitted. Flags indicating that a transaction is in progress can be omitted. In this case, checking this flag is simply omitted during the recovery phase. Furthermore, flags indicating whether the file system has been formatted can also be omitted. Again, this check is simply omitted during the recovery phase.

[0325] Instead of the flag in the file system header used to indicate that a transaction is committed, a committed file type can be introduced to indicate this information. Specifically, the ETYPE_COMMITTED_FILE type can be introduced to mark a write transaction operation as committed. During a write operation, when all new files have been written to an uncommitted type, the header type code for each new file is changed to ETYPE_COMMITTED_FILE in S109. Then, all original files with the same ID are erased, and the type code on the new files is now changed to ETYPE_FILE. During the repair action, if any ETYPE_COMMITTED_FILE is found, the commit process is completed. If an ETYPE_UNCOMMITTED_FILE file exists, it is assumed that the interruption occurred after all new files were written (because there are files with the COMMITTED type code set), and therefore all uncommitted files are changed to ETYPE_COMMITTED_FILE. Then, any ETYPE_FILE with the same ID as any ETYPE_COMMITTED_FILE is erased. Then, update any ETYPE_COMMITTED_FILE type codes to ETYPE_FILE type codes. If a COMMITTED file is not found during the repair action, simply erase ETYPE_UNCOMMITTED_FILE.

[0326] File erasure can be performed by (effectively) performing file write steps backward. This means that transaction operations with new / updated files must be performed separately from transaction operations with erasure, allowing them to be distinguished for recovery purposes in the event of a rolled-back or rolled-forward interrupted transaction. First, all ETYPE_FILE type codes of the files to be erased are updated to ETYPE_COMMITTED_FILE. Then, each of these files is downgraded to ETYPE_UNCOMMITTED_FILE. Finally, all uncommitted files are erased. This is essentially the reverse of what write operations do when committing the set of files and shares the same recovery code.

[0327] An ETYPE_FRAGMENT_INDEX entry can be created on the fly to record the state of the compression operation; that is, to store the compression information described above that is stored in the file system header. Free space entries can have space reserved for spare length in their headers, and the spare length of a fragment can be stored in the fragment index.

[0328] The above description outlines various file types. These file types are not exhaustive and may include additional file types. For example, separate private files and user files may be included, where private files include files that can only be read internally by the HSM and are not visible to users, and user files include files that can be read by users of the HSM and are subject to appropriate access control. Alternatively, this can be achieved in variable-length filenames with appropriate naming conventions, rather than including separate file types.

[0329] The concept of persistent and non-persistent file types can be introduced using the ETYPE_PERSIST flag on Entry_Type (alternatively, this distinction can be encoded into the filename if needed). An operation to delete all non-persistent files as a single transaction can be included. This call will fail if the file system is invalid. If this call is interrupted, it will be rolled back to the initial state where all non-persistent files and persistent files still exist, or rolled forward to the expected state where only persistent files exist, when the file system is read again. This operation erases all files that do not have the ETYPE_PERSIST flag set. First, the ETYPE_FILE type code for all non-persistent files is updated to ETYPE_COMMITTED_FILE. Then, each of these files is downgraded to ETYPE_UNCOMMITTED_FILE. Finally, all uncommitted files are erased.

[0330] Although some embodiments have been described, these embodiments are presented by way of example only and are not intended to limit the scope of the invention. Indeed, the novel methods and apparatus described herein can be modified in various other forms; furthermore, various omissions, substitutions, and changes can be made to the form of the methods and apparatus described herein.

Claims

1. A method for performing file transactions, the method comprising: Provides a set of transaction instructions to perform one or more transaction operations on the device; In response to determining that the transaction instruction includes one or more write transaction operations, wherein the one or more write transaction operations collectively involve a first filegroup comprising at least one file object, the first filegroup having a first size, and each of the at least one file object including identification information, if the first size does not exceed the available device... For memory, then: Write each of the at least one file object to the uncommitted file type, wherein writing the file object to the uncommitted file type includes: The length of the file object is stored in the first location; Update the entry type of the free space block to indicate the length of the uncommitted file entry; Update the header of the free space block with the length of the file object; Update the entry type of the free space block to indicate the uncommitted file type; and Write data from the file object into the free space block; After the first file group is written, transaction information indicating that the transaction has been committed is stored in the device memory; In response to determining that one or more pre-existing file objects share identification information with any file object in the first filegroup, the one or more pre-existing file objects are erased; and After erasing the one or more pre-existing file objects, the type of each file object in the first file group is updated to the final type.

2. The method according to claim 1, wherein, The method of storing transaction information indicating that a transaction has been committed includes setting a committed flag, and further includes clearing the committed flag after updating the type of each of the at least one file object to the final type.

3. The method according to claim 1, wherein, The transaction information indicating that a transaction has been committed includes updating the type of each of the at least one file objects to a committed type.

4. The method according to any one of the preceding claims, wherein, Erasing one or more pre-existing file objects includes: Update the entry type of the pre-existing file object to indicate an uncommitted free space type; Rewrite the pre-existing file object with zeros; and Update the entry type to indicate the free space type.

5. The method according to any one of claims 1-3, wherein, Writing to a file object with an uncommitted file type includes: searching for any free space block with a size greater than or equal to that of the file object; and in response to finding two or more free space blocks, selecting the free space block with the minimum excess available space, which, when written, will not cause the remaining space block to fall below the minimum size.

6. The method of claim 5, further comprising, in response to no free space block being found, performing a compression method, the compression method comprising: Move a pre-existing file object of length L; The storage contains information about the entry type of the entry being moved, the offset of the original start F2 of the entry data relative to the new start F1, and the length M of the data that has been successfully moved, where M is initially set to zero; where F1 represents the address to which the file object is moved and F2 represents the address from which the file object is moved. Moving the pre-existing file includes: i. Write the next X bytes of the entry starting at address F1+M, where X is the minimum value selected from the following: the offset and (LM); ii. Update the value of M; iii. Set M = M+X and repeat steps i to iii until M equals L.

7. The method according to any one of claims 1-3, further comprising: The first step is to determine whether there is enough space to write to the file object. In response to determining that there is sufficient space to write to the file object, the file object is written; In response to determining that there is not enough space to write to the file object: The device stores transaction information indicating that a transaction has been committed; Determine whether any pre-existing file object shares identification information with any file object in the first file group that has been written with the Uncommitted file type, and in response to determining that one or more pre-existing file objects share identification information, erase the one or more pre-existing file objects; A second determination is made to determine whether there is enough space to write to the file object. In response to determining in the second determination that there is sufficient space to write to the file object, the file object is written.

8. The method according to claim 7, further comprising: In response to determining in the second determination that there is not enough space to write the file object: Determine whether any pre-existing file object shares identification information with the file object, and in response to determining that the pre-existing file object shares identification information, erase the pre-existing file object; Write to the file object.

9. The method according to any one of claims 1-3, further comprising: In response to determining that the transaction instruction includes one or more erase transaction operations, the one or more erase transaction operations collectively involving a second file group to be erased from the device memory, the second file group including at least one file object, each of the at least one file object in the second file group is updated to an uncommitted erase type; as well as After the second file group is updated to the uncommitted erase type, transaction information indicating that the transaction has been committed is stored on the device; Erase the first file group.

10. A method for performing an access operation, the method comprising: Determine whether the device stores transaction information indicating that a transaction has been committed; Determine if any file objects of the Uncommitted file type exist; In response to determining that the device stores transaction information indicating that a transaction has been committed and that one or more file objects of the uncommitted file type exist: In response to determining that one or more pre-existing file objects share identification information with any of the one or more files having the Uncommitted file type, erase the one or more pre-existing file objects; as well as Update the type of the one or more file objects with the Uncommitted File type to the final type; In response to determining that the device does not store transaction information indicating that a transaction has been committed and that there are one or more file objects with the uncommitted file type, the file with the uncommitted file type is erased; as well as In response to determining that there is an entry type that includes an indication of the length of an unsubmitted document entry, the header of the entry is updated with the length stored in the first location and the entry type is updated.

11. A non-transitory data carrier carrying processor control code to cause a computer to perform the method according to any one of claims 1 to 10 at runtime.

12. An apparatus comprising: One or more processors are configured as follows: A set of transaction instructions to perform one or more transaction operations on a device memory located on the device or another device; In response to determining that the transaction instruction includes one or more write transaction operations, wherein the one or more write transaction operations collectively involve a first filegroup comprising at least one file object, the first filegroup having a first size, and each of the at least one file object including identification information, if the first size does not exceed the available device memory, then: Write each of the at least one file object to the uncommitted file type, wherein writing the file object to the uncommitted file type includes: The length of the file object is stored in the first location; Update the entry type of the free space block to indicate the length of the uncommitted file entry; Update the header of the free space block with the length of the file object; Update the entry type of the free space block to indicate the uncommitted file type; and Write data from the file object into the free space block; After the first file group is written, transaction information indicating that the transaction has been committed is stored in the device memory; In response to determining that one or more pre-existing file objects share identification information with any file object in the first filegroup, the one or more pre-existing file objects are erased; and After erasing the one or more pre-existing file objects, the type of each file object in the first file group is updated to the final type.

13. An apparatus comprising: One or more processors are configured as follows: Determine whether the device memory stores transaction information indicating that a transaction has been committed, wherein the device memory is located on the device or on another device; Determine whether any file objects with an unsubmitted file type exist in the device's memory; In response to determining that the device memory stores transaction information indicating that a transaction has been committed and that one or more file objects of the uncommitted file type exist: In response to determining that one or more pre-existing file objects share identification information with any of the one or more files having the Uncommitted file type, the one or more pre-existing file objects are erased; and Update the type of the one or more file objects with the Uncommitted File type to the final type; In response to determining that the device memory does not store transaction information indicating that a transaction has been committed and that one or more file objects of the uncommitted file type exist, the files of the uncommitted file type are erased; and In response to determining that there is an entry type that includes an indication of the length of an unsubmitted document entry, the header of the entry is updated with the length stored in the first location and the entry type is updated.

14. A system comprising the device according to claim 12 or 13 and another device, wherein, The device memory is located on the other device.

Images