Method and apparatus for modifying a file
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HUAWEI TECH CO LTD
- Filing Date
- 2021-04-09
- Publication Date
- 2026-06-05
Smart Images
Figure CN115203148B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer storage, and more particularly to a method and apparatus for modifying files. Background Technology
[0002] Many file systems offer compression features to compress file data before storing it on storage devices. These file systems perform full compression on user data, which means splitting the complete file into multiple parts, compressing each part into a single compressed unit. The compressed units and their data structures are stored independently in different locations on the storage device, and the file system records the indexes of all compressed units corresponding to each file within the data structure.
[0003] When a file system needs to modify the content of a compressed file, it locates the corresponding compressed unit based on the index of the specific compressed unit being modified, decompresses the entire compressed unit, recompresses it after modification, and writes it back to the storage device. However, full compression cannot accurately predict the specific location of the user's modification request within the compressed unit. Therefore, it needs to read the entire compressed unit from the storage device, modify it, recompress it, and write it back. But on various storage devices such as flash memory, the write endurance of the storage medium is limited. Excessive file read / write operations can degrade the lifespan of the storage device, reducing its lifespan and user experience.
[0004] Therefore, there is an urgent need to provide a file storage method to solve the problem of excessive read / write operations affecting the lifespan of storage devices. Summary of the Invention
[0005] This application provides a method for modifying files, which helps to reduce the number of read and write operations on the storage device, thereby reducing the wear and tear on the storage device caused by read and write operations.
[0006] In a first aspect, a method for modifying a file is provided, comprising: determining the data to be modified in the file based on a file modification request; obtaining differential data based on the data to be modified and the baseline data of the file when the data volume of the data to be modified is less than or equal to a first threshold; compressing the differential data to obtain differential compressed data; and storing the differential compressed data in a first storage space when the data volume of the differential compressed data is less than or equal to a second threshold.
[0007] In this embodiment, the storage device obtains differential data using the data to be modified and the baseline data of the file, and compresses this differential data to obtain differential compressed data. The storage device only needs to store this differential compressed data in the first storage space and does not need to modify the file immediately. Subsequently, it can accurately locate the position where the decompressed data needs to be modified based on the file offset address of the differential compressed data and update the compressed data as needed. The file modification method of this embodiment helps to reduce the number of read and write operations on the storage medium, thereby reducing the lifespan wear of the storage device caused by read and write operations. Moreover, during the modification process, the storage device can avoid the problem of file read and write amplification and reduce the latency of file read and write operations.
[0008] In conjunction with the first aspect, in some implementations of the first aspect, before obtaining the differential data based on the aforementioned data to be modified and the aforementioned file's baseline data, the method further includes: obtaining the baseline data of the aforementioned file from a cached baseline data page linked list, wherein the aforementioned baseline data page linked list includes multiple baseline data pages sorted according to access time.
[0009] In this embodiment, the storage device sets up a reference data page list in the page cache of the operating system. The reference data is cached in the reference data page list in the order of access time. This avoids the need to read the reference data from the storage medium and compare it with the data to be modified to obtain the differential data every time the file is modified. Therefore, the file modification method in this embodiment helps to reduce the number of read and write operations on the storage medium, thereby reducing the lifespan loss of the storage device caused by read and write operations.
[0010] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: determining the file offset address of the differential data and the index of the original compressed data corresponding to the differential data; and storing the file offset address and the index of the original compressed data in the first storage space.
[0011] In conjunction with the first aspect, in some implementations of the first aspect, the method includes: determining the original compressed data of the file stored in the second storage space based on the index of the original compressed data, wherein the second storage space is different from the first storage space; decompressing the original compressed data to obtain decompressed data; modifying the decompressed data based on the differential compressed data and the file offset address to obtain updated data; compressing the updated data and storing the new compressed data in the second storage space.
[0012] In this embodiment, the storage device can accurately locate the position where the decompressed data of the file needs to be modified based on differential compressed data and file offset address, without having to read the entire or even multiple compressed data units corresponding to the compressed data of the file. This is beneficial to improving the efficiency and accuracy of file modification and avoids the problem of file read / write amplification.
[0013] In conjunction with the first aspect, in some implementations of the first aspect, before determining the original compressed data of the file stored in the second storage space based on the index of the original compressed data, the method further includes: determining whether differential compressed data of the file exists in the first storage space based on the file read request of the file; determining the original compressed data of the file stored in the second storage space based on the index of the original compressed data includes: if the differential compressed data exists in the first storage space, determining the original compressed data based on the index of the original compressed data; the method further includes: reading the updated data of the file.
[0014] In this embodiment, after storing the differentially compressed data in the first storage space, the storage device does not need to update the base data, nor does it need to modify the file multiple times, resulting in multiple reads of the storage space. When a file read request is received, the storage device can modify the original compressed data of the file based on the differentially compressed data. This helps reduce the number of read / write operations on the storage medium, thereby reducing the lifespan wear and tear on the storage device. Furthermore, during the modification process, the storage device can accurately locate the position where the decompressed data needs to be modified based on the file offset address, avoiding the problem of file read / write amplification and reducing file read / write latency.
[0015] In conjunction with the first aspect, in some implementations of the first aspect, before determining the original compressed data of the file stored in the second storage space based on the index of the original compressed data, the method further includes: after storing the differential compressed data in the first storage space, starting a timer; determining the original compressed file data of the file stored in the second storage space based on the index of the original compressed data includes: when the timer times out, determining the original compressed data based on the index of the original compressed data.
[0016] In this embodiment, after storing the differentially compressed data in the first storage space, the storage device can periodically modify the original compressed data of multiple files based on the differentially compressed data. During this process, the storage device does not need to update the base data, nor does it need to modify the files multiple times, resulting in multiple reads of the storage space. This reduces the latency of file read and write operations and helps to reduce the number of read and write operations on the storage medium, thereby reducing the wear and tear on the storage device caused by read and write operations. Furthermore, during the modification process, the storage device can accurately locate the places where the decompressed data needs to be modified based on the file offset address of the differentially compressed data, avoiding the problem of file read and write amplification and reducing the latency of file read and write operations.
[0017] In conjunction with the first aspect, in some implementations of the first aspect, the above method further includes: deleting the aforementioned differential data, the file offset address of the aforementioned differential data, and the index of the original compressed file data.
[0018] In this embodiment of the application, after modifying the decompressed data, the storage device can delete the differential data, the file offset address of the differential data, and the index of the original compressed file data to release the corresponding space and reduce the load on the storage space.
[0019] In a second aspect, an apparatus for modifying a file is provided, comprising: a determining module and a processing module; wherein the determining module is configured to determine, based on a file modification request, data to be modified in the file; the processing module is configured to, when the amount of the data to be modified is less than or equal to a first threshold, obtain differential data based on the data to be modified and the baseline data of the file; compress the differential data to obtain differential compressed data; and, when the amount of the differential compressed data is less than or equal to a second threshold, store the differential compressed data in a first storage space.
[0020] In conjunction with the second aspect, in one implementation of the second aspect, the above processing module is used to: obtain the base data of the above file from the base data page linked list in the cache, wherein the base data page linked list includes multiple base data pages sorted according to access time.
[0021] In conjunction with the second aspect, in one implementation of the second aspect, the above processing module is used to: determine the file offset address of the differential data and the index of the original compressed data corresponding to the differential data; and store the file offset address and the index of the original compressed data into the first storage space.
[0022] In conjunction with the second aspect, in one implementation of the second aspect, the processing module is used to: determine the original compressed data of the file stored in the second storage space based on the index of the original compressed data, wherein the second storage space is different from the first storage space; decompress the original compressed data to obtain decompressed data; modify the decompressed data based on the differential compressed data and the file offset address to obtain updated data; compress the updated data and store the new compressed data in the second storage space.
[0023] In conjunction with the second aspect, in one implementation of the second aspect, the above-mentioned processing module is further configured to: determine whether differential compressed data of the above-mentioned file exists in the first storage space based on the file read request of the above-mentioned file; if differential compressed data exists in the first storage space, determine the original compressed data according to the index of the original compressed data; and read the updated data of the above-mentioned file.
[0024] In conjunction with the second aspect, in one implementation of the second aspect, the above-mentioned device further includes: a timing module, used to start a timer after storing the differential compressed data in the first storage space; the processing module is used to: when the timer times out, determine the original compressed data according to the index of the original compressed data.
[0025] In conjunction with the second aspect, in one implementation of the second aspect, the above processing module is used to: delete the above differential data, the file offset address of the above differential data, and the index of the original compressed file data.
[0026] Thirdly, another means for modifying a file is provided, including a processor coupled to memory, which can be used to execute instructions in the memory to implement the method in any possible implementation of any of the foregoing aspects. Optionally, the means also includes memory. Optionally, the means also includes a communication interface, to which the processor is coupled.
[0027] Fourthly, a processor is provided, comprising: an input circuit, an output circuit, and a processing circuit. The processing circuit is used to receive signals through the input circuit and transmit signals through the output circuit, causing the processor to execute the method in any possible implementation of the first aspect described above.
[0028] In specific implementation, the processor can be a chip, the input circuit can be an input pin, the output circuit can be an output pin, and the processing circuit can be a transistor, gate circuit, flip-flop, and various logic circuits. The input signal received by the input circuit can be received and input by, for example, but not limited to, a receiver, and the signal output by the output circuit can be output to, for example, but not limited to, a transmitter and transmitted by the transmitter. Furthermore, the input circuit and the output circuit can be the same circuit, which is used as the input circuit and the output circuit at different times. This application does not limit the specific implementation of the processor and various circuits.
[0029] Fifthly, a processing apparatus is provided, including a processor and a memory. The processor is used to read instructions stored in the memory and to receive signals via a receiver and transmit signals via a transmitter to execute the method in any of the possible implementations of the first aspect described above.
[0030] Optionally, there may be one or more processors and one or more memories.
[0031] Alternatively, the memory can be integrated with the processor, or the memory can be set up separately from the processor.
[0032] In specific implementation, the memory can be a non-transitory memory, such as read-only memory (ROM), which can be integrated with the processor on the same chip or set on different chips. The embodiments of this application do not limit the type of memory or the way the memory and processor are set.
[0033] The processing device in the fifth aspect above can be a chip. The processor can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor that reads software code stored in memory. The memory can be integrated into the processor or located outside the processor and exist independently.
[0034] In a sixth aspect, a computer program product is provided, comprising: a computer program (also referred to as code or instructions) that, when executed, causes a computer to perform the method in any of the possible implementations of the first aspect described above.
[0035] In a seventh aspect, a computer-readable storage medium is provided that stores a computer program (also referred to as code or instructions) that, when executed on a computer, causes the computer to perform the methods in any of the possible implementations of the first aspect described above. Attached Figure Description
[0036] Figure 1 This is a schematic diagram of the storage device provided in an embodiment of this application;
[0037] Figure 2 This is a schematic diagram illustrating the read / write amplification problem caused by the modified file provided in the embodiments of this application;
[0038] Figure 3 This is a schematic flowchart illustrating the method for modifying documents provided in an embodiment of this application;
[0039] Figure 4 This is a schematic diagram of the inode data structure provided in an embodiment of this application;
[0040] Figure 5 This is a schematic flowchart illustrating another method for modifying a document provided in an embodiment of this application;
[0041] Figure 6 This is a schematic flowchart of a method for reading files provided in an embodiment of this application;
[0042] Figure 7 This is a schematic flowchart of a method for storing files provided in an embodiment of this application;
[0043] Figure 8 This is a schematic diagram of the apparatus for modifying documents provided in the embodiments of this application;
[0044] Figure 9 This is a schematic diagram of an apparatus for another modified document provided in an embodiment of this application. Detailed Implementation
[0045] The technical solutions in this application will now be described with reference to the accompanying drawings.
[0046] To facilitate understanding, the relevant terms used in this application will be explained first.
[0047] 1. Inode Data Structure
[0048] An inode is a fundamental data structure common to Unix / Linux file systems. Typically, each file corresponds to one inode data structure. An inode object internally records metadata such as file type, file size, modification time, access permissions, and data index information, as well as the file data itself. User file data usually consists of one or more data pages, and the addresses of these data pages are stored in the inode object. The file system can access all data pages through the inode data structure.
[0049] 2. Metadata
[0050] Metadata, also known as intermediary data or relay data, is data about data. It mainly describes the properties of data and is used to support functions such as indicating storage location, historical data, resource lookup, and file records.
[0051] With the development of mobile internet technology, terminal devices have become indispensable in people's work, life, and entertainment, leading to a surge in terminal device applications. These applications continuously generate and modify files during user interaction. The file system, a subsystem within the terminal operating system specifically responsible for file management, provides interfaces for the application layer to respond to application requests, adding, deleting, querying, and reading / writing files. The performance of the file system directly impacts the user experience of upper-layer applications. The terminal device described in this application possesses storage capabilities, and will be referred to herein as a storage device.
[0052] Figure 1 The storage device 100 provided in an embodiment of this application is shown, such as Figure 1As shown, the storage device 100 contains an application program and a file system. The application program issues file read / write requests, and the file system responds to these requests, processes the file data accordingly, and stores the file data in the storage device.
[0053] The file system stores file metadata and user data on persistent storage devices, such as hard disks, solid-state drives (SSDs), and optical discs. Different types of storage devices have different read / write speeds, granularity, and lifespan characteristics. Therefore, academia and major manufacturers have designed and developed different file systems to achieve optimal file read / write performance on different storage devices.
[0054] Many file systems (e.g., f2fs, btrfs) provide compression features to compress file data before storing it on storage devices. These file systems perform full compression on user data, splitting the complete file into multiple parts and compressing each part into a single compressed unit. These compressed units and their data structures are stored independently in different locations on the storage device. The file system records the indexes of all compressed units corresponding to each file in the data structure. When the file system needs to modify the content of a compressed file, it locates the corresponding compressed unit based on the index of the specific unit to be modified, decompresses the entire compressed unit, modifies it, recompresses it, and writes it back to the storage device. However, full compression cannot accurately predict the exact location of the user's modification request within a compressed unit. It requires reading the entire compressed unit from the storage device, modifying it, recompressing it, and writing it back, thus creating a read / write amplification problem.
[0055] For small modification requests of a few bytes, full compression amplifies the amount of data read and written to the storage device to the entire compression unit. When the data just crosses the boundary of the compression unit, it may even be necessary to read and write multiple compression units. Figure 2 This is a schematic diagram of read / write magnification in existing technology, such as... Figure 2 As shown, three unfilled rectangles represent three compressed data units before modification. When the storage device modifies the file data based on a modification request, such as... Figure 2 As shown, the actual data size requested for modification in the modification request is the padded portion of the diagram. However, during the actual modification process, the storage device needs to read the entire compressed data unit containing the data to be modified and decompress it. This results in the storage device reading a much larger amount of data than modifying it, causing read / write amplification. Read / write amplification increases file read / write latency and may even negate the space gains from compression. Furthermore, on various storage devices such as flash memory, read / write amplification from full compression can increase device lifespan, reducing the lifespan of the storage device and the user experience.
[0056] In view of this, this application provides a method and apparatus for modifying a file. A storage device obtains differential data from the file's data to be modified and its baseline data, and compresses this differential data to obtain differential compressed data. The storage device can first store this differential compressed data in a first storage space, without immediately modifying the compressed data of the file. Subsequently, based on the file offset address of the differential compressed data, the location where the decompressed data needs modification can be accurately located, and the compressed data can be updated. In this way, embodiments of this application can accumulate multiple data to be modified for updating as needed, which helps reduce the number of read / write operations on the storage device, thereby reducing the lifespan wear and tear on the storage device.
[0057] Before introducing the methods and apparatus for modifying documents provided in the embodiments of this application, the following points should be made first.
[0058] First, in the embodiments shown below, the terms and English abbreviations, such as baseline data or differential data, are merely exemplary examples given for ease of description and should not constitute any limitation on this application. This application does not preclude the possibility of defining other terms that can achieve the same or similar functions in existing or future agreements.
[0059] Second, in the embodiments shown below, the terms "first," "second," and various numerical designations are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. For example, "first storage space" and "second active storage space" are simply used to distinguish different storage spaces, etc.
[0060] Third, "at least one" means one or more, while "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can mean: a, or b, or c, or a and b, or a and c, or b and c, or a, b, and c, where a, b, and c can be single or multiple.
[0061] To make the objectives and technical solutions of this application clearer and more intuitive, the methods and apparatus provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.
[0062] It should be understood that the methods in the embodiments of this application can be executed by a storage device with read and write capabilities, which is referred to as a storage device in the embodiments of this application.
[0063] Figure 3 This is a schematic flowchart of a method 300 for modifying a file according to an embodiment of this application. Figure 3 As shown, method 300 includes the following steps:
[0064] S301, the storage device determines the data to be modified in the file based on the file modification request.
[0065] S302, when the amount of data to be modified is less than or equal to the first threshold, the storage device obtains differential data based on the data to be modified and the baseline data of the file.
[0066] It should be understood that the aforementioned first threshold can be configured by the system administrator, and this first threshold is used to measure the size of the data to be modified.
[0067] For example, the first threshold can be preset to the size of a compressed data unit. When the amount of data to be compressed is less than or equal to the size of a compressed data unit, in order to avoid read / write amplification problems, the storage device can obtain differential data based on the data to be modified and the baseline data of the file.
[0068] In one possible implementation, the storage device can obtain the difference data before and after the file data is modified based on the data to be modified and the baseline data of the file, and encode the difference data to obtain differential data.
[0069] S303, the storage device compresses the differential data to obtain differential compressed data.
[0070] S304, if the amount of differential compressed data is less than or equal to the second threshold, the storage device stores the differential compressed data in the first storage space.
[0071] It should be understood that the aforementioned second threshold can be configured by the system administrator. This second threshold is used to measure whether the amount of differentially compressed data exceeds the amount of data that the free space of the first storage space can hold.
[0072] For example, the second threshold can be preset as the maximum amount of data that the free space of the first storage space can carry. When the differential data is less than or equal to the maximum amount of data that the free space of the first storage space can carry, the storage device can store the differential compressed data in the first storage space.
[0073] It should be understood that while storing the differential compressed data into the first storage space, the storage device can determine the file offset address of the differential data and the index of the original compressed data corresponding to the differential data, and store the file offset address and the index of the original compressed data into the first storage space.
[0074] In this embodiment, the storage device can obtain differential data from the data to be modified and the baseline data of the file, and compress the differential data to obtain differential compressed data. The storage device can store the differential compressed data in the first storage space without immediately modifying the compressed data of the file. The compressed data of the file can be modified as needed later. In this way, this embodiment can accumulate multiple data to be modified as needed for updating, which helps to reduce the number of read and write operations on the storage device, thereby reducing the wear and tear on the storage device.
[0075] Optionally, when a user's read request is received, the storage device can update the original compressed data by combining it with previously stored differential compressed data, and then read the updated data. Alternatively, the storage device can periodically update the original compressed data by combining it with differential compressed data stored during that period.
[0076] It should be understood that the aforementioned first storage space can be the space of an inode data structure. The following describes in detail the solution provided in this application embodiment using an inode as an example.
[0077] Figure 4 This is a schematic diagram of the inode data structure provided in an embodiment of this application. Figure 4 As shown, the inode data structure consists of a header metadata area, a middle data index pointer area, and a tail direct / indirect node area. The inode tail extends from the inode data structure to the end of the entire physical page. The entire inode occupies a 4KB physical page. In this application, the metadata area records metadata such as file type, file size, modification time, access permissions, and data index information. The middle data index pointer area can be used to store differentially compressed data, file offset addresses, the index of the original compressed data, and the original compressed data of small files.
[0078] Figure 5 This application illustrates another modified document method 500 provided in an embodiment of the present application. For example... Figure 5 As shown, method 500 includes the following steps:
[0079] S501, the storage device determines the data to be modified in a file based on a file modification request.
[0080] S502, the storage device determines whether the amount of the data to be modified is less than or equal to the first threshold.
[0081] S503, when the amount of data to be modified is less than or equal to the first threshold, the storage device obtains differential data based on the data to be modified and the reference data of the file.
[0082] S504, the storage device compresses differential data to obtain differential compressed data.
[0083] S505, the storage device determines whether the amount of differentially compressed data is less than or equal to a second threshold.
[0084] If the amount of differential compressed data is less than or equal to the second threshold, execute S506, and the storage device stores the differential compressed data in the inode, then the modification process ends.
[0085] If the amount of data to be modified is less than or equal to the first threshold, or if the amount of data in the differentially compressed data is less than or equal to the second threshold, the following steps shall be performed:
[0086] S507 and storage devices can decompress the original compressed data of a file to obtain the decompressed data.
[0087] S508, the storage device modifies the decompressed data based on the file modification data to obtain the updated data of the file.
[0088] S509, the storage device compresses the updated data and stores it in the second storage space, then ends the modification process.
[0089] It should be understood that the second storage space mentioned above is different from the first storage space mentioned above, and the two are independent of each other. The first storage space can store the index of the original compressed data of the file, and the second storage space can be used to store the original compressed data of the file. The index of the compressed data in the first storage space corresponds one-to-one with the compressed data in the second storage space.
[0090] As an optional embodiment, after the differential compressed data is stored in the first storage space, the storage device can determine the original compressed data of the file stored in the second storage space based on the index of the original compressed data, decompress the original compressed data to obtain decompressed data; modify the decompressed data based on the differential compressed data and the file offset address to obtain updated data; compress the updated data and store the new compressed data in the second storage space.
[0091] Specifically, the storage device can modify the decompressed data based on the differentially compressed data to obtain updated data in several possible ways.
[0092] In one possible implementation, the storage device can modify the decompressed data of the file if it determines that differentially compressed data of the file exists in the inode based on the file read request of the file mentioned above.
[0093] Figure 6 This application illustrates a method 600 for reading a file when differentially compressed data exists in an inode, as provided in an embodiment of this application. For example... Figure 6 As shown, the method 600 includes the following steps:
[0094] S601, the storage device receives a file read request.
[0095] S602, the storage device determines whether differentially compressed data of a file exists in the inode based on the file read request.
[0096] S603, if differential compressed data of a file exists in the inode, the storage device can determine the original compressed data stored in the second storage space based on the index of the original compressed data in the inode, and decompress the compressed data to obtain decompressed data.
[0097] S604, the storage device can modify the decompressed data based on the differential compressed data and file offset address in the inode data structure to obtain updated data.
[0098] S605, the storage device reads the updated data.
[0099] S606, the storage device compresses the updated data to obtain new compressed data.
[0100] S607, the storage device stores the new compressed data to the second storage space and then ends the current read process.
[0101] It should be understood that if differentially compressed data does not exist in the inode, the following steps can be performed:
[0102] S608, the storage device can directly determine the original compressed data stored in the second storage space based on the index of the original compressed data in the inode, and decompress the compressed data.
[0103] S609 directly reads the decompressed data and then ends the current reading process.
[0104] In another possible implementation, when differential compressed data exists in the inode data structure space, or after the storage device stores the differential compressed data in the inode data structure space, the storage device can start a timer. When the timer expires, the storage device can determine the original compressed data based on the index of the original compressed data, and decompress the original compressed data to obtain decompressed data. The storage device can modify the decompressed data based on the differential compressed data and the file offset address to obtain updated data, compress the updated data, and store the new compressed data in the aforementioned second storage space for later reading.
[0105] It should be understood that the space of the inode data structure mentioned above can contain differential compressed data of multiple files. The storage device can start a timer. When the timer expires, the storage device can modify and update the decompressed data of the corresponding multiple files based on the multiple differential data in the space of the inode data structure. Here, the multiple differential compressed data and the decompressed data of the multiple files correspond one-to-one.
[0106] For example, the storage device modifies and updates the decompressed data of the file sequentially based on the arrangement order of the differential compressed data in the inode data structure, and stores the updated data of the multiple files in the second storage space.
[0107] It should be understood that storage devices can have a dedicated background thread that periodically modifies and updates the decompressed data of files based on differential compression data. The frequency of this background thread can be configured and controlled by the system administrator.
[0108] As an optional embodiment, before obtaining the differential data based on the data to be modified and the baseline data of the file, the method further includes: obtaining the baseline data of the file from a cached baseline data page list, the baseline data page list including multiple baseline data pages sorted according to access time.
[0109] In this embodiment, since each write operation to modify a file requires comparing the data to be modified with the reference data to generate differential data, in order to avoid repeatedly reading the reference data from the storage medium, the storage device can set up a reference data page linked list in the page cache of the operating system, and cache the reference data pages in the reference data page linked list in the order of access time. This helps to reduce the number of times the storage medium is read by read and write operations, thereby reducing the lifespan wear of the storage medium caused by read and write operations.
[0110] Optionally, before obtaining the differential data based on the aforementioned data to be modified and the baseline data of the aforementioned file, the storage device may pre-load and cache frequently accessed baseline data pages in a baseline data page linked list set in the operating system's page cache. Each baseline data page is sorted in the baseline data page linked list according to its access frequency. It should be understood that the aforementioned baseline data pages can also be sorted in other ways, and this embodiment of the application does not limit this.
[0111] Optionally, during memory cleanup, the storage device can also prioritize removing the baseline data pages that are at the end of the baseline data page list from the baseline data page list to free up storage space and reduce the load on the storage space.
[0112] As an optional embodiment, after modifying the decompressed data based on the differential compressed data, the storage device can also delete the differential data, the file offset address of the differential data, and the index of the original compressed file data to release the corresponding storage space and reduce the load on the storage space.
[0113] The method for modifying files in this application involves obtaining differential data from the file's data to be modified and its baseline data, and then compressing this differential data to obtain differential compressed data. The storage device only needs to store this differential compressed data in the first storage space, without immediately modifying the file data. In one implementation, the storage device can periodically modify the original compressed data of multiple files based on multiple differential compressed data. In another implementation, the storage device can modify the original compressed data of a file based on the differential compressed data when a file read request is received. In both possible implementations, the storage device does not need to update the baseline data, nor does it need to read the original compressed data of the file multiple times from the reading storage medium due to multiple file modifications. This helps reduce the number of read / write operations on the storage medium, thereby reducing the lifespan wear of the storage device caused by read / write operations. Furthermore, during the modification process, the storage device can accurately locate the position where the decompressed data needs to be modified based on the file offset address of the differential compressed data, avoiding the problem of file read / write amplification and reducing file read / write latency.
[0114] It should be understood that the storage device can store the file before modifying or reading it, as will be discussed below. Figure 7 The file storage method provided in the embodiments of this application will be described in detail.
[0115] Figure 7 A method 700 for storing files according to an embodiment of this application is illustrated. For example... Figure 7 As shown, the method 700 includes the following steps:
[0116] S701, the storage device determines the amount of file data based on the file storage request.
[0117] S702, determine whether the amount of data in the file is less than or equal to the third threshold.
[0118] S703, if the amount of file data is less than or equal to the third threshold mentioned above, the storage device can compress the file data to obtain the original compressed data of the file.
[0119] S704, the storage device determines whether the amount of the original compressed data of the file is less than or equal to the fourth threshold.
[0120] S705, if the amount of the original compressed data is less than or equal to the fourth threshold mentioned above, the storage device may store the original compressed data into the space of the inode data structure.
[0121] S706, if the amount of file data is greater than the third threshold, or if the amount of the original compressed data is greater than or equal to the fourth threshold, the storage device may split the file data into multiple parts, and each part may be compressed into an original compressed data.
[0122] S707, the storage device stores the aforementioned multiple original compressed data into the aforementioned second storage space.
[0123] In this embodiment, small files require very little data storage, resulting in a significant amount of unused storage space. Therefore, this application directly stores the original compressed data of small files in the first storage space, eliminating the need for additional storage in the second storage space. This reduces storage space usage and improves space utilization. Furthermore, since the original compressed data of small files is directly stored in the first storage space, it is not necessary to use the index of the original compressed data in the first storage space to determine and read the original compressed data in the second storage space. This improves the read / write speed of small files, reduces the number of read / write operations on the storage medium, and consequently reduces the lifespan of the storage device.
[0124] It should be understood that the embodiments of this application are merely described using the example of the first storage space being the space of the inode data structure. The first storage space can also be the space of other data structures, and this application does not limit it in this regard.
[0125] It should be understood that the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0126] The above text combines Figures 1 to 7 The present application describes in detail the method for modifying files provided in the embodiments of this application. The following will refer to the appendix. Figure 8 and Figure 9 This application describes in detail the apparatus for modifying documents provided in the embodiments of this application.
[0127] Figure 8 An apparatus 800 for modifying documents according to an embodiment of this application is shown, including: a determination module 801 and a processing module 802.
[0128] The determination module 801 is used to determine the data to be modified in a file based on a file modification request; the processing module 802 is used to obtain differential data based on the data to be modified and the baseline data of the file when the amount of the data to be modified is less than or equal to a first threshold; to compress the differential data to obtain differential compressed data; and to store the differential compressed data in a first storage space when the amount of the differential compressed data is less than or equal to a second threshold.
[0129] Optionally, the processing module 802 is used to retrieve the baseline data of the aforementioned file from a baseline data page linked list in the cache, the baseline data page linked list including multiple baseline data pages sorted according to access time.
[0130] Optionally, the processing module 802 is used to determine the file offset address of the differential data and the index of the original compressed data corresponding to the differential data; and to store the file offset address and the index of the original compressed data into the first storage space.
[0131] Optionally, the processing module 802 is configured to: determine the original compressed data of the file stored in the second storage space based on the index of the original compressed data, wherein the second storage space is different from the first storage space; decompress the original compressed data to obtain decompressed data; modify the decompressed data based on the differential compressed data and the file offset address to obtain updated data; compress the updated data and store the new compressed data in the second storage space.
[0132] Optionally, the processing module 802 is further configured to determine whether differential compressed data of the file exists in the first storage space based on the file read request of the file; if differential compressed data exists in the first storage space, determine the original compressed data according to the index of the original compressed data; and read the updated data of the file.
[0133] Optionally, the above device further includes: a timing module 803, used to start a timer after storing the differential compressed data in the first storage space; and a processing module 802, used to determine the original compressed data according to the index of the original compressed data when the timer expires.
[0134] Optionally, the processing module 802 is used to delete the aforementioned differential data, the file offset address of the aforementioned differential data, and the index of the original compressed file data.
[0135] It should be understood that the device 800 here is embodied in the form of a functional module. The term "module" here can refer to application-specific integrated circuits (ASICs), electronic circuits, processors (e.g., shared processors, proprietary processors, or group processors, etc.) and memories for executing one or more software or firmware programs, integrated logic circuits, and / or other suitable components supporting the described functions. In an alternative example, those skilled in the art will understand that the device 800 may specifically be the storage device in the above embodiments, or the functions of the storage device in the above embodiments may be integrated into the device 800. The device 800 may be used to execute the various processes and / or steps corresponding to the storage device in the above method embodiments; to avoid repetition, these will not be described further here.
[0136] The aforementioned device 800 has the function of implementing the corresponding steps performed by the storage device in the aforementioned method; the aforementioned function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the aforementioned function.
[0137] In the embodiments of this application, Figure 8 The device 800 in the text can also be a chip or a chip system, such as a system on chip (SoC).
[0138] Figure 9 An apparatus 900, as shown in another modified document provided in this application embodiment, is illustrated. The apparatus 900 includes a processor 901, a memory 902, a communication interface 903, and a bus 904. The memory 902 stores instructions, and the processor 901 executes the instructions stored in the memory 902. The processor 901, memory 902, and communication interface 903 are interconnected via the bus 904.
[0139] The processor 901 is configured to: determine the data to be modified in a file based on a file modification request; obtain differential data based on the data to be modified and the baseline data of the file when the amount of the data to be modified is less than or equal to a first threshold; compress the differential data to obtain differential compressed data; and store the differential compressed data in a first storage space when the amount of the differential compressed data is less than or equal to a second threshold.
[0140] It should be understood that device 900 may specifically be the storage device in the above embodiments, or the functions of the storage device in the above embodiments may be integrated into device 900. Device 900 may be used to execute the various steps and / or processes corresponding to the storage device in the above method embodiments. Optionally, the memory 903 may include read-only memory and random access memory, and provide instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store device type information. The processor 901 may be used to execute instructions stored in the memory, and when the processor executes the instructions, the processor may execute the various steps and / or processes corresponding to the storage device in the above method embodiments.
[0141] It should be understood that, in the embodiments of this application, the processor may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.
[0142] In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The steps of the method disclosed in the embodiments of this application can be directly manifested as execution by a hardware processor, or as a combination of hardware and software modules within the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor executes the instructions in the memory, combining them with its hardware to complete the steps of the above method. To avoid repetition, detailed descriptions are omitted here.
[0143] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0144] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0145] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0146] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0147] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0148] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0149] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method for modifying a file, characterized in that, include: Based on the file modification request, determine the data to be modified in the file; If the amount of data to be modified is less than or equal to a first threshold, differential data is obtained based on the data to be modified and the baseline data of the file. The differential data is compressed to obtain differential compressed data; If the amount of differentially compressed data is less than or equal to the second threshold, the differentially compressed data is stored in the first storage space; the first storage space is the space of the inode data structure in the file system. When a preset trigger condition is met, the original compressed data of the file is updated based on the differential compressed data; The preset triggering conditions include: receiving a file read request and / or detecting a timer timeout.
2. The method according to claim 1, characterized in that, Before obtaining the differential data based on the data to be modified and the baseline data of the file, the process further includes: The baseline data of the file is obtained from a linked list of baseline data pages in the cache, the linked list of baseline data pages comprising multiple baseline data pages sorted by access time.
3. The method according to claim 1 or 2, characterized in that, The method further includes: Determine the file offset address of the differential data and the index of the original compressed data corresponding to the differential data; The file offset address and the index of the original compressed data are stored in the first storage space.
4. The method according to claim 3, characterized in that, The step of updating the original compressed data of the file based on the differential compressed data includes: Based on the index of the original compressed data, the original compressed data of the file stored in the second storage space is determined, and the second storage space is different from the first storage space; The original compressed data is decompressed to obtain decompressed data; Based on the differential compressed data and the file offset address, modify the decompressed data to obtain updated data; The updated data is compressed, and the new compressed data is stored in the second storage space.
5. The method according to claim 4, characterized in that, Before determining the original compressed data of the file stored in the second storage space based on the index of the original compressed data, the method further includes: Based on the file read request of the file, determine whether differentially compressed data of the file exists in the first storage space; Determining the original compressed data of the file stored in the second storage space based on the index of the original compressed data includes: If the differential compressed data exists in the first storage space, the original compressed data is determined according to the index of the original compressed data; The method further includes: Read the updated data from the file.
6. The method according to claim 4, characterized in that, After storing the differentially compressed data in the first storage space, and before updating the original compressed data of the file based on the differentially compressed data, the method further includes: Start the timer; When the timer times out, the original compressed data is determined based on the index of the original compressed data.
7. The method according to any one of claims 4 to 6, characterized in that, The method further includes: Delete the differential data, the file offset address of the differential data, and the index of the original compressed file data.
8. An apparatus for modifying a file, characterized in that, include: The determination module is used to determine the data to be modified in a file based on a file modification request; The processing module is used to obtain differential data based on the data to be modified and the baseline data of the file when the amount of data to be modified is less than or equal to a first threshold. The differential data is compressed to obtain differential compressed data; Furthermore, if the amount of differentially compressed data is less than or equal to the second threshold, the differentially compressed data is stored in the first storage space; the first storage space is the space of the inode data structure in the file system. When a preset trigger condition is met, the original compressed data of the file is updated based on the differential compressed data; The preset triggering conditions include: receiving a file read request and / or detecting a timer timeout.
9. The apparatus according to claim 8, characterized in that, The processing module is used for: The baseline data of the file is obtained from a linked list of baseline data pages in the cache, the linked list of baseline data pages comprising multiple baseline data pages sorted by access time.
10. The apparatus according to claim 8 or 9, characterized in that, The processing module is used for: Determine the file offset address of the differential data and the index of the original compressed data corresponding to the differential data; The file offset address and the index of the original compressed data are stored in the first storage space.
11. The apparatus according to claim 10, characterized in that, The processing module is used for: Based on the index of the original compressed data, the original compressed data of the file stored in the second storage space is determined, and the second storage space is different from the first storage space; The original compressed data is decompressed to obtain decompressed data; Based on the differential compressed data and the file offset address, modify the decompressed data to obtain updated data; The updated data is compressed, and the new compressed data is stored in the second storage space.
12. The apparatus according to claim 11, characterized in that, The processing module is also used for: Before determining the original compressed data of the file stored in the second storage space based on the index of the original compressed data, it is determined whether differential compressed data of the file exists in the first storage space based on the file read request of the file; If the differential compressed data exists in the first storage space, the original compressed data is determined according to the index of the original compressed data; And, read the updated data of the file.
13. The apparatus according to claim 11, characterized in that, The device further includes: A timing module is used to start a timer after the differential compressed data is stored in the first storage space and before the original compressed data of the file is updated based on the differential compressed data; The processing module is also used for: When the timer times out, the original compressed data is determined based on the index of the original compressed data.
14. The apparatus according to any one of claims 11 to 13, characterized in that, The processing module is used for: Delete the differential data, the file offset address of the differential data, and the index of the original compressed file data.
15. An apparatus for modifying a file, characterized in that, include: A processor coupled to a memory for storing a computer program, which, when invoked by the processor, causes the apparatus to perform the method as described in any one of claims 1 to 7.
16. A computer-readable storage medium, characterized in that, Used to store a computer program, the computer program including instructions for implementing the method as described in any one of claims 1 to 7.
17. A computer program product, said computer program product comprising computer program code, characterized in that, When the computer program code is run on a computer, the computer causes the computer to implement the method as described in any one of claims 1 to 7.