File processing method and device, equipment and storage medium

Abstract

The invention provides a file processing method and device, equipment and a storage medium, and relates to the technical field of databases and the technical field of file systems. The method is applied to a target node, the target node is any one node in a cluster file system, and the method comprises the steps that when a file write-in request is received, data corresponding to the file write-in request is written into a shared disk in a direct input and output mode; under the condition that mapping information corresponding to the first logic address of the file writing request is not stored, allocating a first physical address to the first logic address corresponding to the file writing request; wherein for the file write-in request of the same logic address, the physical addresses allocated to the nodes of the cluster file system are different; and storing mapping information of the first logic address and the first physical address and a timestamp for recording file writing. According to the method, lock competition can be eliminated to improve the performance of the cluster file system.

Description

Technical Field

[0001] This application relates to the fields of database technology and file system technology, and in particular to a file processing method, apparatus, device and storage medium. Background Technology

[0002] In a clustered file system scenario based on shared disks, multiple nodes share the same storage resource to achieve efficient data read / write and load balancing. For example, in a database load balancing scenario, multiple database instances may simultaneously perform read / write operations on the same data file.

[0003] Related technologies rely on distributed lock mechanisms to ensure data consistency, but lock contention can lead to performance bottlenecks. Furthermore, in scientific computing or real-time data processing scenarios, cluster nodes frequently need to access shared files; delays caused by lock waiting can directly impact the efficiency of computational tasks. Summary of the Invention

[0004] This application provides a file processing method, apparatus, device, and storage medium to eliminate lock contention and improve the performance of a clustered file system.

[0005] In a first aspect, this application provides a file processing method applied to a target node, which is any node in a cluster file system. The method includes: upon receiving a file write request, writing the data corresponding to the file write request to a shared disk via direct input / output; allocating a first physical address to the first logical address corresponding to the file write request when the mapping information corresponding to the first logical address of the file write request is not stored; wherein, for file write requests with the same logical address, the physical addresses allocated to different nodes in the cluster file system are different; storing the mapping information between the first logical address and the first physical address and recording the timestamp of the file write.

[0006] In one possible embodiment, the method further includes: if mapping information corresponding to a first logical address of a file write request is stored, overwriting the write data at the first physical address and updating the timestamp of the mapping information.

[0007] In one possible embodiment, the method further includes: storing mapping information to a shared disk, and recording the addresses of the mapping information of each node of the cluster file system in the shared disk by the inode number of the file.

[0008] In one possible embodiment, storing the mapping information to a shared disk includes: representing the mapping information using a hash table structure and storing the hash table structure to the shared disk, wherein the first logical address is the key of the hash table structure, and the first physical address and timestamp are the values ​​of the hash table structure.

[0009] In one possible embodiment, the method further includes: upon receiving a file read request, obtaining the address of the mapping information of each node in the shared disk from the inode number of the file, the file read request being used to read data at a first logical address; based on the address of the mapping information of each node in the shared disk, obtaining the mapping information of each node from the shared disk; selecting the first physical address with the latest timestamp from the mapping information of each node and reading data from the first physical address with the latest timestamp.

[0010] In one possible embodiment, the shared disk includes blocks of consecutive addresses for storing mapping information of each node; obtaining the mapping information of each node from the shared disk includes: obtaining the mapping information of each node one at a time from the blocks of consecutive addresses of the shared disk via block input / output.

[0011] In one possible embodiment, the method further includes: scanning the mapping information of each node in the shared disk; for the first logical address, deleting expired data in the shared disk whose write timestamp is earlier than a preset threshold and updating the mapping information according to the timestamp corresponding to the mapping information of each node.

[0012] Secondly, this application provides a file processing apparatus applied to a target node, which is any node in a clustered file system. The apparatus includes: a writing module, used to write data corresponding to a file write request to a shared disk via direct input / output when a file write request is received; an allocation module, used to allocate a first physical address to the first logical address corresponding to the file write request when no mapping information corresponding to the first logical address of the file write request is stored; wherein, for file write requests at the same logical address, different nodes in the clustered file system are allocated different physical addresses; and a storage module, used to store the mapping information between the first logical address and the first physical address and to record the timestamp of the file write.

[0013] Thirdly, this application provides an electronic device, including: a processor and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method as described in any of the first aspects.

[0014] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any of the first aspects.

[0015] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method of any one of the first aspects.

[0016] The file processing method of this application embodiment can improve the performance of multiple nodes concurrently writing to the same logical address without the need for distributed locks, while avoiding network communication and computational overhead caused by lock switching. Furthermore, since each node is assigned a different physical address for the same logical address, the risk of deadlock due to locks not being released in time can be avoided, further enhancing the reliability and throughput of the cluster file system. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] Figure 1 This is a schematic diagram illustrating an application scenario of the file processing method according to an embodiment of this application;

[0019] Figure 2 This is a flowchart of a file processing method according to an embodiment of this application;

[0020] Figure 3 This is a flowchart illustrating the file writing and mapping information storage process according to an embodiment of this application.

[0021] Figure 4 This is a flowchart illustrating a document processing method according to yet another embodiment of this application;

[0022] Figure 5 This is a schematic diagram of a document processing apparatus according to an embodiment of this application;

[0023] Figure 6 This is a schematic diagram of an electronic device according to an embodiment of this application.

[0024] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0025] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0026] The technical terms used in the embodiments of this application will be explained below.

[0027] File system: The interface layer between the operating system and storage devices, which is a logical mechanism for organizing, storing, managing and accessing data.

[0028] Inode number: uniquely identifies a file and stores file metadata.

[0029] Direct Input / Output (DIO): By bypassing the operating system's file system cache, it directly reads and writes to the physical storage medium.

[0030] Block: The smallest physical storage unit on a disk.

[0031] Block Input / Output: Describes input / output operations on a block.

[0032] In related technology 1, cluster file systems based on shared disks commonly employ distributed lock mechanisms to ensure data consistency. Specifically, when a node needs to read or write a shared file, it first requests the corresponding distributed lock (read lock or write lock). Only after the lock is exclusively held by the current node can the actual read / write operation be performed. For example, a write operation requires acquiring a write lock first to ensure that other nodes cannot modify the same data simultaneously; a read operation uses a shared lock to allow multiple nodes to read concurrently. Lock release must occur after data has been written to disk (e.g., dirty data flushing) to avoid data inconsistency. However, this mechanism suffers from at least one of the following drawbacks:

[0033] Lock contention leads to performance bottlenecks: In high-concurrency scenarios, multiple nodes frequently request the same lock, resulting in increased lock waiting time and decreased system throughput.

[0034] Lock switching overhead is high: Lock acquisition, holding and release involve complex distributed coordination (such as lock services based on Paxos or ZooKeeper), which increases network communication and computational overhead.

[0035] Data consistency risk: If a node fails while holding a lock, it may cause deadlock or data loss because the lock is not released in time.

[0036] To address the aforementioned issues, related technology 2 attempts to reduce lock contention through file partitioning or access shuffling algorithms. For example, files can be divided into directories or logical locations, allowing different nodes to access different partitions; or a hash algorithm can be used to route access requests for the same file to specific nodes.

[0037] However, these solutions can only reduce the probability of lock contention, not completely eliminate the locking mechanism, and may introduce additional routing overhead or partition management complexity. Furthermore, if multiple nodes still need to access the same logical address (such as concurrent updates in database transactions), these technologies still cannot avoid lock contention.

[0038] Based on this, embodiments of this application provide a file processing method, apparatus, device, and storage medium. By combining lock-free design with mapping information management and timestamp updates, data consistency is guaranteed and file system performance is improved when multiple nodes access the same logical address in parallel within a clustered file system. Specifically, lock contention is eliminated by directly writing data via I / O and independently allocating physical storage space; each node maintains a mapping between its written logical address and physical address, and achieves efficient management through contiguous storage and inode association; the latest written data is filtered using timestamps, ensuring read consistency without locks; and storage resources are released and the accuracy of mapping information is maintained by cleaning up old data.

[0039] Figure 1 This is a schematic diagram illustrating an application scenario of the file processing method according to an embodiment of this application.

[0040] like Figure 1 As shown, user 1 initiates a request through application 3 installed on terminal device 2. This request is for writing or reading a file, and the request is sent to electronic device 4. Electronic device 4 can be a server, and is a node in a distributed cluster on which a clustered file system is deployed. Electronic device 4 executes the file processing method of this embodiment according to the request.

[0041] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0042] Figure 2 This is a flowchart illustrating a file processing method according to an embodiment of this application. The file processing method of this embodiment can be executed by a target node, which can be any node in a clustered file system, and any node in the clustered file system can be a server.

[0043] like Figure 2 As shown, the file metadata storage method of this application embodiment includes steps S110 to S130.

[0044] S110. When receiving a file write request, write the data corresponding to the file write request to the shared disk through direct input / output.

[0045] A file write request includes a first logical address and the data to be written. The logical address is an offset chain within the file, and the first logical address indicates the relative position of the data to be written within the file. For example, the first logical address indicates byte 1024 of the file.

[0046] A shared disk refers to a unified storage disk accessible to all nodes of a cluster file system.

[0047] S120. If the mapping information corresponding to the first logical address of the file write request is not stored, allocate a first physical address to the first logical address corresponding to the file write request.

[0048] The physical address refers to the address where the data is actually stored on the shared disk. The first physical address is used by the target node to write data for a file write request on the shared disk.

[0049] It's understandable that the target node can be any node in the cluster file system; that is, for the same node, one logical address corresponds to one physical address. However, for file write requests to the same logical address, different nodes in the cluster file system are assigned different physical addresses.

[0050] S130, Store the mapping information between the first logical address and the first physical address, and record the timestamp of the file being written.

[0051] The mapping information is used to record the correspondence between the first logical address and the first physical address.

[0052] In this embodiment of the application, when a target node receives a file write request, it writes the data corresponding to the file write request to the shared disk through a direct input / output method, which can avoid cache consistency issues and eliminate the delay caused by cache synchronization in the locking mechanism.

[0053] By allocating a first physical address to the first logical address corresponding to the file write request when the target node does not store the mapping information for the first logical address, writes to the same logical address by different nodes can be performed without interference. Therefore, there is no need to coordinate the access order of nodes to the same logical address through a locking mechanism, thus eliminating lock contention. For example, when node A and node B simultaneously write data to logical address 1024 bytes, node A and node B are assigned physical addresses 1234 and 4567 respectively. Since node A and node B have different physical addresses for the same logical address, their write operations can be performed in parallel without waiting for the other to release the lock.

[0054] By storing the mapping information between the first logical address and the first physical address, as well as the timestamp of the record file writing, the target node can easily manage the version of the data written to the first physical address, thereby providing a basis for subsequent reading of data from the first physical address and ensuring data consistency in a lock-free scenario.

[0055] In other words, the file processing method of this application embodiment can improve the performance of multiple nodes concurrently writing to the same logical address without the need for distributed locks, while avoiding network communication and computational overhead caused by lock switching. Furthermore, since each node is assigned a different physical address for the same logical address, the risk of deadlock due to locks not being released in time can be avoided, further enhancing the reliability and throughput of the cluster file system.

[0056] For example, taking the mapping information stored on a shared disk as an example, the target node can detect whether the shared disk stores the mapping information corresponding to the first logical address of the file write request. If the shared disk does not store the mapping information corresponding to the first logical address of the file write request, this means that the current file write request is for requesting to write new data that has not been written before at the first logical address. In this case, a first physical address needs to be allocated for the first logical address corresponding to the file write request to support the actual data writing.

[0057] Of course, in addition to the above situations, there may also be cases where a mapping relationship corresponding to the first logical address of a file write request is stored.

[0058] Specifically, such as Figure 2 As shown, in one possible embodiment, the file processing method further includes step S140. Step S140 may be performed after step S110.

[0059] S140. If the mapping information corresponding to the first logical address of the file write request is stored, the timestamp of the written data is overwritten at the first physical address and the mapping information is updated.

[0060] Taking the example of mapping information being stored on a shared disk, the target node can detect whether the shared disk stores mapping information corresponding to the first logical address of the file write request. If the shared disk stores mapping information corresponding to the first logical address of the file write request, this indicates that the current file write request is for modifying historically written data at the first logical address, and the target node has already been assigned the first physical address corresponding to the first logical address. In other words, mapping information representing the correspondence between the first logical address and the first physical address already exists.

[0061] In this embodiment of the application, by storing the mapping relationship corresponding to the first logical address of the file write request, the timestamp of the write data overwritten at the first physical address and the timestamp of the updated mapping information can be used to update the write data at the first physical address and record the update operation of the write data by updating the timestamp of the updated mapping information, thereby ensuring data consistency.

[0062] In one possible implementation, the execution order of file writing, storing mapping information, writing data based on timestamps, and updating mapping information is as follows: first, file writing is performed; then, mapping information is stored; and finally, the timestamps of the written data and mapping information are updated. This ensures that the cluster file system will not experience data inconsistencies caused by updated mapping information leading to outdated data for the corresponding file write request.

[0063] like Figure 2 As shown, in one possible embodiment, the file processing method further includes step S150. Step S150 may be performed after step S140.

[0064] S150. Store the mapping information to the shared disk, and record the address of the mapping information of each node of the cluster file system in the shared disk by the inode number of the file.

[0065] In this embodiment, the target node stores mapping information in a shared disk and records the addresses of each node's mapping information in the shared disk using the file's inode, enabling efficient management of the mapping information for each node. Specifically, during subsequent file reading, the target node can locate the addresses of each node's mapping information in the shared disk using the file's inode, allowing the target node to directly obtain the mapping information of each node from the corresponding address. Furthermore, the target node can determine the corresponding physical address and read data from that physical address based on the mapping information.

[0066] In one possible embodiment, the step S150 of storing the mapping information to the shared disk includes: representing the mapping information with a hash table structure and storing the hash table structure to the shared disk, wherein the first logical address is the key of the hash table structure, and the first physical address and timestamp are the values ​​of the hash table structure.

[0067] For example, when node A writes data to logical address 1024, it calculates its key (e.g., 5) using a hash function. 5 indicates that "logical address 1024 → physical address 1234, timestamp T1" is stored in the 5th slot of the hash table. Subsequent file reads then directly locate the mapping information through the hash table structure, eliminating the need for linear scanning.

[0068] In this embodiment, by using a hash table structure to represent the mapping information and storing the hash table structure in a shared disk, the efficiency of finding the mapping information in the shared disk can be optimized. Specifically, the hash table structure directly locates the mapping information through a hash function, avoiding the performance loss caused by linear scanning.

[0069] The above embodiments can be understood as file writing and mapping storage. Figure 3This is a flowchart illustrating file writing and mapping storage according to an embodiment of this application. For file writing, it further includes whether to allocate new blocks to meet the storage requirements of the data in the file writing request, and, if mapping information is stored in blocks, whether to allocate new blocks to meet the storage requirements of the mapping information.

[0070] like Figure 3 As shown, the file writing and mapping information storage in this embodiment of the application includes steps S201 to S207.

[0071] S201, Receive file write request.

[0072] S202. Check whether the remaining capacity of the currently allocated data block is greater than the amount of data to be written.

[0073] If the remaining capacity of the currently allocated data block is greater than the amount of data to be written, proceed to step S203. If the remaining capacity of the currently allocated data block is less than or equal to the amount of data to be written, proceed to step S204. The data block is a block on the shared disk used to store the data to be written. The data to be written is the data requested to be written to the shared disk by the file write request.

[0074] S203, Allocate a new data block.

[0075] S204. Write the data to be written to the currently allocated data block.

[0076] S205. Check whether the remaining capacity of the currently allocated mapping block is greater than the amount of data in the mapping information.

[0077] If the remaining capacity of the currently allocated mapping block is greater than the amount of data in the mapping information, proceed to step S206. If the remaining capacity of the currently allocated mapping block is less than or equal to the amount of data in the mapping information, proceed to step S207. The mapping block is a block on the shared disk used to store the mapping information.

[0078] S206. Allocate a new mapping block.

[0079] S207. Write the mapping information to the currently allocated mapping block.

[0080] Figure 4 This is a flowchart illustrating a document processing method according to another embodiment of this application. Figure 4 As shown, the file processing method further includes steps S160 to S180. Steps S160 to S180 may be executed after step S140 or step S150.

[0081] S160. Upon receiving a file read request, obtain the address of the mapping information of each node in the shared disk from the file's inode number.

[0082] A file read request is used to read data from the first logical address.

[0083] S170. Based on the addresses of each node's mapping information in the shared disk, obtain the mapping information of each node from the shared disk.

[0084] S180. From the mapping information of each node, select the first physical address with the latest timestamp and read data from the first physical address with the latest timestamp.

[0085] It should be noted that the mapping information for each node represents the correspondence between its first logical address and first physical address, and all nodes share the same first logical address but have different first physical addresses. Therefore, for the same first logical address, there are write data to that first logical address from various nodes. The writing order of each node to that first logical address can be distinguished based on the timestamp.

[0086] In this embodiment, upon receiving a file read request, the addresses of the mapping information of each node in the shared disk are obtained from the file's inode number. Based on these addresses, the mapping information of each node is retrieved from the shared disk, thus locating the storage location of the data written to the first logical address by each node. By selecting the first physical address with the latest timestamp from the mapping information of each node and reading data from that address, it is ensured that the most recently written data to the first logical address is read.

[0087] The above process does not require coordinating the lock states of nodes other than the target node; it directly selects the version of the data to be read by comparing timestamps. This improves file reading efficiency and ensures data consistency.

[0088] In one possible embodiment, the shared disk includes blocks of contiguous addresses for storing mapping information for each node. These blocks for storing the mapping information for each node can be understood as... Figure 3 The mapping block in the middle.

[0089] For example, blocks with consecutive addresses can be allocated sequentially according to the nodes of the cluster file system. For instance, node A is assigned mapping block 1 and mapping block 2, and node 2 is assigned mapping block 3 and mapping block 4, with mapping blocks 1 to 4 having consecutive addresses on the shared disk.

[0090] It should be noted that, in this embodiment of the application, a physical storage unit block (block) on the disk can be divided into data blocks or mapping blocks based on storage function. When the number of mapping blocks in a physical storage unit block is sufficient to store the mapping information of each node, and there are multiple mapping blocks storing the mapping information of each node, the addresses of these multiple mapping blocks are contiguous. Similarly, when multiple mapping blocks in multiple physical storage unit blocks are needed to store the mapping information of each node, the addresses of these multiple mapping blocks are contiguous. Of course, usually, multiple mapping blocks in a single physical storage unit block are sufficient to store the mapping information of each node. Step S170, obtaining the mapping information of each node from the shared disk, includes: obtaining the mapping information of each node from blocks with contiguous addresses on the shared disk at one time using a block input / output method.

[0091] In this embodiment, multiple blocks with contiguous addresses are used to store the mapping information of each node, thereby ensuring that the mapping information of each node is centrally stored. When retrieving the mapping information of each node, the mapping information of all nodes can be obtained through a single block input / output operation, reducing the number of reads from the shared disk and optimizing the access efficiency of the mapping information.

[0092] It should be noted that each node allocates its own physical address to store data for the same logical address. Therefore, in a multi-node scenario of a clustered file system, there may be multiple data entries. When a file read request requests to read the data corresponding to that logical address, only one of the multiple data entries is actually the latest and valid data. In another embodiment of this application, data reclamation reduces the storage space occupied by multiple data entries at the same logical address, thereby improving file read efficiency.

[0093] Specifically, such as Figure 4 As shown, in another embodiment of this application, the document processing method further includes steps S190 to S200. Steps S190 to S200 may be performed on or after step S150. Figure 4 Taking steps S190 to S200 as an example, which are executed after step S180.

[0094] S190. Scan the mapping information of each node in the shared disk.

[0095] S200: For the first logical address, based on the timestamps corresponding to the mapping information of each node, delete expired data in the shared disk whose write timestamps are earlier than a preset threshold and update the mapping information.

[0096] The preset threshold is the time threshold for determining whether the data written by each node to the first logical address in the shared disk has expired.

[0097] For example, the target node can, for the first logical address, delete expired data in the shared disk whose write timestamp is earlier than the latest timestamp, and update the mapping information according to the timestamps corresponding to the mapping information of each node. That is, the preset threshold is the latest timestamp. Alternatively, the preset threshold is the latest timestamp minus 10 seconds.

[0098] For deleted, expired data, updating the mapping information means deleting the corresponding mapping information. This achieves the updating of mapping information for each node.

[0099] For example, the data recycling operations in steps S190 to S200 can be performed periodically through background tasks.

[0100] In this embodiment of the application, by scanning the mapping information of each node in the shared disk and for the first logical address, according to the timestamp corresponding to the mapping information of each node, expired data in the shared disk whose write timestamp is earlier than a preset threshold is deleted and the mapping information is updated, which can realize the identification and deletion of expired data, thereby releasing the storage space of the shared disk occupied by expired data.

[0101] In summary, the file processing method of this application embodiment does not require the use of distributed locks to ensure consistency during the read and write process of the cluster file system, thereby improving the performance of the cluster file system in lock conflict scenarios.

[0102] Figure 5 This is a schematic diagram of the file processing apparatus provided in an embodiment of this application. The apparatus is applied to a target node, which is any node in a clustered file system, such as... Figure 5 As shown, the device includes: a writing module 310, an allocation module 320, and a storage module 330.

[0103] The write module 310 is used to write the data corresponding to the file write request to the shared disk through direct input / output when a file write request is received.

[0104] The allocation module 320 is used to allocate a first physical address to the first logical address corresponding to the file write request when the mapping information corresponding to the first logical address of the file write request is not stored; wherein, for the same logical address, the physical addresses allocated to each node of the cluster file system are different.

[0105] Storage module 330 is used to store the mapping information between the first logical address and the first physical address, as well as the timestamp of the record file being written.

[0106] In one possible embodiment, the apparatus further includes an update module, configured to overwrite the write data at the first physical address and update the timestamp of the mapping information when the first logical address corresponding to the file write request is stored.

[0107] In one possible embodiment, the device further includes a recording module for storing mapping information to a shared disk and recording the addresses of the mapping information of each node of the cluster file system in the shared disk using the file's inode number.

[0108] In one possible embodiment, the recording module is specifically used to represent the mapping information using a hash table structure and store the hash table structure to a shared disk, wherein the first logical address is the key of the hash table structure, and the first physical address and timestamp are the values ​​of the hash table structure.

[0109] In one possible embodiment, the device further includes: a first acquisition module, configured to, upon receiving a file read request, acquire the address of the mapping information of each node in the shared disk from the inode number of the file, wherein the file read request is used to read data at a first logical address; a second acquisition module, configured to acquire the mapping information of each node from the shared disk based on the address of the mapping information of each node in the shared disk; and a selection module, configured to select a first physical address with the latest timestamp from the mapping information of each node and read data from the first physical address with the latest timestamp.

[0110] In one possible embodiment, the shared disk includes blocks of consecutive addresses for storing mapping information of each node; the second acquisition module is specifically used to acquire the mapping information of each node from the blocks of consecutive addresses of the shared disk in a block input / output manner.

[0111] In one possible embodiment, the device further includes: a scanning module for scanning the mapping information of each node in the shared disk; and a deletion module for deleting expired data in the shared disk whose write timestamps are earlier than a preset threshold and updating the mapping information, based on the timestamps corresponding to the mapping information of each node, for the first logical address.

[0112] The file processing apparatus provided in this application embodiment can be used to execute the technical solutions of the file processing methods in any of the above embodiments of this application. Its implementation principle and technical effect are similar, and will not be described again here.

[0113] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 6 As shown, this application embodiment provides an electronic device including a processor 401 and a memory 402. Optionally, the device further includes a communication component 403. The processor 401, memory 402, and communication component 403 are connected via a bus 404.

[0114] In the specific implementation process, the memory 402 stores code, and the processor 401 runs the code stored in the memory 402 to execute the method of the above method embodiment.

[0115] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0116] In the above Figure 6 In the illustrated embodiments, it should be understood that the processor 401 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0117] The memory 402 may include high-speed RAM memory, and may also include non-volatile memory (NVM), such as at least one disk storage.

[0118] Bus 404 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 404 can be divided into address bus, data bus, control bus, etc. For ease of illustration, the bus 404 in the accompanying drawings of this application is not limited to only one bus or one type of bus.

[0119] This application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods described in the above-described method embodiments.

[0120] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0121] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0122] This application provides a computer program product, including a computer program that, when executed by a processor, implements the methods provided in any of the embodiments described above.

[0123] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0124] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0125] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0126] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0127] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0128] If the integrated unit / module is implemented as a software program module and sold or used as an independent financial product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software financial product. This computer software financial product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0129] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0130] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0131] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A file processing method, characterized in that, Applied to a target node, which is any node in the cluster file system, the method includes: When a file write request is received, the data corresponding to the file write request is written to the shared disk through direct input / output. In the absence of stored mapping information corresponding to the first logical address of the file write request, a first physical address is allocated to the first logical address corresponding to the file write request; wherein, for the same logical address, the physical addresses allocated to each node of the cluster file system are different. The system stores the mapping information between the first logical address and the first physical address, as well as the timestamp of the file being written.

2. The method according to claim 1, characterized in that, The method further includes: In the case of storing mapping information corresponding to the first logical address of the file write request, the timestamp of the mapping information is updated after overwriting the data at the first physical address.

3. The method according to claim 2, characterized in that, The method further includes: The mapping information is stored in the shared disk, and the address of the mapping information of each node of the cluster file system in the shared disk is recorded by the inode number of the file.

4. The method according to claim 3, characterized in that, The process of storing the mapping information to the shared disk includes: The mapping information is represented by a hash table structure and the hash table structure is stored on a shared disk, wherein the first logical address is the key of the hash table structure, and the first physical address and the timestamp are the values ​​of the hash table structure.

5. The method according to claim 3 or 4, characterized in that, The method further includes: Upon receiving a file read request, the mapping information of each node in the shared disk is obtained from the inode number of the file, and the file read request is used to read data from the first logical address; Based on the addresses of each node in the shared disk according to the mapping information of each node, the mapping information of each node is obtained from the shared disk; From the mapping information of each node, select the first physical address with the latest timestamp and read data from the first physical address with the latest timestamp.

6. The method according to claim 5, characterized in that, The shared disk includes blocks of contiguous addresses for storing mapping information of each node; obtaining the mapping information of each node from the shared disk includes: The mapping information of each node is obtained one at a time from the blocks of consecutive addresses of the shared disk using a block input / output method.

7. The method according to claim 3, characterized in that, The method further includes: Scan the mapping information of each node in the shared disk; For the first logical address, based on the timestamps corresponding to the mapping information of each node, expired data in the shared disk whose write timestamps are earlier than a preset threshold are deleted and the mapping information is updated.

8. A document processing device, characterized in that, Applied to a target node, wherein the target node is any node in the cluster file system, the device includes: The writing module is used to write the data corresponding to the file write request to the shared disk through direct input / output when a file write request is received. The allocation module is used to allocate a first physical address to the first logical address corresponding to the file write request when the mapping information corresponding to the first logical address of the file write request is not stored; wherein, for the same logical address, the physical addresses allocated to each node of the cluster file system are different. The storage module is used to store the mapping information between the first logical address and the first physical address, as well as the timestamp of the record file being written.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

Images