A unified file transparent access method, device and storage medium based on heterogeneous backend segment mapping

Abstract

The application discloses a unified file transparent access method and device based on heterogeneous backend segmented mapping and a storage medium, and relates to the technical field of data processing. The method segments a logical file according to a segmented configuration rule and sets a capacity threshold, specifies a main segment, creates a main file carrying a logical file identifier in a corresponding backend of the main segment, generates a physical file name with unique pointing based on an inode number of the main file, a backend path of a subsequent segment and a segment serial number, and determines a logical offset interval according to the capacity threshold of each segment, thereby establishing a mapping relationship between a logical address space and a physical storage segment. When a vector read / write request is received, the logical file identifier and the pre-constructed segmented mapping relationship can be directly used to accurately locate the segment where target data is located and the corresponding heterogeneous backend, and when a non-vector read / write request is received, metadata in the main file is directly operated, so that the access response efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a unified transparent file access method, device and storage medium based on heterogeneous backend segmentation mapping. Background Technology

[0002] When a complete logical file is split into multiple parts and stored in different types of heterogeneous storage backends, when faced with access requests from the application, the traditional global addressing access method can only determine the target access file based on the file descriptor in the access request. Then, it traverses all backends according to the order set by the system, performs probing access on all backends, finds the target backend storing the target access file, and then performs the access operation on the target backend. This results in low access response efficiency for logical files distributed across heterogeneous backends.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a unified transparent file access method, device and storage medium based on heterogeneous backend segmentation mapping, which aims to solve the technical problem of how to improve the access efficiency of distributed logical files.

[0005] To achieve the above objectives, this application proposes a unified transparent file access method based on heterogeneous backend segmentation mapping, wherein the unified transparent file access method based on heterogeneous backend segmentation mapping includes: Based on the segmentation configuration rules corresponding to the logical file, determine each segment of the logical file and the capacity threshold of each segment; In each segment corresponding to the logical file, a main segment is specified, and in the backend corresponding to the main segment, a main file is created according to the logical file path of the logical file. The main file is used to carry the path, directory entries, basic attributes, extended attributes and first segment data of the logical file. The logical file identifier is used as the physical file name of the main file, and the physical file names of other segments are generated according to the inode number of the main file, the backend path corresponding to other segments, and the segment sequence number of the other segments, wherein the other segments are the subsequent segments of the main segment; Based on the capacity thresholds of the main segment and the other segments, the logical offset intervals of the main segment and the other segments are determined, and a segment mapping relationship of the logical file is created. The segment mapping relationship represents each segment of the logical file and the logical offset interval of each segment. When a vector read / write request for the logical file is received, the corresponding segments of the logical file are accessed based on the logical file identifier and the segment mapping relationship to execute the vector read / write request; When a non-vector read / write request is received for the logical file, the non-vector read / write request is executed within the main file.

[0006] In one embodiment, the step of accessing each segment corresponding to the logical file based on the logical file identifier and the segment mapping relationship to execute the vector read / write request when a vector read / write request for the logical file is received includes: Upon receiving a vector read request for the logical file, the segments corresponding to the logical file are determined based on the logical file identifier of the logical file. Based on the first logical offset and read length of the vector read request, and the segment mapping relationship, the target read segment corresponding to the vector read request is determined in each segment; When the number of target read segments is greater than 1, the vector read request is split into local vector read requests corresponding to each target read segment according to the logical offset interval of the target read segment; Each of the local vector read requests is sent to the backend corresponding to each of the target read segments for execution, so as to read the data in the backend corresponding to each of the target read segments.

[0007] In one embodiment, the logical offset interval includes a start position and an end position of the interval, and the step of splitting the vector read request into local vector read requests corresponding to each target read segment according to the logical offset interval of the target read segment includes: Based on the first logical offset and the segment mapping relationship, determine the first local offset within the read start segment of the target read segment; The first local offset is used as the logical offset of the local read vector corresponding to the read start segment, and the difference between the interval end position of the read start segment and the first local offset is used as the local read length of the local read vector corresponding to the read start segment. The sum of the first logical offset and the read length is taken as the first logical end position of the vector read request, and the local end offset within the read end segment in the target read segment is determined according to the first logical end position and the segment mapping relationship. The starting position of the interval of the read end segment is used as the logical offset of the local read vector corresponding to the read end segment, and the difference between the local end offset and the starting position of the interval of the read end segment is used as the local read length of the local read vector corresponding to the read end segment. The starting position of the interval of other target segments is used as the logical offset of the local read vector corresponding to the other target segments, and the capacity threshold of the other target segments is used as the local read length of the local read vector corresponding to the other target segments.

[0008] In one embodiment, the step of accessing each segment corresponding to the logical file based on the logical file identifier and the segment mapping relationship to execute the vector read / write request when a vector read / write request for the logical file is received includes: Upon receiving a vector write request for the logical file, the segments corresponding to the logical file are determined based on the logical file identifier of the logical file. Based on the second logical offset of the vector write request and the segment mapping relationship, the write start segment corresponding to the vector write request and the second local offset within the write start segment are determined in the segment; The sum of the write length of the vector write request and the second logical offset is used as the second logical end position of the vector write request; When the second logical end position is greater than the interval end position of the write start segment, the difference between the interval end position of the write start segment and the second logical offset is taken as the processing length of the write start segment. The second local offset is used as the logical offset of the local vector write request corresponding to the write start segment, and the processing length is used as the local write length of the local vector write request corresponding to the write start segment. Traverse the subsequent segments of the write start segment. If the capacity threshold of the current segment is less than the remaining write length, then take the interval start position of the current segment as the logical offset of the local vector write request corresponding to the current segment, and take the capacity threshold of the current segment as the local write length of the local vector write request corresponding to the current segment. If the capacity threshold of the current segment is greater than or equal to the remaining write length, then the starting position of the interval of the current segment is used as the logical offset of the local vector write request corresponding to the current segment, and the remaining write length is used as the local write length of the local vector write request corresponding to the current segment, wherein the remaining write length is the difference between the write length and the sum of each local write length.

[0009] In one embodiment, the main file includes the logical file path entry, directory entry, basic attribute entry, and header data of the logical file. The unified transparent file access method based on heterogeneous backend segment mapping further includes: When an attribute query request for the logical file is received, the attribute information stored in the backend corresponding to the main segment is read according to the basic attribute entry of the main file. The actual length of the other segments is queried in reverse order, and the existence of valid data in the other segments is determined based on the actual length. The first segment with valid data determined in the reverse query is taken as the tail segment, and the sum of the capacity thresholds of all preceding segments of the tail segment is determined. The total length of the logical file is obtained by summing the capacity thresholds of all preceding segments of the tail segment and adding them to the actual length of the tail segment. The attribute information and the total length of the logical file are used as the query results.

[0010] In one embodiment, after the step of reversing the query to determine the actual length of the other segments and determining whether there is valid data in the other segments based on the actual length, the method further includes: If no valid data exists in the other segments, the attribute information will be used as the query result.

[0011] In one embodiment, the unified file transparent access method based on heterogeneous backend segmentation mapping further includes: Upon receiving a deletion request for the logical file, determine the inode number of the main file corresponding to the logical file; Determine the physical file in the backend corresponding to the segment associated with the inode number; Delete the main file and the physical file.

[0012] In one embodiment, the unified file transparent access method based on heterogeneous backend segmentation mapping further includes: Upon receiving a renaming request for the logical file, the filename of the main file corresponding to the logical file is modified; The inode number of the main file is used to determine the corresponding physical files, and the filenames of the corresponding physical files are modified.

[0013] Furthermore, to achieve the above objectives, this application also proposes a unified file transparent access device based on heterogeneous backend segmentation mapping. The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer program is configured to implement the steps of the unified file transparent access method based on heterogeneous backend segmentation mapping as described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the unified file transparent access method based on heterogeneous backend segmentation mapping as described above.

[0015] This application provides a unified transparent file access method based on heterogeneous backend segmentation mapping. The method first segments logical files according to segmentation configuration rules and sets capacity thresholds. It then specifies a main segment and creates a main file carrying a logical file identifier on its corresponding backend. Finally, it generates a unique physical filename based on the inode number of the main file, the backend path of subsequent segments, and the segment sequence number, eliminating ambiguity in the physical storage location. Simultaneously, it determines the logical offset range of each segment based on its capacity threshold, establishing a mapping relationship between the logical address space and the physical storage segments. When a vector read / write request is received, the method can directly and accurately locate the segment containing the target data and its corresponding heterogeneous backend based on the logical file identifier and the pre-built segmentation mapping relationship. When a non-vector read / write request is received, it directly operates on the metadata in the main file, allowing parallel or targeted access to multiple segments without trial-and-error traversal. This achieves efficient access to logical files distributed across heterogeneous backends, significantly improving access response efficiency. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating an embodiment of the unified transparent file access method based on heterogeneous backend segmentation mapping provided in this application. Figure 2 This is a schematic diagram of the segmented mapping relationship provided in Embodiment 1 of this application; Figure 3 This is a schematic diagram illustrating the creation and opening process of a physical file as provided in Embodiment 1 of this application; Figure 4 This is a flowchart illustrating Embodiment 2 of the unified transparent file access method based on heterogeneous backend segmentation mapping in this application. Figure 5This is a flowchart illustrating Embodiment 3 of the unified transparent file access method based on heterogeneous backend segmentation mapping in this application; Figure 6 This is a schematic diagram of the data reading process provided in Embodiment 3 of this application; Figure 7 This is a schematic diagram of the data writing process provided in Embodiment 3 of this application; Figure 8 This is a schematic diagram of the attribute query process for a logical file provided in Embodiment 4 of this application; Figure 9 This is a schematic diagram of the logical file deletion process provided in Embodiment 5 of this application; Figure 10 This is a schematic diagram of the logical file renaming process provided in Embodiment 5 of this application; Figure 11 This is a schematic diagram of the file access system architecture provided in Embodiment 5 of this application; Figure 12 This is a schematic diagram of the device structure of the hardware operating environment involved in the unified file transparent access method based on heterogeneous backend segmentation mapping in the embodiments of this application.

[0019] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of this application and are not intended to limit this application.

[0021] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments. It should be noted that all actions involving the acquisition of signals, information, or data in this application are performed in accordance with the relevant data protection laws and regulations of the country where the application is located, and with authorization from the owner of the corresponding device.

[0022] When a complete logical file is split into multiple parts and stored in different types of heterogeneous storage backends, when faced with access requests from the application, the traditional global addressing access method can only determine the target access file based on the file descriptor in the access request. Then, it traverses all backends according to the order set by the system, performs probing access on all backends, finds the target backend storing the target access file, and then performs the access operation on the target backend. This results in low access response efficiency for logical files distributed across heterogeneous backends.

[0023] In view of the above problems, this application proposes a unified transparent file access method based on heterogeneous backend segmentation mapping. This method first segments logical files according to segmentation configuration rules and sets capacity thresholds. It then specifies the main segment and creates a main file carrying a logical file identifier on its corresponding backend. Finally, it generates a unique physical filename based on the inode number of the main file, the backend path of subsequent segments, and the segment sequence number, eliminating the ambiguity of the physical storage location. Simultaneously, it determines the logical offset range of each segment according to its capacity threshold, establishing a mapping relationship between the logical address space and the physical storage segments. When a vector read / write request is received, it can directly and accurately locate the segment containing the target data and its corresponding heterogeneous backend based on the logical file identifier and the pre-built segmentation mapping relationship. Multiple segments can be accessed in parallel or in a targeted manner without trial traversal, thus achieving efficient access to logical files distributed across heterogeneous backends and significantly improving access response efficiency.

[0024] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or file access system capable of performing the above functions. The following embodiments will be described using a file access system as an example.

[0025] First Embodiment The first embodiment of this application provides a unified transparent file access method based on heterogeneous backend segmentation mapping, referring to... Figure 1 In this embodiment, the unified file transparent access method based on heterogeneous backend segmentation mapping includes steps S10~S50: Step S10: Determine each segment of the logical file and the capacity threshold of each segment according to the segmentation configuration rules corresponding to the logical file.

[0026] Step S20: Specify the main segment in each segment corresponding to the logical file, and create the main file in the backend corresponding to the main segment according to the logical file path of the logical file.

[0027] A logical file is the file view seen and used by the application. It corresponds to a complete logical file path, such as " / data / bigfile.log". It is actually composed of physical files distributed across multiple heterogeneous storage extents. The application only perceives this logical file and does not need to know its underlying segment layout. Segmentation configuration rules are preset for logical files and are a set of rules used to divide each segment. These rules include, but are not limited to, the capacity threshold, logical offset range, backend type, backend path, driver initialization parameters, segment sequence numbering rules, and main segment specification rules for each segment.

[0028] Preferably, the segmentation configuration rule can be expressed in a concatenated manner as "TYPE:SIZE:DATA", where TYPE represents the backend type, SIZE represents the capacity threshold (different values ​​can be set according to different backend storage characteristics), and DATA represents the corresponding backend path or driver initialization parameters. Multiple segments can be connected by a separator. The first preset number of segments can be set with a fixed capacity threshold, and the last segment can be set to unlimited length to accommodate the remaining data exceeding the capacity threshold range of the preceding segments.

[0029] For example, the main segment can be determined in the segment configuration rules according to the main segment specification rules, or the specified segment can be used as the main segment in response to the segment specification operation received by the system.

[0030] Preferably, the main segment carries the logical path entry, directory entry, and basic attribute entry of the logical file. The application can access all core metadata of the logical file solely through the main segment, without needing to be aware of the multi-backend layout, maintaining interface transparency. Subsequent segments of the main segment carry data segments at different offset intervals of the logical file. Based on the metadata of the main segment, the physical location of subsequent segments can be directly deduced, eliminating the need for an additional central metadata service, reducing system complexity. Furthermore, with basic attributes concentrated in the main segment, it is easy to quickly reconstruct the overall attributes of the logical file, improving attribute query efficiency. The data of the logical file is distributed across subsequent segments of the main segment according to logical offset intervals, allowing for flexible matching of the performance and capacity characteristics of different backends, balancing storage performance and cost.

[0031] The main file refers to the actual storage object where the main segment of a logical file is located in its corresponding backend. It is the physical carrier of the main segment, containing the logical file's path, directory entries, basic attributes, extended attributes, and header data. For example, the logical file path of the logical file can be used as the filename of the main file, and the main file can be created in the backend corresponding to the main segment.

[0032] Step S30: Use the logical file identifier as the physical file name of the main file, and generate the physical file names corresponding to other segments based on the inode number of the main file, the backend paths corresponding to other segments, and the segment sequence numbers of the other segments.

[0033] The inode number of the main file is used to uniquely identify the main file. Preferably, the inode of the logical file can be used as the inode number of the main file. Then, the backend paths corresponding to other segments are obtained, along with the segment numbers of each other segment, to generate the physical file names corresponding to the other segments. This ensures that each physical file name is strongly associated with the main segment and is unique. The other segments are subsequent segments of the main segment.

[0034] Optionally, based on the physical filenames of the other segments determined above, each physical file is created in the corresponding backend of each other segment. The physical file is the actual storage object where the segment is stored in its corresponding backend, used to carry the data segment of the offset range corresponding to the logical file. Alternatively, when writing data to the logical file, physical files can be created in each other segment based on the physical filenames of the other segments determined above.

[0035] Step S40: Based on the capacity thresholds of the main segment and the other segments, determine the logical offset intervals of the main segment and the other segments, and create the segment mapping relationship of the logical file.

[0036] Step S50: When a vector read / write request for the logical file is received, the corresponding segments of the logical file are accessed based on the logical file identifier and the segment mapping relationship to execute the vector read / write request.

[0037] When a vector read / write request for the logical file is received, the main file can be directly determined based on the logical file. After determining the physical file corresponding to each segment based on the inode number of the main file, the backend path and corresponding segment number of the backend to which each physical file belongs are determined. This allows the actual physical backend where the corresponding data segment of the logical file is located to be determined based on the backend path and corresponding segment number of the backend to which each physical file belongs when accessing each physical file in the future, thereby improving file access efficiency.

[0038] Step S60: When a non-vector read / write request for the logical file is received, the non-vector read / write request is executed within the main file.

[0039] Non-vector read / write requests include rename requests, directory creation requests, directory query requests, directory deletion requests, attribute query requests, attribute deletion requests, attribute change requests, file locking, and file unlocking; among them, attribute query requests may include file permission mode queries, file owner queries, and file time queries.

[0040] Since the metadata of logical files, such as paths, directory entries, basic attributes, and extended attributes, are all stored in the main file, when the above non-vector read / write requests are received, the relevant operations can be performed directly in the main file, thus speeding up file access.

[0041] For example, please refer to Figure 2 , Figure 2 A schematic diagram of the segmentation mapping relationship of a logical file is provided, specifically: The logical file is split into a main file, a second segment physical file, and a third segment physical file, stored in the backends corresponding to the main segment, second segment, and third segment, respectively. The main file uses the logical file path " / data / fileA" as its filename and is displayed in the application as a logical file view. The backend of the main segment containing the main file stores the first segment data and metadata entries of the logical file, namely the aforementioned logical path entry, directory entry, and basic attribute entry. The logical file identifier Index Node is 'I', and this identifier is used as the inode number of the main file. In the second segment, the backend path corresponding to the second segment is used as the target path, and "target path + I + segment number 2" is used as the filename of the second segment physical file. In the third segment, the backend path corresponding to the third segment is used as the target path, and "target path + I + segment number 3" is used as the filename of the third segment physical file.

[0042] Further, please refer to Figure 3 , Figure 3 A flowchart illustrating the creation and opening process of physical files is provided, specifically: First, the segmentation configuration rules are parsed. Based on these rules, a primary segment is selected from each segment, and a primary file is created or opened within that primary segment. Then, the inode number of the primary file is obtained. Using the inode number, the segment sequence number of the other segments, and the backend path of the corresponding backend for each other segment, physical filenames within those other segments are generated. Each physical file is then created or opened in its respective backend. Finally, the file descriptors of the primary and physical files are aggregated to obtain an aggregated file handle, which is returned to the application as the file descriptor of the logical file. Alternatively, when the disk I / O overhead of creating or opening a new physical file is significant, it can be configured to create or open the physical file only when data is written to it, thus saving backend resources. The file descriptors of the primary and physical files can be pre-configured. In this embodiment, "other segments" refers to segments following the primary segment.

[0043] In this embodiment, a primary segment is first designated from each segment according to the segmentation configuration rules, serving as the unified metadata entry point for the logical files. Then, in the backend corresponding to the primary segment, a master file is created directly using the original logical file path of the logical file. This master file carries the core metadata of the logical file, such as directory entries and attributes, presenting a complete logical file view to the application. Next, the system reads the unique identifier of this master file and uses it as a seed, combined with the preset backend paths and segment numbers of other segments, to batch generate filenames for all other physical files corresponding to each segment, and creates these physical files in their respective backends. Finally, the system summarizes and records the correspondence between all physical files and their respective backends and segment numbers, thus forming a complete mapping relationship between logical files and segments. These steps enable the system to locate the storage object within the specific backend where the target data resides when receiving an access request from the application, without performing time-consuming global lookups or negotiations. It only needs to calculate the local offset of that storage object based on the logical offset of the request and the known segmentation capacity rules, improving access efficiency across heterogeneous backend storage.

[0044] Second Embodiment Based on the first embodiment described above, in this embodiment, referring to... Figure 4 The unified file transparent access method based on heterogeneous backend segmentation mapping proposed in this embodiment further includes steps S51~S54: Step S51: When a vector read request for a logical file is received, the segments corresponding to the logical file are determined according to the logical file identifier of the logical file.

[0045] A vector read request is a read operation request initiated by the application to a logical file, also known as a preadv request. Its format can be represented as preadv(fd, buf, len, offset). Here, fd represents the file descriptor, corresponding to a unique logical file; buf is the data buffer, an array of memory blocks into which data will be read from the backend; len is the read length; and offset is the logical offset. The segment mapping relationship of the logical file represents the various segments of the logical file and the logical offset range of each segment.

[0046] For example, suppose a 7GB logical file has 4 segments, and the capacity threshold of each of the 4 segments is 3GB, then: Segment 1, with a logical offset range of 0-f1, stores data up to 3GB from the beginning of the logical file; Segment 2, with a logical offset range of f1-f2, stores data from the 3GB position of the logical file to the 6GB position; Segment 3, with a logical offset range of f2-f3, stores data from the 6GB position of the logical file to the 7GB position; Segment 4, with a logical offset range of f3-f4, does not store logical file data.

[0047] After receiving the above vector read request, the file descriptor, i.e. the logical file identifier, can be obtained from the vector read request. Therefore, the physical file name in each backend corresponding to the logical file can be directly located from the file descriptor, and then the segment can be located.

[0048] Step S52: Based on the first logical offset and read length of the vector read request, and the segment mapping relationship, determine the target read segment corresponding to the vector read request in each segment.

[0049] It should be noted that the first logical offset corresponding to a vector read request refers to the global offset relative to the beginning of the entire logical file, specified by the application when initiating the vector read request. For example, suppose the system receives a read request from the application as preadv(fd,buf,len,0), where 0 is the first logical offset, representing reading from the beginning of the entire logical file.

[0050] For example, determining the target read segment corresponding to the vector read request based on the first logical offset and read length of the vector read request, as well as the segment mapping relationship, specifically includes the following steps: First, determine the first local offset within the read start segment of the target read segment based on the first logical offset and the segment mapping relationship; then, take the sum of the first logical offset and the read length as the first logical end position of the vector read request. If the first logical end position is greater than the interval end position of the read start segment, it indicates that the number of target read segments is greater than 1, that is, the request range of the vector read request exceeds the capacity threshold of the read start segment, and data in the subsequent segments of the read start segment still needs to be read. At this time, based on the segment mapping relationship, determine the read start segment corresponding to the first logical offset and the read end segment corresponding to the read end position, and take the read start segment, the read end segment, and all segments in between as the target read segment. If the first logical end position is less than the interval end position of the read start segment, it indicates that the request range of the vector read request is within the logical region interval of the read start segment, and the target read segment is the read start segment.

[0051] Step S53: When the number of target read segments is greater than 1, the vector read request is split into local vector read requests corresponding to each target read segment according to the logical offset interval of the target read segment.

[0052] When it is determined that the scope of a vector read request exceeds the capacity threshold of the initial read segment, and data needs to be read from subsequent segments of the initial read segment, it indicates that data needs to be read from multiple different backends. In this case, the vector read request is split into local vector read requests corresponding to each target read segment, so that the local vector read requests can be directly sent to the backend corresponding to the target read segment for execution, reducing the time and I / O overhead caused by accessing all backends and improving the response efficiency of the access.

[0053] Furthermore, step S53 above includes steps S531 to S535: Step S531: Based on the first logical offset and the segment mapping relationship, determine the first local offset within the read start segment of the target read segment.

[0054] For example, after receiving a vector read request, the system determines the capacity threshold and logical offset range of each segment corresponding to the logical file according to the segment mapping relationship, matches the first logical offset with the logical offset range of each segment, and determines the first local offset within the read start segment to which the first logical offset belongs.

[0055] To better understand the above example, let's further explain it in a specific application scenario. Assume the logical file corresponds to two segments. Segment 1 has a capacity threshold of 10GB, with a corresponding logical offset range of 0~10GB; segment 2 has unlimited length, with a corresponding logical offset range of 10GB~∞. When the application initiates a predv read request, and the first logical offset corresponding to this predv read request is 6GB, the system matches this first logical offset of 6GB with the logical offset ranges of segment 1 and segment 2 one by one. It determines that the first logical offset of 6GB falls within the logical offset range of segment 1, thus identifying segment 1 as the starting segment for this predv read request, and determining that the first local offset of this predv read request within the starting segment is 6GB.

[0056] Step S532: The first local offset is used as the logical offset of the local read vector corresponding to the read start segment, and the difference between the interval end position of the read start segment and the first local offset is used as the local read length of the local read vector corresponding to the read start segment.

[0057] Referring to the example above, the first local offset of 6GB is used as the logical offset of the local read vector preadv1 corresponding to the read start segment. The difference between the end position of the read start segment (i.e., segment 1) of 10GB and the first local offset of 6GB is used as the local read length of the local read vector preadv1, which is 4GB.

[0058] Step S533: The sum of the first logical offset and the read length is taken as the first logical end position of the vector read request, and the local end offset within the read end segment in the target read segment is determined according to the first logical end position and the segment mapping relationship.

[0059] The first logical end position is calculated based on the first logical offset of the vector read request. It is the termination position of the read operation in the entire logical file and is used to determine whether the vector read request crosses the end position of the interval of the read start segment. The first logical end position is represented by a global offset relative to the beginning of the logical file. The step of determining the local end offset within the read end segment based on the first logical end position and the segment mapping relationship is consistent with the principle of determining the first local offset within the read start segment in step S531 above, and will not be repeated here.

[0060] The end position of a segment is the sum of the preceding segment and the capacity threshold of the segment.

[0061] For example, suppose the logical file is divided into segments 1 through 10, and each segment has a capacity threshold of 10GB. If the read start segment corresponding to the vector read request of this logical file is segment 3, then the end position of the read start segment interval is the sum of the capacity thresholds of segments 1, 2, and 3, which is 30GB. If the read start segment is the first segment, i.e., segment 1, meaning the start segment has no preceding segments, then the end position of the read start segment interval is the capacity threshold of the read start segment.

[0062] For example, assuming the read length corresponding to the predv read request in the above example is 10GB, then the first logical end position of the predv read request is 10GB+6GB=16GB, the corresponding read end segment is segment 2, and the local end offset is 16GB.

[0063] Step S534: The start position of the interval of the read end segment is used as the logical offset of the local read vector corresponding to the read end segment, and the difference between the local end offset and the start position of the interval of the read end segment is used as the local read length of the local read vector corresponding to the read end segment.

[0064] Referring to the example above, the starting position of the interval of the end-of-read segment, i.e., segment 2, is 10GB, which is taken as the logical offset of the local read vector preadv2 corresponding to the end-of-read segment. The local end offset and the starting position of the interval of segment 2 are taken as the local read length of the local read vector preadv2, which is 6GB.

[0065] Step S535: Use the start position of the interval of other target segments as the logical offset of the local read vector corresponding to the other target segments, and use the capacity threshold of the other target segments as the local read length of the local read vector corresponding to the other target segments.

[0066] If there are other target segments between the start and end read segments, the start position of the interval of the other target segments is directly used as the logical offset of their corresponding local read vectors, and the capacity threshold of the other target segments is used as the read length of their corresponding local read vectors. Here, "other target segments" refers to target read segments other than the start and end read segments mentioned above.

[0067] Through the above steps S531~S535, the vector read request of the logical file can be split into local read request vectors for each target read segment, and the logical offset and local read length of each local read request vector can be determined so that the backend corresponding to each target read segment can read the data stored in each backend according to the logical offset and local read length of the local read request vector and store it in the specified data buffer.

[0068] Step S54: Send each of the local vector read requests to the backend corresponding to each of the target read segments for execution, so as to read the data in the backend corresponding to each of the target read segments.

[0069] Furthermore, the system can receive the byte data returned by the backend corresponding to the target read segment, and concatenate the returned byte data according to the segment number order of each target segment as the result, and return it to the application end to be stored in the specified data buffer.

[0070] This embodiment utilizes the segmented mapping relationship of logical files to quickly convert the first logical offset in a vector read request into a first local offset within a specific read start segment, thereby directly locating the actual storage location of the first data block. Next, by calculating the first logical end position of the vector read request and comparing it with the interval end position of the read start segment, it is determined whether the vector read request spans multiple segments. If a span occurs, the original global vector read request is split into a series of local vector read requests precisely corresponding to different target read segments, based on the logical offset intervals of each segment. Finally, these local vector read requests are directly distributed to the corresponding heterogeneous backends for execution. This method fundamentally transforms a complex I / O operation requiring global addressing into a simple I / O operation directly distributed to a specific backend, thereby significantly improving the system's response efficiency for accessing distributed storage on heterogeneous backends.

[0071] Third Embodiment Based on the first and second embodiments described above, in this embodiment, referring to... Figure 5The unified file transparent access method based on heterogeneous backend segmentation mapping proposed in this embodiment further includes steps S55~S511: Step S55: When a vector read request for the logical file is received, the segments corresponding to the logical file are determined according to the logical file identifier of the logical file.

[0072] Step S56: Based on the second logical offset of the vector write request and the segment mapping relationship, determine the write start segment corresponding to the vector write request and the second local offset within the write start segment in the segment.

[0073] Step S57: The sum of the write length of the vector write request and the second logical offset is used as the second logical end position of the vector write request.

[0074] A vector read request is a write operation request initiated by the application to a logical file, i.e., a pwritev request, whose format can be represented as pwritev(fd, buf, len, offset). Here, fd represents the file descriptor, corresponding to a unique logical file; buf is the data buffer, indicating that the data in the data buffer will be written to the backend; len is the write length; and offset is the logical offset. Similarly, after receiving the above vector write request, the file descriptor, i.e., the logical file identifier, can be obtained from the vector write request. This file descriptor can then be used to directly locate the physical filenames in the respective backends corresponding to the logical file, and thus locate the respective segments.

[0075] The steps for determining the second local offset are similar to those for determining the first local offset in step S531, and the steps for determining the second logic end position are similar to those for determining the first logic end position in step S533. Therefore, they will not be described in detail here.

[0076] Step S58: When the second logical end position is greater than the interval end position of the write start segment, the difference between the interval end position of the write start segment and the second logical offset is used as the processing length of the write start segment.

[0077] When the second logical end position of a vector write request is greater than the end position of the interval of the write start segment, it means that the request range of the vector write request exceeds the capacity threshold of the write start segment, and the remaining data needs to be written to the backend corresponding to the subsequent segment of the write start segment.

[0078] The processing length of the write start segment refers to the number of bytes that the write start segment carries for the vector write request, that is, the amount of data to be written for the local vector write request corresponding to the write start segment.

[0079] For example, after determining the end position of the write start segment interval and the second logical offset, the difference between the end position of the write start segment interval and the second logical offset is used as the processing length of the write start segment, that is, the processing length of the write start segment = end position of the write start segment interval - second logical offset.

[0080] Step S59: Use the second local offset as the logical offset of the local vector write request corresponding to the write start segment, and use the processing length as the local write length of the local vector write request corresponding to the write start segment.

[0081] Once the processing length of the write start segment and the second local offset within the write start segment are determined, the local vector read request corresponding to the write start segment can be determined. That is, within the write start segment, writing begins from the second local offset within the write start segment, and the number of data bytes written is equal to the processing length of the write start segment.

[0082] Step S510: Traverse the subsequent segments of the write start segment. If the capacity threshold of the current segment is less than the remaining write length, then take the interval start position of the current segment as the logical offset of the local vector write request corresponding to the current segment, and take the capacity threshold of the current segment as the local write length of the local vector write request corresponding to the current segment.

[0083] Step S511: If the capacity threshold of the current segment is greater than or equal to the remaining write length, then the starting position of the interval of the current segment is used as the logical offset of the local vector write request corresponding to the current segment, and the remaining write length is used as the local write length of the local vector write request corresponding to the current segment, wherein the remaining write length is the difference between the write length and the sum of each local write length.

[0084] For example, suppose the second logical offset corresponding to the vector write request of the logical file is denoted as O, the write length is denoted as S, and the end position of the interval of the write start segment extent0 is denoted as E0. Then the logical end position O_end of the vector write request = O + S. Compare it with the end position E0 of the interval of the write start segment. If O_end > E0, it means that the vector read request needs to write part of the data to the write start segment and the other part to the subsequent segment of the write start segment, triggering the vector write request split.

[0085] Furthermore, the specific steps for splitting vector write requests include the following: Step 1: Determine the logical end position of the write start segment extent0 = E0, and determine the processing length that extent0 can handle = E0 - 0. Step 2: Use the second local offset corresponding to extent0 as the logical offset of the local vector write request pwritev0, and use the processing length that extent0 can handle as the local write length of pwritev0. Send this local vector write request pwritev0 to the backend corresponding to extent0 for execution. Step 3: Update the remaining write length of the vector write request = S - (E0 - 0). Step 4: Traverse the subsequent segments of extent0, and use its first subsequent segment extent1 as the current segment. If the capacity threshold T0 of the current segment is less than the remaining write length, then use the start position of the current segment extent1, i.e., the end position E0 of extent0, as the logical offset of the local vector write request pwritev1 corresponding to extent1. The capacity threshold T0 of the current segment extent1 is used as the local write length of the local vector write request pwritev2 corresponding to the current segment extent1. The difference S - [(E0 - 0) + T0] between the remaining length and the sum of all local write lengths is used as the new remaining length. If the capacity threshold T0 of the current segment extent1 is greater than or equal to the remaining write length, the start position of the interval of the current segment extent1, i.e., the end position E0 of the interval of extent0, is used as the logical offset of the local vector write request pwritev1 corresponding to the current segment extent1. The remaining write length S - (E0 - 0) is used as the local write length of the local vector write request pwritev2 corresponding to the current segment extent1. The local vector write request pwritev2 is then sent to the backend corresponding to the current segment extent1 for execution. Here, the remaining write length represents the write length of the vector read request minus the local write length already allocated to the local vector write request, i.e., the number of bytes not yet allocated.

[0086] Furthermore, to aid in understanding the implementation flow of the unified file transparent access method based on heterogeneous backend segmentation mapping obtained in the third embodiment in combination with the first and second embodiments described above, it is assumed that the physical files in the other segments are created when writing data to a logical file. Please refer to... Figure 6 , Figure 6 A schematic diagram illustrating the data reading process of a unified transparent file access method based on heterogeneous backend segmentation mapping is provided, specifically: The system first receives a vector read request from the application. Based on the logical file identifier in the vector read request, it locates the segments of the logical file. Then, it locates the starting segment based on the logical offset corresponding to the vector read request and calculates the local offset within the starting segment. Next, based on the logical end position of the vector read request and the interval end position of the starting segment, it determines whether the vector read request crosses the logical boundary of the starting segment, i.e., the logical offset interval. If it is determined that the vector read request crosses the logical boundary of the starting segment and the physical file is already open, then based on the logical boundaries of the starting segment and its subsequent segments, the vector read request is split according to the segment intervals, resulting in local vectorized I / O structures corresponding to each target segment, i.e., the aforementioned local vector read requests. These local vectorized I / O structures are then sent synchronously serially or asynchronously in parallel to the corresponding backends for execution. If the vector read request does not cross the logical boundary of the starting segment and the physical file is already open, then the vector read request, i.e., the original vectorized I / O structure, is directly sent to the backend corresponding to the starting segment for execution. If the physical file is not open, meaning no data has been written to the logical file, then 0 is returned directly, and subsequent segments of the main segment are not accessed. After each segment is executed, the execution results are obtained, summarized, and returned to the application.

[0087] Please refer to Figure 7 , Figure 7 A schematic diagram of the write data flow for a unified transparent file access method based on heterogeneous backend segmentation mapping is provided, specifically: The system first receives a vector write request from the application. Based on the logical file identifier in the vector write request, it locates the segments of the logical file. Then, based on the logical offset corresponding to the vector write request, it locates the starting segment and calculates the local offset within the starting segment. Next, based on the logical end position of the vector write request and the interval end position of the starting segment, it determines whether the vector write request crosses the logical boundary of the starting segment, i.e., the logical offset interval. If it is determined that the vector write request crosses the logical boundary of the starting segment and the physical file is already open, then based on the logical boundaries of the starting segment and its subsequent segments, the vector write request is split according to the segment intervals, resulting in local vectorized I / O structures corresponding to each target segment, i.e., the aforementioned local vector write requests. These local vectorized I / O structures are then sent synchronously serially or asynchronously in parallel to the corresponding backends for execution. If the vector write request does not cross the logical boundary of the starting segment and the physical file is already open, then the vector write request, i.e., the original vectorized I / O structure, is directly sent to the backend corresponding to the starting segment for execution. If the physical file is not open, create and open a new physical file, or open a previously created physical file, and then perform the data writing steps described above. After each segment is executed, obtain the execution results and summarize them to return to the application.

[0088] Fourth embodiment Based on the above embodiments, in the unified file transparent access method based on heterogeneous backend segmentation mapping proposed in this embodiment, the main file includes the logical file path entry, directory entry, basic attribute entry, and first segment data of the logical file.

[0089] The logical file path entry refers to the identifier pointing to the logical file path stored in the main file. It is the only global path entry point for applications to access logical files. This logical file path entry is exposed as a single logical file path, such as " / data / file.log". It is associated with the backend path of the main segment at the underlying level. Applications only need to initiate all operations through this logical file path and do not need to be aware of the underlying physical path layout of multiple backends. The directory entry is a data structure that associates logical files with their respective directories stored in the main file. It is used to represent the mapping relationship of "logical file path - inode number of the main file - physical file name". When an application accesses a logical file through a logical file path, the system can quickly find the corresponding inode number from the directory through this directory entry, and then locate the physical file. The basic attribute entry is a fixed data area specially defined in the main file. It is a dedicated entry point for storing the basic metadata of logical files. The basic metadata of logical files includes, but is not limited to, the creation time, modification time, access time, file access permissions, file owner / group, and file type. These basic attribute information are global basic attributes of logical files and are uniformly displayed to applications, without changing with the distribution of subsequent segments. The first data segment is the actual byte data stored in the main segment at the beginning of the logical file; it is the first stored part of the logical file's data content.

[0090] Based on this, in the unified file transparent access method based on heterogeneous backend segmentation mapping proposed in this embodiment, step S60 above further includes steps S61 to S65: Step S61: When the attribute query request of the logical file is received, the attribute information stored in the backend corresponding to the main segment is read according to the basic attribute entry of the main file corresponding to the logical file.

[0091] The attribute query request is initiated by the application to retrieve the basic attributes, extended attributes, and overall length of a logical file. Since the basic attributes of the logical file are already stored in the main file, the main file corresponding to the main segment of the logical file can be determined first based on the logical file path associated with the attribute query request. Then, based on the basic attribute entry point of the main file, the attribute information stored in the backend corresponding to the main segment can be read.

[0092] Step S62: Query the actual length of the other segments in reverse order, and determine whether there is valid data in the other segments based on the actual length.

[0093] The actual length of other segments refers to the number of bytes of valid data actually stored in the physical file corresponding to the subsequent segments of the main segment. For example, all other segments of the logical file are determined, and the query is performed in reverse order from the last subsequent segment according to the segment number of each other segment. The actual length of the currently queried other segments is determined, and the existence of valid data in the subsequent segments of each main segment is checked one by one, using whether the actual length is greater than 0 as the criterion, until the first other segment with an actual length greater than 0 is found. In this embodiment, "other segments" refers to the subsequent segments of the main segment.

[0094] Step S63: The first segment with valid data determined in the reverse query is taken as the tail segment, and the sum of the capacity thresholds of all preceding segments of the tail segment is determined.

[0095] If other segments with valid data are found, the first segment with an actual length greater than 0 in reverse order is identified as the tail segment. Next, a preset capacity threshold for all preceding segments of the tail segment is determined, and the capacity thresholds of all preceding segments of the tail segment are summed to obtain the sum of the capacity thresholds of all preceding segments of the tail segment.

[0096] Step S64: The sum of the capacity thresholds of all preceding segments of the tail segment is added to the actual length of the tail segment to obtain the total length of the logical file.

[0097] The total length of the logical file is obtained by summing the capacity thresholds of all preceding segments of the tail segment calculated above with the actual length of the tail segment. The formula is: Total length of logical file = Sum of capacity thresholds of all preceding segments of the tail segment + Actual length of the tail segment.

[0098] Step S65: Use the attribute information and the total length of the logical file as the query result.

[0099] The attribute information read above is integrated with the calculated total length of the logical file to form a complete query result, which is then returned to the application.

[0100] Furthermore, after determining the total length of the logical file, the result of dividing the total length of the logical file by the system standard block size can be rounded up to obtain the number of logical blocks in the logical file. The number of logical blocks, attribute information, and total length of the logical file can then be used as query results and returned to the application.

[0101] Optionally, if step S70 finds that none of the subsequent segments of the main segment have valid data, then there is no need to determine the tail segment. The attribute information of the main segment is used directly as the query result and returned to the application.

[0102] For example, to help understand the implementation process of the unified transparent file access method based on heterogeneous backend segmentation mapping obtained in this embodiment in combination with the above embodiments, please refer to... Figure 8 , Figure 8 A schematic diagram of the attribute query process for a logical file is provided, specifically: First, the attribute information of the main file stored in the main segment is obtained. Then, starting from the last subsequent segment of the main segment, the actual length of each subsequent segment is checked in reverse order to determine whether valid data is detected in the currently checked subsequent segment. If valid data is detected in the currently checked subsequent segment, the currently checked subsequent segment is taken as the tail segment, and the actual length of the tail segment is summed with the capacity threshold of all preceding segments of the tail segment to obtain the total length of the logical file. Then, the number of logical blocks is calculated based on the total length of the logical file, and the number of logical blocks, attribute information, and total length of the logical file are used as query results and returned to the application. If no valid data is detected in the currently checked subsequent segment, it is determined whether the currently checked subsequent segment has any preceding segments. If so, its preceding segments are traversed in a loop to continue the detection and calculation. If not, the attribute information of the main file stored in the main segment is directly used as the query result and returned to the application.

[0103] Fifth embodiment Based on the above embodiments, in the unified file transparent access method based on heterogeneous backend segmentation mapping proposed in this embodiment, refer to Figure 9 When a deletion request for a logical file is received, the inode number of the main file corresponding to the logical file is determined, then the physical file in the backend corresponding to the segment sequence number associated with the inode number is determined, and then the main file and the physical file are deleted.

[0104] Because the physical file name contains the inode number of the main file, when the system receives a deletion request for a logical file from the application, it can first determine the main segment through the logical file path associated with the deletion request, obtain the inode number of the main file within the main segment, and then determine the physical location information of the subsequent segments based on the inode number and segment sequence number. This allows for the coordinated deletion of physical files in both the main segment and its subsequent segments. After deletion, the system can record the result and return it to the application.

[0105] Similarly, refer to Figure 10 When a renaming request for a logical file is received, the system first modifies the filename of the corresponding main file, thereby synchronously modifying the filenames of all physical files corresponding to the main file to maintain consistency in the multi-backend mapping relationship. After renaming is completed, the system can record the exception or compensation processing results and return the renaming result to the application.

[0106] Furthermore, when the system receives a request to release, close, or synchronize a logical file, it can perform unified processing on multiple segment handles in the aggregate file handle, and perform release, close, or synchronization operations on the physical file of the logical file in parallel. This allows the application to maintain only a single aggregate file handle, while the underlying layer completes the joint scheduling of multiple backend resources.

[0107] For example, to help understand the implementation process of the unified transparent file access method based on heterogeneous backend segmentation mapping obtained by combining this embodiment with the above embodiments, please refer to... Figure 11 , Figure 11 A system architecture diagram of a file access system is provided, specifically: The file access system may include a unified interface module, a segmented configuration parsing module, a multi-backend driver management module, an aggregated file handle module, a logical attribute reconstruction module, a cross-segment input / output splitting module, and a consistency maintenance module.

[0108] The unified interface module receives file system call requests from the application. The segment configuration parsing module parses segment configuration rules such as backend type, capacity threshold, and backend path corresponding to multiple segments. The multi-backend driver management module initializes and manages the driver context corresponding to each segment. The aggregated file handle module establishes a set of multi-segment handles corresponding to a single logical file. The logical attribute reconstruction module reconstructs the logical file length and number of logical blocks based on the capacity threshold and actual length of each subsequent segment of the main segment. The cross-segment input / output splitting module splits the vector read / write request according to the logical boundaries of the starting segment and its subsequent segments, obtaining local vector read / write requests corresponding to each target segment, which are then sent to the corresponding backend, and the results are aggregated and returned. The consistency maintenance module coordinates the state of multiple segments during deletion, renaming, release, and synchronization operations.

[0109] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the unified transparent file access method based on heterogeneous backend segmentation mapping in this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0110] This application provides a unified file transparent access device based on heterogeneous backend segmentation mapping. The unified file transparent access device based on heterogeneous backend segmentation mapping includes: at least one processor; and a memory communicatively connected to at least one processor; wherein the memory stores instructions executable by at least one processor, and the instructions are executed by at least one processor to enable at least one processor to execute the unified file transparent access method based on heterogeneous backend segmentation mapping in the first embodiment above.

[0111] The following is for reference. Figure 12This document illustrates a structural diagram of a unified file transparent access device based on heterogeneous backend segmentation mapping suitable for implementing embodiments of this application. The unified file transparent access device based on heterogeneous backend segmentation mapping in these embodiments may include, but is not limited to, mobile terminals such as laptops, tablets (PADs), portable multimedia players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 12 The unified file transparent access device based on heterogeneous backend segmentation mapping shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0112] like Figure 12 As shown, the unified file transparent access device based on heterogeneous back-end segmentation mapping may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The random access memory 1004 also stores various programs and data required for the operation of the unified file transparent access device based on heterogeneous back-end segmentation mapping. The processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the unified file transparent access device based on heterogeneous back-end segmentation mapping to wirelessly or wiredly communicate with other devices to exchange data. Although the figure shows a unified file transparent access device based on heterogeneous back-end segmentation mapping with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented alternatively.

[0113] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0114] The unified file transparent access device based on heterogeneous backend segmentation mapping provided in this application, employing the unified file transparent access method based on heterogeneous backend segmentation mapping in the above embodiments, can solve the technical problem of how to improve the access efficiency of distributed logical files. Compared with the prior art, the beneficial effects of the unified file transparent access device based on heterogeneous backend segmentation mapping provided in this application are the same as the beneficial effects of the unified file transparent access method based on heterogeneous backend segmentation mapping provided in the above embodiments, and other technical features in this unified file transparent access device based on heterogeneous backend segmentation mapping are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0115] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0116] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0117] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the unified file transparent access method based on heterogeneous backend segmentation mapping in the above embodiments.

[0118] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0119] The aforementioned computer-readable storage medium may be included in a unified file transparent access device based on heterogeneous back-end segmentation mapping; or it may exist independently and not assembled into a unified file transparent access device based on heterogeneous back-end segmentation mapping.

[0120] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a unified file transparent access device based on heterogeneous backend segmentation mapping, enable the unified file transparent access device based on heterogeneous backend segmentation mapping to write computer program code for performing the operations of this application in one or more programming languages ​​or combinations thereof. These programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, or as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0121] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0122] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0123] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described unified file transparent access method based on heterogeneous backend segmentation mapping, thereby solving the technical problem of how to improve the access efficiency of distributed logical files. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the unified file transparent access method based on heterogeneous backend segmentation mapping provided in the above embodiments, and will not be repeated here.

[0124] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the unified file transparent access method based on heterogeneous backend segmentation mapping as described above.

[0125] The computer program product provided in this application can solve the technical problem of how to improve the access efficiency of distributed logical files. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the unified file transparent access method based on heterogeneous backend segmentation mapping provided in the above embodiments, and will not be repeated here.

[0126] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A unified transparent file access method based on heterogeneous backend segmentation mapping, characterized in that, The unified transparent file access method based on heterogeneous backend segmentation mapping includes: Based on the segmentation configuration rules corresponding to the logical file, determine each segment of the logical file and the capacity threshold of each segment; In each segment corresponding to the logical file, a main segment is specified, and in the backend corresponding to the main segment, a main file is created according to the logical file path of the logical file. The main file is used to carry the path, directory entries, basic attributes, extended attributes and first segment data of the logical file. The logical file identifier is used as the physical file name of the main file, and the physical file names of other segments are generated according to the inode number of the main file, the backend path corresponding to other segments, and the segment sequence number of the other segments, wherein the other segments are the subsequent segments of the main segment; Based on the capacity thresholds of the main segment and the other segments, the logical offset intervals of the main segment and the other segments are determined, and a segment mapping relationship of the logical file is created. The segment mapping relationship represents each segment of the logical file and the logical offset interval of each segment. When a vector read / write request for the logical file is received, the corresponding segments of the logical file are accessed based on the logical file identifier and the segment mapping relationship to execute the vector read / write request; When a non-vector read / write request is received for the logical file, the non-vector read / write request is executed within the main file.

2. The unified file transparent access method based on heterogeneous backend segmentation mapping as described in claim 1, characterized in that, The step of accessing each segment corresponding to the logical file based on the logical file identifier and the segment mapping relationship when a vector read / write request for the logical file is received, in order to execute the vector read / write request, includes: Upon receiving a vector read request for the logical file, the segments corresponding to the logical file are determined based on the logical file identifier of the logical file. Based on the first logical offset and read length of the vector read request, and the segment mapping relationship, the target read segment corresponding to the vector read request is determined in each segment; When the number of target read segments is greater than 1, the vector read request is split into local vector read requests corresponding to each target read segment according to the logical offset interval of the target read segment; Each of the local vector read requests is sent to the backend corresponding to each of the target read segments for execution, so as to read the data in the backend corresponding to each of the target read segments.

3. The unified file transparent access method based on heterogeneous backend segmentation mapping as described in claim 2, characterized in that, The logical offset interval includes a start position and an end position. The step of splitting the vector read request into local vector read requests corresponding to each target read segment based on the logical offset interval of the target read segment includes: Based on the first logical offset and the segment mapping relationship, determine the first local offset within the read start segment of the target read segment; The first local offset is used as the logical offset of the local read vector corresponding to the read start segment, and the difference between the interval end position of the read start segment and the first local offset is used as the local read length of the local read vector corresponding to the read start segment. The sum of the first logical offset and the read length is taken as the first logical end position of the vector read request, and the local end offset within the read end segment in the target read segment is determined according to the first logical end position and the segment mapping relationship. The starting position of the interval of the read end segment is used as the logical offset of the local read vector corresponding to the read end segment, and the difference between the local end offset and the starting position of the interval of the read end segment is used as the local read length of the local read vector corresponding to the read end segment. The starting position of the interval of other target segments is used as the logical offset of the local read vector corresponding to the other target segments, and the capacity threshold of the other target segments is used as the local read length of the local read vector corresponding to the other target segments.

4. The unified file transparent access method based on heterogeneous backend segmentation mapping as described in claim 1, characterized in that, The step of accessing each segment corresponding to the logical file based on the logical file identifier and the segment mapping relationship when a vector read / write request for the logical file is received, in order to execute the vector read / write request, includes: Upon receiving a vector write request for the logical file, the segments corresponding to the logical file are determined based on the logical file identifier of the logical file. Based on the second logical offset of the vector write request and the segment mapping relationship, the write start segment corresponding to the vector write request and the second local offset within the write start segment are determined in the segment; The sum of the write length of the vector write request and the second logical offset is used as the second logical end position of the vector write request; When the second logical end position is greater than the interval end position of the write start segment, the difference between the interval end position of the write start segment and the second logical offset is taken as the processing length of the write start segment. The second local offset is used as the logical offset of the local vector write request corresponding to the write start segment, and the processing length is used as the local write length of the local vector write request corresponding to the write start segment. Traverse the subsequent segments of the write start segment. If the capacity threshold of the current segment is less than the remaining write length, then take the interval start position of the current segment as the logical offset of the local vector write request corresponding to the current segment, and take the capacity threshold of the current segment as the local write length of the local vector write request corresponding to the current segment. If the capacity threshold of the current segment is greater than or equal to the remaining write length, then the starting position of the interval of the current segment is used as the logical offset of the local vector write request corresponding to the current segment, and the remaining write length is used as the local write length of the local vector write request corresponding to the current segment, wherein the remaining write length is the difference between the write length and the sum of each local write length.

5. The unified file transparent access method based on heterogeneous backend segmentation mapping as described in claim 1, characterized in that, The non-vector read / write request includes an attribute query request. The main file includes the logical file path entry, directory entry, basic attribute entry, and header data of the logical file. The step of executing the non-vector read / write request in the main file when the logical file is received further includes: When an attribute query request for the logical file is received, the attribute information stored in the backend corresponding to the main segment is read according to the basic attribute entry of the main file. The actual length of the other segments is queried in reverse order, and the existence of valid data in the other segments is determined based on the actual length. The first segment with valid data determined in the reverse query is taken as the tail segment, and the sum of the capacity thresholds of all preceding segments of the tail segment is determined. The total length of the logical file is obtained by summing the capacity thresholds of all preceding segments of the tail segment and adding them to the actual length of the tail segment. The attribute information and the total length of the logical file are used as the query results.

6. The unified file transparent access method based on heterogeneous backend segmentation mapping as described in claim 5, characterized in that, After the step of reversing the query to determine the actual length of the other segments and determining whether there is valid data in the other segments based on the actual length, the method further includes: If no valid data exists in the other segments, the attribute information will be used as the query result.

7. The unified file transparent access method based on heterogeneous backend segmentation mapping as described in claim 1, characterized in that, The non-vector read / write request also includes a rename request, and the step of executing the non-vector read / write request in the main file when the non-vector read / write request of the logical file is received further includes: Upon receiving a renaming request for the logical file, the filename of the main file corresponding to the logical file is modified; The inode number of the main file is used to determine the corresponding physical files, and the filenames of the corresponding physical files are modified.

8. The unified file transparent access method based on heterogeneous backend segmentation mapping as described in claim 1, characterized in that, The unified transparent file access method based on heterogeneous backend segmentation mapping also includes: Upon receiving a deletion request for the logical file, determine the inode number of the main file corresponding to the logical file; Determine the physical file in the backend corresponding to the segment associated with the inode number; Delete the main file and the physical file.

9. A unified transparent file access device based on heterogeneous backend segmentation mapping, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the unified transparent file access method based on heterogeneous backend segmentation mapping as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the unified transparent file access method based on heterogeneous backend segmentation mapping as described in any one of claims 1 to 8.

Images