Data access method, data processing method and file system

Abstract

The application provides a data access method, a data processing method and a file system. According to an embodiment of the application, system files are recorded in the file system, a file system image is stored in a storage medium, the system files correspond to data units in system image files of the file system image, and the offset position information of a target data unit requested to be accessed in the system image file can be determined in response to a data access request, wherein the same data units in one or more file systems correspond to the same position in the file system image, and each system image file is pre-allocated with a corresponding memory space; the target memory address corresponding to the target data unit is determined according to the offset position information of the target data unit in the system image file; and the target data unit is read from the target memory address in the case that the target memory address contains the target data unit, thereby saving the memory occupation.

Description

Technical Field

[0001] This application relates to the field of file system technology, and more particularly to data access methods, data processing methods and apparatus, as well as file systems, electronic devices and storage media. Background Technology

[0002] Some existing file systems currently store only one copy of duplicate files identified by the same filename on the storage medium, thus saving file system storage resources. However, in actual operation, when the data inside the file remains unchanged but the filename changes, duplicate storage of the same data is still unavoidable. Furthermore, although existing file systems can save storage resources, accessing data involves copying data from the storage medium to memory, so duplicate data still occupies multiple copies of memory.

[0003] How to avoid duplicate storage of data in storage media and memory, thereby saving file system storage resources and memory resources, has become one of the urgent problems to be solved. Summary of the Invention

[0004] This application provides a data access method, a data processing method, and a file system to solve one or more of the above-mentioned technical problems.

[0005] In a first aspect, embodiments of this application provide a data access method, wherein a file system records system files, a storage medium stores a file system image, and the system files correspond to data units in a system image file referencing the file system image. The method includes: responding to a data access request, determining the offset position information of the target data unit to be accessed in the system image file, wherein the same data unit in one or more file systems is stored at the same position in the file system image, and each system image file has a pre-allocated corresponding memory space; determining the target memory address corresponding to the target data unit based on the offset position information of the target data unit in the system image file; and, if the target data unit exists at the target memory address, reading the target data unit from the target memory address.

[0006] Secondly, embodiments of this application provide a data access method for a container, wherein the container records a container file, a storage medium stores a container image, and the container file corresponds to a data unit in a container image file that references the container image. The method includes: responding to a data access request, determining the offset position information of the target data unit to be accessed in the container image, wherein the same data unit in multiple containers corresponding to a container group is stored at the same position in the container image, and each container image file has a corresponding memory space pre-allocated; determining the target memory address corresponding to the target data unit based on the offset position information of the target data unit in the container image file; and reading the target data unit from the target memory address if the target data unit exists at the target memory address.

[0007] Thirdly, embodiments of this application provide a data processing method, wherein the method includes: creating a file system image corresponding to a file system in a storage medium, wherein system files recorded in the file system are configured as data units referencing the system image file of the file system image, and identical data units in one or more file systems are configured as corresponding references to the same location in the file system image; allocating corresponding memory space for each system image file included in the file system image; and obtaining the target data unit through the data access method after receiving a data access request.

[0008] Fourthly, embodiments of this application provide a database system, comprising computing nodes and storage nodes; on the computing node, a data unit in a system image file of a file system image in the storage node is referenced for a system file recorded in the file system; in the storage node, the same data unit in one or more file systems references the same location in the file system image, and each system image file has a corresponding memory space pre-allocated; the computing node is also used to execute the data access method.

[0009] Fifthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method described in any of the above-mentioned embodiments.

[0010] Sixthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any of the preceding claims.

[0011] Compared with related technologies, this application has the following advantages:

[0012] According to embodiments of this application, a file system records system files, and a storage medium stores a file system image. Each system file corresponds to a data unit in a system image file that references the file system image. In response to a data access request, the offset position information of the target data unit to be accessed within the system image file can be determined. Specifically, identical data units in one or more file systems are stored at the same location within the file system image, and each system image file has pre-allocated corresponding memory space. Since the file system can record data units for system files via references, duplicate storage of identical data units within a single file system and between multiple file systems on the storage medium can be avoided.

[0013] Based on the offset position information of the target data unit in the system image file, the target memory address corresponding to the target data unit is determined. If the target data unit exists at the target memory address, it is read from that address. Furthermore, if the target data unit does not exist at the target memory address, a data read / write process can be invoked based on the offset position information of the target data unit in the system image file to perform I / O operations, read the target data unit from the system image file of the storage medium, and store it at the target memory address.

[0014] Finally, the target data unit is read from the target memory address. Since there's no need to maintain separate memory space for system files, but instead, data units are accessed by referencing data units in the memory space maintained by the image file. Data units are copied from the storage medium to memory only if the target memory address does not exist. This avoids duplicate storage of identical data units on the storage medium and also prevents duplicate storage of identical data units from one or more file systems in memory. Furthermore, it saves storage and memory resources, reduces I / O read / write operations, and decreases bandwidth usage.

[0015] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0016] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0017] Figure 1 A schematic diagram of a data access scheme using related technologies is shown;

[0018] Figure 2 A schematic diagram of another related technology for implementing data access is shown;

[0019] Figure 3 A schematic diagram of another related technology for implementing data access is shown;

[0020] Figure 4 A schematic diagram of a data access scheme provided in an embodiment of this application is shown.

[0021] Figure 5 A flowchart of a data access method provided in an embodiment of this application is shown;

[0022] Figure 6 This illustration shows a schematic diagram of an offset position information mapping relationship provided in an embodiment of this application;

[0023] Figure 7 A flowchart illustrating an implementation of a data access method provided in an embodiment of this application is shown;

[0024] Figure 8 A flowchart of a data access method for a container provided in an embodiment of this application is shown;

[0025] Figure 9 A flowchart of a data file processing method provided in an embodiment of this application is shown;

[0026] Figure 10 A schematic diagram of a file system provided in an embodiment of this application is shown;

[0027] Figure 11 This paper shows a structural block diagram of a data access device provided in an embodiment of this application;

[0028] Figure 12 This paper shows a structural block diagram of a data access device for a container provided in an embodiment of this application;

[0029] Figure 13 This application shows a structural block diagram of a data file processing apparatus provided in an embodiment of the present application; and

[0030] Figure 14 A block diagram of an electronic device used to implement embodiments of this application is shown. Detailed Implementation

[0031] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0032] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0033] First, the relevant concepts involved in the embodiments of this application will be explained.

[0034] A file system is a subsystem within a computer's operating system responsible for managing persistent data. It maintains the storage media (such as read-only memory, disks, or optical discs) and memory space where the data in a file resides, and provides users with data storage and access functions. A file is a contiguous block of stored data uniquely identified by its filename. By dividing the contiguous data within a file into defined intervals, data units can be obtained; a data unit is a contiguous interval of stored data, which can specifically include data blocks (chunks) and data extents (extents).

[0035] From a data storage perspective, a single file can consist of one or more data units. The data units contained in one or more files recorded by the file system can be stored on one or more local or remote storage media. The remote storage media can include the storage media in the storage nodes provided by the cloud platform. From an application layer perspective, the file system can provide users with an interface for accessing files. In this interface, users access the data units written into a file by using the filename as an identifier. In this embodiment, to distinguish different files, files recorded in the file system are referred to as system files.

[0036] In a related technology prior to this application, multiple system files recorded in the file system occupy different storage locations on the storage medium, and each file maintains its own corresponding memory space in memory. When accessing data units in a system file, I / O (Input / Output) operations are required to read the data unit from the storage medium and copy it to the corresponding location in the memory space. For example... Figure 1 A schematic diagram illustrating a data access scheme implemented using this related technology is shown. For example... Figure 1As shown, the first file system contains system files named "foo" and "bar," while the second file system contains a system file named "baz." The "foo" file in the first file system contains data units A, B, and C; the "bar" file contains data units A and C; and the "baz" file in the second file system contains data unit A. These files occupy different storage locations on the storage medium, resulting in the same data units existing repeatedly in the storage medium. Since each system file maintains its own memory space, the same data units also exist repeatedly in memory.

[0037] In one application example, taking access to data unit A as an example, since the files foo, bar, and baz all contain data unit A, data unit A occupies three different storage locations in the storage medium. When a user accesses data unit A through the foo file, if data unit A does not yet exist in the memory space maintained by the foo file, the first file system needs to locate data unit A from the storage location of data unit A in the storage medium of the foo file and perform an I / O operation to copy data unit A into the memory space maintained by the foo file, thus enabling access to the data in data A. Similarly, when a user accesses data unit A through the bar or baz file, the file systems corresponding to the above files will perform the same steps again, copying data unit A to the memory space corresponding to the system file. In this application example, if data unit A is accessed through the foo, bar, and baz files respectively, three I / O operations are required, and data unit A also occupies three different storage locations in the memory space.

[0038] Therefore, in this scenario, the same data units exist repeatedly in storage media and memory within a single file system (such as within the first file system) and between multiple file systems (such as between the first and second file systems). The duplicate data occupies multiple storage and memory resources. When distributing data over the network, distributing the same data multiple times will also occupy the corresponding network bandwidth resources, thus leading to resource waste.

[0039] In another related technology prior to this application, duplicate storage of system files in the storage medium can be avoided by using an overlay file system. An overlay file system can solve the problem of duplicate storage of system files among multiple file systems. It can achieve deduplication among multiple file systems at the file layer level, where a file layer is a collection of multiple files, and a file layer can include one or more system files. Figure 2 A schematic diagram illustrates a scheme for achieving data access using a stacked file system. For example... Figure 2 As shown, the first stacked file system records system files named foo, bar, and baz, while the second stacked file system records system files named foo, bar, and qux. That is, the first and second stacked file systems respectively record the same system files foo and bar. In this scenario, the first and second stacked file systems share file layer A on the storage medium, thus avoiding duplicate storage of the same system files on the storage medium.

[0040] In scenarios using stacked file systems, each file layer maintains its own memory space, such as... Figure 2 The files foo, bar, baz, and qux shown are all stored in the memory space corresponding to the file layer, ensuring that only one copy of the same system file is stored in memory. Ideally, in this scenario... Figure 2 The first and second stacked file systems shown in the figure have the same system files occupying the same storage locations in the storage medium and memory space, which can be done at the system file level, thus avoiding the waste of storage and memory resources.

[0041] However, in practical applications, since the file layer is built based on the system file's timestamp, which describes the time of the most recent access to the system file, each access to the system file will cause the timestamp to change, and thus the stacked file system will build a new file layer for the system file. Figure 3 Another schematic diagram illustrates a scheme for achieving data access using a stacked file system. For example... Figure 3 As shown, compared to... Figure 2 In the illustrated scenario, because the access times for the foo and bar files recorded in the first stacked file system differ from those recorded in the second stacked file system, the first stacked file system constructs a corresponding file layer A in the storage medium to store the foo and bar files accessed at the time identified by the first timestamp (timestamp1). In file layer A, the foo and bar files are stored as files named foo.1 and bar.1, respectively. Similarly, the second stacked file system constructs a corresponding file layer D in the storage medium to store the foo and bar files accessed at the time identified by the second timestamp (timestamp2). In file layer D, the foo and bar files are stored as files named foo.2 and bar.2, respectively.

[0042] In this scenario, the memory space maintained by the file layer AD stores the files foo.1, bar.1, baz, foo.2, bar.2, and qux. Therefore, even with a stacked file system, it is impossible to avoid the duplicate storage of system files in the storage medium and memory space.

[0043] In view of this, embodiments of this application provide a new data access scheme to solve all or part of the above-mentioned technical problems.

[0044] Figure 4 This is a schematic diagram illustrating an application scenario for implementing the method of the embodiments of this application. Figure 4 The two file systems shown in the image record files and are composed of... Figure 1 The two file systems shown record the same files. To simplify the scenario, Figure 4 The data units other than data unit A are indicated by different styles of shading. For example... Figure 4 As shown, files recorded in the first and second file systems are actually stored in an image created jointly by the first and second file systems on the storage medium.

[0045] In scenarios involving file system data management, data recorded by the file system can be stored on the storage medium by creating a mirror corresponding to the file system. A mirror is a file storage format, and in practice, multiple files can be created as a single mirror. For example, in one application, multiple system files recorded within a single file system, or multiple system files recorded in multiple related file systems, can be created as a single mirror. In this embodiment, a mirror created based on one or more file systems is referred to as a file system mirror. In practical applications, data stored on the storage medium can be fragmented according to a preset space usage. In this embodiment, the file system mirror can also be fragmented, dividing the file system mirror into multiple system mirror files; that is, a file system mirror can consist of one or more system mirror files.

[0046] It is understandable that system image files in storage media can correspond to data units in storage system files. However, system image files obtained by partitioning file system images usually do not have a one-to-one correspondence with system files in the file system. Multiple data units recorded in a system file can be stored in multiple system image files. A system image file may contain some or all data units from multiple system files.

[0047] like Figure 4As shown, after creating a common system image (i.e., the image shown in the figure) for the first and second file systems, the same data unit A within the first file system and between the first and second file systems is actually stored in the first system image file (blobs1) on the storage medium. The first and second file systems can record the storage location of data units in the system image of the files foo, bar, and baz, and reference the address corresponding to the recorded storage location when accessing the data unit to avoid storing the same data unit repeatedly on the storage medium. Based on this, memory space is maintained in memory for the system image file in the file system image. Therefore, when accessing data units, the address of the data unit in memory can also be referenced to avoid storing the same data unit repeatedly in memory. Thus, when accessing a data unit, only an I / O operation is needed on the first access to copy the data unit from the system image file on the storage medium to the corresponding location in the system image file in memory.

[0048] For example, by Figure 4 In the illustrated memory space, data unit A has been copied to the corresponding location of the first system image file in memory. When an access request for data unit A is received in the files foo, bar, or baz, the corresponding location where data unit A is stored in the first system image file can be directly accessed in memory. This avoids duplicate storage of data units on the storage medium within a single file system and between multiple file systems, as well as duplicate storage of data units in memory within a single or multiple file systems. Therefore, while saving storage and memory resources, it also reduces I / O read / write operations and bandwidth usage.

[0049] This application provides a data access method, such as... Figure 5 The diagram shows a flowchart of a data access method 500 according to an embodiment of this application. The file system records system files, and the storage medium stores a file system image. The system files correspond to data units in a system image file that references the file system image. The method 500 may include:

[0050] In step S501, in response to a data access request, the offset position information of the target data unit to be accessed in the system image file is determined, wherein the same data unit in one or more file systems is stored in the same position in the file system image, and each system image file has a corresponding memory space pre-allocated.

[0051] In this embodiment, by constructing a file system image corresponding to the file system in the storage medium, it is possible to store the same data units from one or more file systems in the same location of the file system image, thereby avoiding duplicate storage of the same data units on the storage medium. The file system image has pre-allocated corresponding memory space, which may be allocated by the file system for the file system image during file system initialization. In one possible implementation, memory space can also be allocated separately for each system image file included in the file system image. For example, in a container scenario, initial memory space can be allocated for the container image in memory when starting a container instance. In this embodiment, system files no longer maintain separate memory space. Furthermore, when a new system image file is created from newly added data units in the corresponding file system, memory space can be allocated for the new system image file.

[0052] In this embodiment, data access requests can be initiated by an application process. The application process involved can be initiated by a user of the application; for example, in a container scenario, the user can be a user of a container service. In one possible implementation, when the file system includes containers, the file system image can include container images, system files can include container files, the system image file can include container image files within the container image, and the application process involved can include container processes. When the file system includes programs, the file system image can include program images, system files can include program files, the system image file can include program image files, and the application process involved can include program processes. Taking a container scenario as an example, container scenarios involve managing files within a container cluster, which typically includes a large number of container groups (Pods), and each container group may contain multiple container instances. Since container instances within a container group often record the same files, avoiding duplicate storage of files in storage media and memory space in a container scenario can save significant resources.

[0053] A file system can record one or more system files, and a system file can include multiple data units. When accessing data recorded in the file system, a data access request initiated by an application process can be used to request access to a specific data unit in a system file. To distinguish different data units, the data unit requested by the data access request is denoted as the target data unit. In this embodiment, the data units included in the file system are actually stored in the file system image. Therefore, when accessing the target data unit, it is necessary to first determine the actual storage location of the target data unit. The actual storage location of the target data unit can be determined based on the system image file where the target data unit is located and the offset position information of the target data unit in the system image. The offset position information refers to the distance between the actual address of the data unit and the file address of its location.

[0054] In one possible implementation, when determining the offset position information of the target data unit requested for access within the system image file, the identification information of the target data unit in the system file can first be determined based on the offset position information of the target data unit carried in the data request. In one possible application example, the identification information of the data unit can be the serial number of the data unit in the system file. In a calculation example, the identification information of the target data unit in the system file can be determined using a formula.

[0055]

[0056] The sequence number of the data unit within the system file is determined, and this sequence number is used as the identification information of the data unit. In the above formula, i represents the sequence number of the data unit within the system file, offset X is the offset position information of the data unit within the system file, and S is the space occupied by the data unit. S is recorded in the metadata of the system file and can be obtained by retrieving the metadata about the system file stored in the system image.

[0057] Then, the offset position information mapping table can be queried to determine the offset position information of the target data unit in the system image file based on the identification information of the target data unit in the system file. In one possible application example, the offset position information mapping table is stored in the metadata of the system file. The metadata of the requested system file can be obtained first, and then the offset position information mapping table included in the metadata can be further obtained. The offset position information mapping table records the mapping relationship between the offset position information of the data unit in the system file and the system image file, respectively. The mapping relationship refers to the mapping relationship between the offset position information of a certain data unit in the system file and the offset position information of the same data unit in the system image file. Figure 6A schematic diagram illustrating a mapping relationship of offset position information provided in an embodiment of this application is shown. For example... Figure 6 As shown, the system file named foo includes data unit A and some other data units. The offset position information of data unit A in the foo file is offset X, which is the offset position information of the target data unit carried by the aforementioned data request in the system file.

[0058] To distinguish between different offset position information in the embodiments of this application, the offset position information of the data unit in the system file is denoted as offset X, and the offset position information of the data unit in the system image file is denoted as offset Y. Based on the mapping relationship recorded in the offset position information mapping table, the offset Y of the target data unit can be determined using the offset X of the target data unit. Combined with... Figure 6 After determining the offset X of data unit A in the foo file, the offset position information mapping table corresponding to the foo file can be obtained through the metadata of the foo file, and the offset Y of data unit A can be determined according to the mapping relationship recorded in the offset position information mapping table.

[0059] Table 1 shows a possible offset location information mapping table. The process of determining the offset location information of the target data unit to be accessed in the system image file based on the offset location information mapping table is explained below with reference to Table 1.

[0060] Identification information of data unit System image file identification information Offset position information 1 First system image file Y1 2 Second system image file Y2 3 First system image file Y3 … … … n Third system image file Yn

[0061] Table 1 Offset Position Information Mapping Table

[0062] As shown in Table 1, the offset location information mapping table records the identification information of multiple data units that make up the system file within the system file, the identification information of the system image file that actually stores the data unit within the system image, and the offset location information of the data unit within the system image file. In an application example where the sequence number of the data unit in the system file is used as the identification information of the data unit, after determining the sequence number of the target data unit in the system file, the system image file that actually stores the target data unit in the storage medium, as well as the offset location information of the target data unit within the system image file, can be found according to the offset location information mapping table. For example, after calculating the identification information of data unit A as 3, according to Table 1, the data unit with identification information 3 is actually stored in the first system image file, and the offset of this data unit in the first system image file is Y3.

[0063] In one possible implementation, when the requested data is a portion of data within a data unit, the requested data is designated as the target data. Based on the first offset position information of the target data within the target data unit carried in the data request within the system file, the identifier information of the target data unit in the system file, and the second offset position information of the target data within the target data unit, are obtained. Then, an offset position information table is queried, and based on the identifier information of the target data unit in the file system, the third offset position information of the target data unit in the system image file is obtained. Finally, based on the second and third offset position information, the fourth offset position information of the target data in the system image file is generated.

[0064] In step S502, the target memory address corresponding to the target data unit is determined based on the offset position information of the target data unit in the system image file.

[0065] The target memory address refers to the storage address of the target data unit in memory space. After determining the offset location information of the target data unit in the system image file, the target memory address can be determined based on the determined offset location information, and then the target data unit can be accessed by accessing the target memory address.

[0066] In one possible implementation, when determining the target memory address corresponding to the target data unit based on its offset position information in the system image file, the memory space corresponding to the system image file containing the target data unit is first determined. Specifically, after determining the system image file that actually stores the target data unit in the storage medium, the location of the memory space pre-allocated to the system image file can be further determined. Then, based on the offset position information of the target data unit in the system image file, the offset position information of the target data unit in the memory space is determined. It is understood that the system image file in the storage medium maintains corresponding memory space, and the same data unit occupies only one storage location in the storage medium. In the embodiments of this application, the offset position information of the same data unit in the system image file in the storage medium and in the memory space has a one-to-one correspondence. Finally, the target memory address corresponding to the target data unit can be determined based on the offset position information of the target data unit in the memory space.

[0067] In step S503, if the target data unit exists at the target memory address, the target data unit is read from the target memory address.

[0068] Figure 7 A flowchart illustrating an implementation of a data access method provided in an embodiment of this application is shown. For example... Figure 7As shown, after determining the offset location information corresponding to the target data unit and identifying the target memory space in memory for storing the target data unit, in one possible implementation, even if the target data unit does not exist at the target memory address, the data read / write process can be invoked to read the target data unit from the system image file of the storage medium and store the target data unit at the target memory address, based on the offset location information of the target data unit in the system image file. In other words, when it is determined that the target memory space is empty, i.e., when this is the first access to the target unit data, corresponding memory can be allocated to the target memory space, and I / O operations can be performed to copy the target data unit from the storage medium to the target memory address. Then, the target data unit is read from the target memory address, completing the access to the data unit.

[0069] Therefore, the same data units in one or more file systems are stored in the same location in the memory space corresponding to the file system image. When accessing the target data unit, the data in the target data unit can be read by referencing the target memory address corresponding to the target data unit, thereby avoiding the waste of memory space resources.

[0070] This application also provides a data access method for containers, such as... Figure 8 The diagram shows a flowchart of a data access method 800 for a container according to an embodiment of this application. The container contains a container file, and a container image is stored in the storage medium. The container file corresponds to a data unit in a container image file that references the container image. The method 800 may include:

[0071] In step S801, in response to a data access request, the offset position information of the target data unit to be accessed in the container image is determined. The same data unit in multiple containers of a container group is stored in the same position in the container image, and each container image file has a corresponding memory space pre-allocated.

[0072] In step S802, the target memory address corresponding to the target data unit is determined based on the offset position information of the target data unit in the container image file.

[0073] In step S803, if the target data unit exists at the target memory address, the target data unit is read from the target memory address.

[0074] In this embodiment, the file system involved can be the file system of multiple container instances included in the same container group, and the data access request can be initiated by the container process. Specific implementation methods can be found in the above embodiments, and will not be repeated here.

[0075] This application also provides a data processing method, such as... Figure 9 The diagram shown is a flowchart of a data processing method 900 according to an embodiment of this application. The method 900 may include:

[0076] In step S901, a file system image corresponding to the file system is created in the storage medium. The system files recorded in the file system are configured as data units in the system image file that references the file system image. The same data units in one or more file systems are configured as the same location in the corresponding referenced file system image.

[0077] In step S902, memory space is allocated for each system image file included in the file system image.

[0078] In step S903, after receiving the data access request, the target data unit is obtained through the data access method 500.

[0079] The specific implementation method of data processing in the embodiments of this application can be found in the foregoing embodiments, and will not be repeated here.

[0080] See Figure 10 This application also provides a file system 1000, which may include a compute node 1001 and a storage node 1002, wherein:

[0081] On the computing node 1001, the corresponding data unit in the system image file of the file system image in the storage node 1002 is referenced for the system file recorded in the file system.

[0082] In the storage node 1002, the same data unit in one or more file systems corresponds to the same location in the file system image, and each system image file has a corresponding memory space pre-allocated.

[0083] The computing node 1001 is also used to execute the method described in the foregoing embodiments.

[0084] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide a data access device, wherein a file system records system files, a storage medium stores a file system image, and the system files correspond to data units in the system image file of the file system image, such as... Figure 11 The diagram shown is a structural block diagram of a data access device 1100 according to an embodiment of this application. The device 1100 may include:

[0085] Information determination module 1101 is used to determine the offset position information of the target data unit to be accessed in the system image file in response to a data access request, wherein the same data unit in one or more file systems is stored in the same position in the file system image, and each system image file is pre-allocated with corresponding memory space.

[0086] Address determination module 1102 is used to determine the target memory address corresponding to the target data unit based on the offset position information of the target data unit in the system image file;

[0087] The data unit reading module 1103 is used to read the target data unit from the target memory address when the target data unit exists at the target memory address.

[0088] In one possible implementation, the information determination module 1101 may include:

[0089] The identification information determination submodule is used to determine the identification information of the target data unit in the system file based on the offset position information of the target data unit carried in the data request in the system file;

[0090] The mapping relationship determination submodule is used to query the offset position information mapping table, determine the offset position information of the target data unit in the system image file according to the identification information of the target data unit in the system file, and the offset position information mapping table records the mapping relationship of the offset position information of the data unit in the system file and the system image file respectively.

[0091] In one possible implementation, the information determination module 1101 may be specifically configured to: obtain, based on the first offset position information of the target data in the target data unit carried in the data request within the system file, the identification information of the target data unit in the system file, and the second offset position information of the target data in the target data unit; query the offset position information table, and based on the identification information of the target data unit in the file system, obtain the third offset position information of the target data unit in the system image file; and generate the fourth offset position information of the target data in the system image file based on the second offset position information and the third offset position information.

[0092] In one possible implementation, the address determination module 1102 may be specifically used to: determine the memory space corresponding to the system image file where the target data unit is located; determine the offset position information of the target data unit in the memory space based on the offset position information of the target data unit in the system image file; and determine the target memory address corresponding to the target data unit based on the offset position information of the target data unit in the memory space.

[0093] In one possible implementation, the device 1100 may further include:

[0094] The data reading module is used to, when the target data unit does not exist in the target memory space, call the data reading and writing process to read the target data unit from the system image file of the storage medium and store the target data unit to the target memory address according to the offset position information of the target data unit in the system image file; and read the target data unit from the target memory address.

[0095] In one possible implementation, the device 1100 may further include:

[0096] The space allocation module is used to allocate corresponding memory space for each system image file included in the file system image.

[0097] In one possible implementation, when the file system includes a container, the file system image includes a container image, the system file includes a container file, and the system image file includes a container image file within the container image; when the file system includes a program, the file system image includes a program image, the system file includes a program file, and the system image file includes a program image file.

[0098] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide a data access device for a container, wherein the container records a container file, the storage medium stores a container image, and the container file corresponds to a data unit in a system image file that references the container image, such as... Figure 12 The diagram shown is a structural block diagram of a data access device 1200 for a container according to an embodiment of this application. The device 1200 may include:

[0099] Information determination module 1201 is used to determine the offset position information of the target data unit to be accessed in the container image in response to a data access request, wherein the same data unit in multiple containers corresponding to a container group is stored in the same position in the container image, and each container image file has a corresponding memory space pre-allocated.

[0100] Address determination module 1202 is used to determine the target memory address corresponding to the target data unit based on the offset position information of the target data unit in the container image file;

[0101] The data unit reading module 1203 is used to read the target data unit from the target memory address when the target data unit exists at the target memory address.

[0102] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide a data processing apparatus, wherein, as Figure 13 The diagram shown is a structural block diagram of a data processing apparatus 1300 according to an embodiment of this application. The apparatus 1300 may include:

[0103] The image creation module 1301 is used to create a file system image corresponding to a file system in a storage medium, wherein system files recorded in the file system are configured as data units in the system image file that references the file system image, and the same data units in one or more file systems are configured as the same location in the corresponding referenced file system image.

[0104] The space allocation module 1302 is used to allocate corresponding memory space for each system image file included in the file system image;

[0105] The data unit acquisition module 1303 is used to acquire the target data unit by means of the method described in the foregoing embodiments after receiving a data access request.

[0106] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0107] Figure 14 This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 14 As shown, the electronic device includes a memory 1401 and a processor 1402. The memory 1401 stores a computer program that can run on the processor 1402. When the processor 1402 executes the computer program, it implements the method described in the above embodiments. The number of memories 1401 and processors 1402 can be one or more.

[0108] The electronic device also includes:

[0109] Communication interface 1403 is used to communicate with external devices and perform data exchange and transmission.

[0110] If the memory 1401, processor 1402, and communication interface 1403 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 14 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0111] Optionally, in a specific implementation, if the memory 1401, processor 1402, and communication interface 1403 are integrated on a single chip, then the memory 1401, processor 1402, and communication interface 1403 can communicate with each other through an internal interface.

[0112] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0113] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0114] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0115] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0116] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0117] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0118] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0119] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0120] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0121] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0122] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0123] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0124] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data access method, wherein, The file system records system files, and the storage medium stores a file system image. The system files correspond to data units in a system image file that references the file system image. The method includes: In response to a data access request, the offset position information of the target data unit requested for access in the system image file is determined, wherein the same data unit in one or more file systems is stored in the same position in the file system image, and each system image file has a corresponding memory space pre-allocated. Based on the offset position information of the target data unit in the system image file, the target memory address corresponding to the target data unit is determined; If the target data unit exists at the target memory address, the target data unit is read from the target memory address.

2. The method according to claim 1, characterized in that, The offset position information of the target data unit requested for access in the system image file includes: Based on the offset position information of the target data unit in the system file carried in the data access request, the identification information of the target data unit in the system file is determined; The offset position information mapping table is queried, and the offset position information of the target data unit in the system image file is determined according to the identification information of the target data unit in the system file. The offset position information mapping table records the mapping relationship of the offset position information of the data unit in the system file and the system image file respectively.

3. The method according to claim 2, characterized in that, The offset position information of the target data unit requested for access in the system image file includes: Based on the first offset position information of the target data in the target data unit carried in the data access request in the system file, the identification information of the target data unit in the system file where the target data is located, and the second offset position information of the target data in the target data unit are obtained. Query the offset position information table, and obtain the third offset position information of the target data unit in the system image file based on the identification information of the target data unit in the file system; Based on the second offset position information and the third offset position information, the fourth offset position information of the target data in the system image file is generated.

4. The method according to claim 1, characterized in that, The step of determining the target memory address corresponding to the target data unit based on the offset position information of the target data unit in the system image file includes: Determine the memory space corresponding to the system image file where the target data unit is located; Based on the offset position information of the target data unit in the system image file, determine the offset position information of the target data unit in the memory space; The target memory address corresponding to the target data unit is determined based on the offset position information of the target data unit in the memory space.

5. The method according to claim 1, characterized in that, The method further includes: If the target data unit does not exist at the target memory address, the data read / write process is invoked to read the target data unit from the system image file of the storage medium and store the target data unit at the target memory address, based on the offset position information of the target data unit in the system image file. Read the target data unit from the target memory address.

6. The method according to claim 1, characterized in that, The method further includes: Allocate corresponding memory space for each system image file included in the file system image.

7. The method according to claim 1, characterized in that, When the file system includes containers, the file system image includes container images, the system files include container files, and the system image file includes container image files within the container image; When the file system includes a program, the file system image includes a program image, the system file includes a program file, and the system image file includes a program image file.

8. A data access method for a container, wherein, The container contains a container file, and the storage medium stores a container image. The container file corresponds to a data unit in a container image file that references the container image. The method includes: In response to a data access request, the offset position information of the target data unit requested for access in the container image is determined, wherein the same data unit in multiple containers corresponding to a container group is stored in the same position in the container image, and each container image file has a corresponding memory space pre-allocated. Based on the offset position information of the target data unit in the container image file, the target memory address corresponding to the target data unit is determined; If the target data unit exists at the target memory address, the target data unit is read from the target memory address.

9. A data processing method, wherein, The method includes: Create a file system image corresponding to the file system in the storage medium, wherein the system files recorded in the file system are configured as data units in the system image file that references the file system image, and the same data units in one or more file systems are configured as the same location in the corresponding referenced file system image; Allocate corresponding memory space for each system image file included in the file system image; Upon receiving a data access request, the target data unit is obtained using the method described in any one of claims 1-7.

10. A file system, wherein, Includes compute nodes and storage nodes; The computing node references the data unit in the system image file of the file system image in the storage node corresponding to the system file recorded in the file system; The same data unit in one or more file systems in the storage node corresponds to the same location in the file system image, and each system image file has a corresponding memory space pre-allocated. The computing node is also used to perform the method of any one of claims 1-7.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1-9.

12. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1-9.

Images