Data access method and device, storage medium and electronic device

Abstract

The embodiment of the application provides a data access method and device, a storage medium and an electronic device, wherein the data access method comprises the following steps: starting a container service in each controller, and adding a metadata service component and a file sharing protocol service component in a container image managed by each container service, wherein each controller is a controller in a disk array; in the case that a data access request sent by a target device is received, accessing target file data through a first container service in a first controller and a first block service in the first controller, wherein the first container service and the first block service interact with each other through a first shared memory in the first controller, and the data access request is a request for accessing target file data corresponding to a first disk space managed by the first controller.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of circuits, and more specifically, to a data access method and device, a storage medium, and an electronic device. Background Art

[0002] Disk arrays are generally used in conjunction with existing hardware, and their software technology is relatively mature and has high performance. However, since they can only provide block services, they cannot adapt to the current file service demand scenarios with growing logs.

[0003] Existing products that integrate block services and file services utilize a stacking approach, with a complete block service at the bottom layer and independent file services deployed on top via virtual machines or containers. However, this stacking approach completely isolates the block and file services, resulting in long paths for users to access data through file interfaces and a lack of high-performance file access. Alternatively, the fusion of block and file services integrates both block and file service implementation logic within a single system, interfering with each other and leading to complex and unstable implementation. However, this fusion approach fully integrates the block and file services, mixing block and file interfaces. This creates a complex implementation, and modifications to one inevitably impact the other, easily introducing issues and preventing independent upgrades and maintenance.

[0004] Therefore, for products integrating block services and file services in related technologies, there is a problem that they cannot provide high-performance file access while ensuring the independence of the block service and file service process spaces, which has not yet been effectively solved. Summary of the Invention

[0005] The embodiments of the present application provide a data access method and device, a storage medium, and an electronic device to at least solve the problem that products integrating block services and file services in related technologies cannot provide high-performance file access while ensuring the independence of the block service and file service process spaces.

[0006] According to one embodiment of the present application, a data access method is provided, specifically including: starting a container service in each controller, and adding a metadata service component and a file sharing protocol service component in the container image managed by each container service, wherein each controller is a controller in a disk array; upon receiving a data access request sent by a target device, accessing target file data through a first container service in a first controller and a first block service in the first controller, wherein the first container service and the first block service interact with each other through a first shared memory in the first controller, and the data access request is a request to access target file data corresponding to a first disk space managed by the first controller.

[0007] In an example embodiment, accessing target file data by a first container service in a first controller through a first block service in the first controller comprises: in response to determining that the data access request is for reading first file data in the first disk space, determining whether the first file data exists in the first shared memory, wherein the target file data comprises the first file data; in response to determining that the first file data does not exist in the first shared memory, querying a first metadata information corresponding to the first file data in a first metadata service component in the first container service, wherein the first metadata information comprises at least a first address of the first file data; sending the first metadata information to the first block service through a first file sharing protocol service component in the first container service, so that the first block service reads the first file data in the first disk space according to the first metadata information and loads the first file data into the first shared memory; obtaining the first file data in the first shared memory through the first file sharing protocol service component and sending the first file data to the target device, so that the target device reads the first file data.

[0008] In an example embodiment, accessing target file data by a first container service in a first controller through a first block service in the first controller comprises: in response to determining that the data access request is for writing second file data into the first disk space, creating second metadata information of the second file data in a first metadata service component in the first container service, wherein the second metadata information comprises at least a second address of the second file data, and the target file data comprises the second file data; writing the second file data into the first shared memory through a first file sharing protocol service component in the first container service and the second metadata information, so that the first block service writes the second file data in the first shared memory into the second address of the first disk space.

[0009] In one example embodiment, before creating the second metadata information of the second file data in the first metadata service component in the first container service, the method further comprises: splitting the second file data into a plurality of target data blocks; performing data block comparison between the plurality of target data blocks and a plurality of first data blocks in the first disk space, and determining a comparison result; in case that the comparison result indicates that the plurality of first data blocks contain the plurality of target data blocks, prohibiting writing the second file data into the first disk space; in case that the comparison result indicates that the plurality of first data blocks contain a second data block of the plurality of target data blocks, writing the plurality of target data blocks except the second data block into the first disk space, and prohibiting writing the second data block into the first disk space; in case that the comparison result indicates that none of the plurality of target data blocks is in the plurality of first data blocks, determining to allow writing the second file data into the first disk space.

[0010] In one example embodiment, accessing target file data by a first container service in a first controller via a first block service in the first controller comprises: determining whether the first controller fails; in case that the first controller is determined to fail, determining a second controller managing the first disk space in the disk array, and accessing the target file data via a second container service in the second controller and a second block service in the second controller, wherein the second container service and the second block service perform data interaction via a second shared memory in the second controller; in case that the first controller is determined not to fail, accessing the target file data via the first container service and the first block service.

[0011] In an exemplary embodiment, determining a second controller in the disk array that manages the first disk space includes: constructing a metadata service component cluster corresponding to the disk array based on each metadata service component in each controller; broadcasting a metadata service component map corresponding to each metadata service component to a container service corresponding to each controller based on the metadata service component cluster, wherein the metadata service component map is used to indicate the running status of each metadata service component; when it is determined that the first controller fails, updating a first metadata service component map corresponding to the first metadata service component, and broadcasting the first metadata service component map to a container service corresponding to each controller, wherein the first metadata service component map is used to indicate that the running status of the first metadata service component is a failure state, the first controller includes: the first metadata service component; remapping the target disk space in the disk array based on the updated first metadata service component map to map the target metadata information of the data in the first disk space managed by the first metadata service component to a healthy metadata service component in the metadata service component cluster, wherein the healthy metadata service component is a metadata service component whose running status is indicated as normal by the metadata service component map; and determining the controller corresponding to the healthy metadata service component as the second controller.

[0012] In an exemplary embodiment, after adding a metadata service component and a file sharing protocol service component to the container image managed by each container service, the method further includes: establishing a service map corresponding to the disk array through the interconnection links between each controller and other controllers in the disk array, wherein the service map is used to indicate a first operating state of each container service and a second operating state of each block service in each controller; when it is determined according to the service map that any container service and / or any block service in any controller in the disk array fails, performing fault repair on the any container service and / or any block service through a distributed consistency protocol, wherein the fault repair strategy corresponding to each container service in each controller in the disk array is the same, and the fault repair strategy for each block service in each controller in the disk array is the same.

[0013] According to another embodiment of the present application, a data access device is provided, including: a startup module, used to start a container service in each controller, and add a metadata service component and a file sharing protocol service component in the container image managed by each container service, wherein each controller is a controller in a disk array; an access module, used to access target file data through a first container service in a first controller and a first block service in the first controller when receiving a data access request sent by a target device, wherein the first container service and the first block service interact with each other through a first shared memory in the first controller, and the data access request is a request to access target file data corresponding to a first disk space managed by the first controller.

[0014] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is configured to execute the steps in the above method embodiment when running.

[0015] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in the above method embodiment.

[0016] According to another embodiment of the present application, a computer program product is provided, including a computer program, which implements the steps in the above method embodiment when executed by a processor.

[0017] The data access method provided in the application adds a metadata service component and a file sharing protocol service component in each container service in each controller in the disk array; when a data access request for accessing target file data corresponding to a first disk space managed by a first controller is received from a target device, the target file data is accessed by a first container service in the first controller and a first block service in the first controller, wherein the first container service and the first block service perform data interaction through a first shared memory in the first controller. That is, first, the metadata service component and the file sharing protocol service component are added in each container service of each controller in the embodiment of the application, and the metadata service component and the file sharing protocol service component operate the disk space managed by the block service through the shared memory, thereby shortening the data access path between the block service and the file service. By adding the metadata service component and the file sharing protocol service component in the corresponding controller, a high-performance file interface is provided, and the high performance of the disk array is thus achieved, and high-performance file access is provided. Second, the metadata service component and the file sharing protocol service component are added in each container service in the embodiment of the application, and the relative independence of the file service and the block service is maintained. Through the embodiment of the application, the product integrating the block service and the file service in the related art can provide high-performance file access, but cannot guarantee the process space independence of the block service and the file service. Thus, the process space independence of the block service and the file service can be guaranteed while providing high-performance file access. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 Fig. 1 is a hardware structure block diagram of a computer terminal of a data access method according to an embodiment of the application;

[0019] Figure 2 Fig. 2 is a flowchart of a data access method according to an embodiment of the application;

[0020] Figure 3 Fig. 3 is a schematic diagram of a product integrating a block service and a file service in the related art (I);

[0021] Figure 4 Fig. 4 is a schematic diagram of a product integrating a block service and a file service in the related art (II);

[0022] Figure 5 Fig. 5 is a schematic diagram of a symmetric distributed NAS cluster according to an optional embodiment of the application;

[0023] Figure 6 Fig. 6 is a schematic diagram of an MDS cluster and an MDS map according to an optional embodiment of the application;

[0024] Figure 7is a structural block diagram of a data access device according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] Embodiments of the present application will be described below in detail with reference to the accompanying drawings and in conjunction with embodiments.

[0026] It should be noted that the terms "first", "second", and the like in the description and claims of the present application and the above-described accompanying drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence.

[0027] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal, or a similar computing device. Taking the case of running on a computer terminal, Figure 1 is a hardware structural block diagram of a computer terminal of a data access method according to an embodiment of the present application. As shown in Figure 1 , the computer terminal can include one or more (only one is shown in Figure 1 ) processors 102 (the processor 102 can include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the above-mentioned computer terminal can further include a transmission device 106 for communication function and an input and output device 108. Those skilled in the art can understand that Figure 1 the structure shown is only for illustration, which does not limit the structure of the above-mentioned computer terminal. For example, the computer terminal can further include more or less components than Figure 1 shown, or have a different configuration than Figure 1 shown.

[0028] The memory 104 can be used to store computer programs, for example, software programs of application software and modules, such as the computer program corresponding to the determination method of the interactive state in the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implements the above-mentioned method. The memory 104 can include a high-speed random access memory, and can further include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 104 can further include process memory arranged with respect to the processor 102, and these process memories can be connected to the computer terminal through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0029] Transmission device 106 is used to receive or transmit data via a network. A specific example of such a network may include a wireless network provided by a computer terminal's communications provider. In one embodiment, transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0030] Figure 2 This is a flow chart of a data access method according to an embodiment of the present application, which can be applied to Figure 1 In a computer terminal, such as Figure 2 As shown, the process includes the following steps:

[0031] Step S202: Start a container service in each controller, and add a metadata service component and a file sharing protocol service component to the container image managed by each container service, wherein each controller is a controller in a disk array;

[0032] The metadata service component is a software component used to manage and maintain file system metadata. For a file system, the metadata service component contains metadata information such as the file name, file size, file permissions, file creation time, file modification time, file storage location (i.e., physical address on the disk array), and file directory structure.

[0033] The metadata service component can have the following responsibilities:

[0034] Metadata storage and maintenance: Store and maintain file system metadata information and ensure the integrity and consistency of this information.

[0035] Metadata query: Provides an Application Programming Interface (API) or interface that allows other components and services to query metadata information, such as the physical storage location of a file.

[0036] Metadata update: When file attributes or file system structure changes, for example, a file is modified, moved, or deleted, the metadata service component is responsible for updating the corresponding metadata.

[0037] Consistency maintenance: In a multi-controller cluster, the metadata service component ensures that metadata information on all controllers is synchronized and consistent through cluster consistency protocols, such as distributed consistent hashing.

[0038] The file sharing protocol service component is a software component that allows different devices or applications to access and share file data over a network. The file sharing protocol service component enables file data to be accessed across network boundaries. The functions of the file sharing protocol service component include:

[0039] File access interface: Provides an interface that complies with a specific file sharing protocol, allowing clients (such as other servers, workstations, or applications) to access file data over the network.

[0040] Data transmission: Responsible for processing the sending and receiving of file data on the network, transferring data from the storage device to the requesting party on the network.

[0041] Protocol management: Manages the status of file sharing protocols, processes protocol-related requests and responses, and ensures that access to file data complies with protocol specifications.

[0042] Permission control: Control client access to file data based on the permission information in the file system metadata to prevent unauthorized access or operation.

[0043] Step S204: Upon receiving a data access request from a target device, access the target file data through the first container service in the first controller and the first block service in the first controller, wherein the first container service and the first block service exchange data through the first shared memory in the first controller, and the data access request is a request to access the target file data corresponding to the first disk space managed by the first controller.

[0044] Through the data access method of the present application, a container service is started in each controller in the disk array, and a metadata service component and a file sharing protocol service component are added to the container image managed by each container service; when a data access request is received from a target device to indicate access to the target file data corresponding to the first disk space managed by the first controller, the target file data is accessed through the first container service in the first controller and the first block service in the first controller, wherein the first container service and the first block service interact with each other through the first shared memory in the first controller. That is to say, firstly, the container service of each controller in the embodiment of the present application adds a metadata service component and a file sharing protocol service component, and the metadata service component and the file sharing protocol service component operate the disk space managed by the block service in a shared memory manner, shortening the data access path between the block service and the file service, thereby giving full play to the high performance of the disk array; secondly, the embodiment of the present application adds a metadata service component and a file sharing protocol service component in each container service, maintaining the relative independence of the file service and the block service. The embodiments of the present application can solve the problem in related art where products integrating block services and file services cannot guarantee the independence of the block and file service process spaces while providing high-performance file access. Instead, the embodiments can ensure the independence of the block and file service process spaces while providing high-performance file access.

[0045] Optionally, accessing data includes multiple scenarios, such as deleting file data, reading file data, writing file data, etc. The following are specific steps for reading file data and accessing file data:

[0046] That is, the accessing of the target file data through the first container service in the first controller and the first block service in the first controller in step S204 includes:

[0047] (1) Reading file data: when it is determined that the data access request is used to indicate reading the first file data in the first disk space, determining whether the first file data exists in the first shared memory, wherein the target file data includes: the first file data; when it is determined that the first file data does not exist in the first shared memory, querying the first metadata information corresponding to the first file data in the first metadata service component in the first container service, wherein the first metadata information at least includes: a first address of the first file data; sending the first metadata information to the first block service through the first file sharing protocol service component in the first container service, so that the first block service reads the first file data in the first disk space according to the first metadata information and loads the first file data into the first shared memory; obtaining the first file data in the first shared memory through the first file sharing protocol service component, and sending the first file data to the target device, so that the target device reads the first file data.

[0048] It is understood that when the first container service in the first controller receives a data read request, the first metadata service component in the first container service queries the metadata information of the first file data. The metadata information may include the physical address of the first file data in the disk array, i.e., the first address. This step allows the location of the first file data to be located.

[0049] The file sharing protocol component sends the first metadata information to the first block service. The block service is the component in the disk array that directly manages disk storage space and is responsible for actual data read and write operations. Through the sharing protocol service component, the block service can obtain the physical location information of the required file data.

[0050] After receiving the first metadata information, the first block service reads the first file data from the first disk space according to the first address and loads the read first file data into the first shared memory. The shared memory is a memory area accessible to both the container service and the block service included in each controller.

[0051] Finally, the file sharing protocol service component obtains the first file data from the first shared memory and sends the first file data to the target device. In this way, file data can be read efficiently without directly interacting with the block service or disk array, simplifying the data access process.

[0052] Through the above technical solution, direct communication between metadata services and block services simplifies the data access path, avoids additional overhead when transmitting data between different services, and improves data transmission efficiency.

[0053] (2) Writing file data: When it is determined that the data access request is used to indicate that the second file data is to be written into the first disk space, second metadata information of the second file data is created in the first metadata service component in the first container service, wherein the second metadata information at least includes: a second address of the second file data, and the target file data includes: the second file data; the second file data is written into the first shared memory through the first file sharing protocol service component in the first container service and the second metadata information, so that the first block service writes the second file data in the first shared memory to the second address of the first disk space.

[0054] It is understood that when the first container service in the first controller receives a data write request, the first metadata service component needs to create second metadata information for the second file data. The second metadata information may include the physical storage address of the second file data, i.e., the second address, and file attributes corresponding to the second file data, such as file size and creation time. Creating the second metadata information is the basis for locating and writing the second file data.

[0055] The first file sharing protocol service component uses the second metadata information to write the second file data into the first shared memory. After the second file data is loaded into the first shared memory, the first block service is responsible for writing the second file data from the first shared memory to a second address in the first disk space, where the second address is obtained based on the second metadata information.

[0056] The above technical solution can improve data writing efficiency: by using shared memory as a temporary buffer for data writing, direct disk write operations are reduced, thereby improving the efficiency and speed of data writing. Ensure data consistency: The metadata service component creates metadata information before writing, ensuring that the physical address of the data is correct. The write operation of the block service further ensures the consistency and integrity of the data during storage. Optimize resource utilization: The use of shared memory makes the allocation of CPU and memory resources more reasonable, avoids resource waste caused by frequent disk I / O operations, and improves resource utilization. Simplify the data management process: The creation and management of metadata information simplifies the management complexity of the data writing process, making data writing operations more systematic and automated.

[0057] Among them, before creating the second metadata information of the second file data in the first metadata service component in the first container service, the method also includes: dividing the second file data into multiple target data blocks; performing data block comparison on the multiple target data blocks and the multiple first data blocks in the first disk space, and determining the comparison result; when the comparison result indicates that the multiple first data blocks include the multiple target data blocks, prohibiting the second file data from being written to the first disk space; when the comparison result indicates that the multiple first data blocks include the second data block among the multiple target data blocks, writing the other data blocks among the multiple target data blocks except the second data block to the first disk space, and prohibiting the second data block from being written to the first disk space; when the comparison result indicates that any data block of the multiple target data blocks is not within the multiple first data blocks, determining that the second file data is allowed to be written to the first disk space.

[0058] It is understandable that before creating the second metadata information, data deduplication processing can be performed on the data blocks already stored in the disk array. Specifically:

[0059] The second file data is split into multiple target data blocks to facilitate a more fine-grained comparison with the data blocks in the disk array. The data blocks can be based on fixed-size blocks (e.g., 4kb, 8kb, etc.) or based on an intelligent splitting strategy.

[0060] Each target data block is compared with multiple first data blocks already in the first disk space. This comparison process can use a hash algorithm (such as the message digest algorithm (MD5), the secure hash algorithm (SHA-1), etc.) to convert each data block into a unique hash value, and then compare these hash values ​​to determine whether the data blocks are identical.

[0061] If the comparison results show that all data blocks already exist in the first disk space, the data blocks corresponding to the second data file are duplicates of the data blocks in the disk array, and there is no need to write the second data file to the first disk space. This avoids unnecessary disk writes and waste of storage space.

[0062] If the comparison results show that some data blocks (i.e., second data blocks) in the target data block already exist in the first disk space, while other data blocks do not, the writing process of the second data blocks can be skipped and only the other data blocks can be written to the first disk space, thereby improving data writing efficiency.

[0063] If the comparison result shows that none of the target data blocks is identical to the data blocks stored in the first disk space, it is determined that the second file data is allowed to be completely written into the first disk space.

[0064] Through data deduplication and intelligent data block comparison, storage space usage is significantly reduced, improving storage efficiency. By avoiding unnecessary disk write operations, the system can complete data writing more quickly, improving write speed and system response time.

[0065] Optionally, the accessing of the target file data through the first container service in the first controller and the first block service in the first controller in step S204 includes: determining whether the first controller fails; if it is determined that the first controller fails, determining a second controller in the disk array that manages the first disk space, and accessing the target file data through the second container service in the second controller and the second block service in the second controller, wherein the second container service and the second block service exchange data through the second shared memory in the second controller; and if it is determined that the first controller does not fail, accessing the target file data through the first container service and the first block service.

[0066] It is understandable that the specific steps of accessing the target file data may include:

[0067] The primary controller's health is continuously monitored. If a failure is detected, a failover mechanism is immediately triggered. The system automatically searches the disk array for a second controller that can take over management responsibilities for the primary disk space. This second controller is typically a backup controller in a cluster configuration or a healthy controller in the same failure domain as the primary controller.

[0068] After the second controller is determined, the target file data is accessed through the second container service and the second block service in the second controller. During this process, the second container service and the second block service exchange data through the second shared memory in the second controller to achieve efficient data transmission and caching.

[0069] Through the second shared memory, the second container service can pass file metadata information (such as the physical address of the file, etc.) to the second block service. When processing data reading and writing, the second block service can directly use the data cached in the second shared memory without frequently accessing the disk, thereby improving data access performance.

[0070] If the first controller does not fail, that is, it is in normal operation, the system will continue to access the target file data through the first container service and the first block service, without the need for complex failover operations, maintaining the consistency and efficiency of data access.

[0071] Wherein, determining a second controller that manages the first disk space in the disk array includes: constructing a metadata service component cluster corresponding to the disk array according to each metadata service component in each controller; broadcasting a metadata service component map corresponding to each metadata service component to a container service corresponding to each controller according to the metadata service component cluster, wherein the metadata service component map is used to indicate the running status of each metadata service component; when it is determined that the first controller fails, updating a first metadata service component map corresponding to the first metadata service component, and broadcasting the first metadata service component map to a container service corresponding to each controller, wherein the first metadata service component map is used to indicate that the running status of the first metadata service component is a failure state, the first controller includes: the first metadata service component; remapping the target disk space in the disk array according to the updated first metadata service component map, so as to map the target metadata information of the data in the first disk space managed by the first metadata service component to a healthy metadata service component in the metadata service component cluster, wherein the healthy metadata service component is a metadata service component whose running status is indicated as normal by the metadata service component map; and determining the controller corresponding to the healthy metadata service component as the second controller.

[0072] It is understandable that, when the first controller fails, the specific steps of determining the second controller that manages the first disk space in the disk array may be:

[0073] First, you can build a metadata service component cluster. After the metadata service component cluster is built, the metadata service components on all controllers can collaborate with each other, share metadata information, and provide unified file system services. The construction of the metadata service component cluster ensures high metadata processing efficiency and data consistency.

[0074] The metadata service component map is then regularly broadcast to the container service corresponding to each controller. The metadata service component map contains the operating status of each metadata service component, such as normal operation, failure, and maintenance. Through the metadata service component map, all controllers can obtain real-time information on the health status of each metadata service component in the cluster.

[0075] When a failure of the first controller is detected, the first metadata service component map in the first controller may be updated to indicate that its operating state is a failure state, and the updated first metadata service component map may be broadcasted to the container services corresponding to all controllers.

[0076] Based on the updated first metadata service component map, the target disk space in the disk array is remapped. The target metadata information for the data in the first disk space, originally managed by the failed metadata service component, is remapped to healthy metadata service components in the metadata service component cluster. Healthy metadata service components are those that are operating normally. This remapping process ensures metadata consistency and ensures that data access is not affected by the failure.

[0077] The controller corresponding to the healthy metadata service component takes over the management responsibilities corresponding to the first controller.

[0078] The metadata service component cluster and dynamic remapping mechanism automatically handle controller failures, avoiding single points of failure and enhancing overall system reliability. Furthermore, automated fault detection and metadata remapping simplify the fault recovery process and improve system maintenance efficiency.

[0079] Optionally, after adding the metadata service component and the file sharing protocol service component to the container image managed by each container service in the above step S202, the method further includes: establishing a service map corresponding to the disk array through the interconnection links between each controller and other controllers in the disk array, wherein the service map is used to indicate the first operating state of each container service and the second operating state of each block service in each controller; when it is determined according to the service map that any container service and / or any block service in any controller in the disk array fails, performing fault repair on any container service and / or any block service through a distributed consistency protocol, wherein the fault repair strategy corresponding to each container service in each controller in the disk array is the same, and the fault repair strategy for each block service in each controller in the disk array is the same.

[0080] It can be understood that the file metadata components and file sharing protocol service components between different controllers reuse the interconnection links between disk array controllers to achieve the same fault domain of the file service cluster and block service cluster between controllers, and then the fault repair strategies corresponding to the container service in each controller are the same.

[0081] In order to better understand the process of the above data access method, the implementation process of the above data access method is described in combination with optional embodiments below, but not used to limit the technical scheme of the embodiments of the application.

[0082] Ceph is a distributed storage system that provides high performance, reliability, and scalability solutions for modern large-scale data storage needs. Ceph File System (CephFS) provides a Portable Operating System Interface (POSIX) compatible file system through Meta Data Services (MDS) and clients, supporting access through Network File System (NFS) or direct mounting. However, the deployment and maintenance of Ceph usually requires certain industry knowledge and technical support, as it involves complex cluster management and configuration.

[0083] MDS starts as a single-node service, which has a single point of failure problem. Even if a single node fails, the entire cluster cannot provide file system services. Subsequently, MDS clusters can be supported, i.e., multiple nodes provide services simultaneously, but there are many parameters, complex configuration, and performance cannot be guaranteed.

[0084] Disk arrays are generally combined with hardware and software technologies that are relatively mature and have high performance, but they can only provide block services and cannot adapt to the growing demand for file service scenarios.

[0085] Existing block service and file service integrated products are one that uses a block service and file service stacking method, Figure 3 is a block service and file service integrated product schematic diagram (one), as Figure 3 shown, the bottom layer is a complete block service in the disk array, and the upper layer deploys a file-independent file service through a virtual machine or container method, wherein the block service is not connected through a block interface, and the file service is not connected through a file interface; the other is a block service and file service fusion method, i.e., the disk array, meta-data management block service, and file service disk array are not connected through a block interface and a file interface, Figure 4 is a block service and file service integrated product schematic diagram (two), as Figure 4 shown, there are both block service implementation logic and file service implementation logic in one system, which influence each other, making the implementation complex and unstable.

[0086] The two modes cannot provide high-performance file access while ensuring the independence of the block service and file service process spaces. To solve the above problems, the optional embodiment of the present application provides a symmetric distributed network file system (Network File System, referred to as NAS) cluster. The optional embodiment of the present application utilizes the block interface provided by the existing disk array, and combines the symmetric high reliability and cascading expansion characteristics of the disk array to provide a high-performance file interface by adding file metadata and file sharing protocol services on the corresponding controller. Meanwhile, the file metadata and file sharing protocol of the optional embodiment of the present application are independent service components, which are deployed in a lightweight container service. The CPU, memory, and hard disk can be shared without the disk array, and the process space independence of each other can be ensured.

[0087] Figure 5 is a schematic diagram of a symmetric distributed NAS cluster according to the optional embodiment of the present application, as Figure 5 shown,

[0088] (1) The Pod manager (Podman) container support software package is installed on the disk array controller of the optional embodiment of the present application, which can manage container images and control the start and stop of containers.

[0089] (2) The container image is made, and the file metadata service (i.e., the metadata service component) and the file sharing protocol service component are placed in the container image. When the container service is selected to be started, it is checked whether the container image is loaded. If the image is not loaded, the container image is first imported through the Pod manager import command (podman import). Then, the container is started through the Pod manager run command (podman run), and the resources such as CPU, memory, hard disk directory, and shared memory used by the container are fixed.

[0090] The optional embodiment of the present application utilizes the block interface provided by the existing disk array, and combines the symmetric high reliability and cascading expansion characteristics of the disk array to provide a high-performance file interface by adding file metadata components and file sharing protocol services on the corresponding controller. The file metadata and file sharing protocol are independent service components, which are deployed in a lightweight container service. The CPU, memory, and hard disk can be shared without the disk array, and the process space independence of each other can be ensured.

[0091] (3) Figure 6 is a schematic diagram of an MDS cluster and an MDS map according to the optional embodiment of the present application, as Figure 6As shown, after the file system is created, the MDS (metadata service) in each controller node container collectively forms an MDS cluster. The MDS state of the current controller is reported to the current disk array node, and according to the disk array distributed consistency protocol, the disk array broadcasts the running state map of the MDS of each controller to the container of all nodes, that is, each controller node container knows the MDS that needs to be started by the node and the state of the MDS running in other nodes.

[0092] Based on the above limitations of the optional embodiments of the application:

[0093] First, in each controller of the disk array, a lightweight container service is deployed.

[0094] The container service can be selected by the end user according to the specific product usage scenario. If the container service is not started, only the block interface is provided, and all controller CPU, memory, and hard disk resources are only used for block services.

[0095] If the container service is started, the block interface and the file interface can be provided at the same time, and the controller CPU, memory, and hard disk resources can be divided according to the business scenario. After the container service is started, the health status of the container is managed by the disk array, that is, when the block service on the controller is normal, the container service continues to be provided, and when the block service on the controller is abnormal, the container does not need to continue to run.

[0096] Second, the symmetric file metadata and file sharing protocol service components are deployed inside each controller.

[0097] The file metadata service component and the file sharing protocol service component run as a whole program. In this way, the same cache (i.e., shared memory) can be used between the metadata service and the sharing protocol service, improving the utilization of the cache, reducing the copying of the cache between services, and improving the performance of data access. The file sharing protocol service component on each controller interacts with the metadata service component on the controller as much as possible and operates the disk space preferentially managed by the controller.

[0098] Third, the file metadata and file sharing protocol service components between different controllers form a cluster.

[0099] The metadata service components and file sharing protocol service components between different controllers reuse the interconnect links between disk array controllers, achieving the same fault domain for the file service cluster and block service cluster across controllers. The disk array's message-based distributed consistency protocol ensures data consistency. Similarly, the file service cluster uses the same distributed consistency protocol on the disk array to send service map messages, ensuring consistent startup and shutdown of services within each node container. This means that a service in the file service cluster is only enabled on a single controller at a time.

[0100] Fourth, the metadata service and file sharing protocol service within the container can be dynamically started and exited.

[0101] By default, the file metadata service and file sharing service in the container only operate on the disk space that is preferentially managed by this controller, avoiding data forwarding between controllers. However, when a controller fails, the disk space it manages needs to be taken over by another controller (i.e., the second controller). Similarly, the file metadata service and file sharing protocol service in the container on it also need to be taken over by the container on the other controller. Therefore, the file metadata service component and file sharing protocol service component in the container need to be dynamically started in the container. Conversely, when the failed controller recovers, in order to ensure the pressure of each controller is balanced, the file metadata service and file sharing protocol service that have been taken over need to be exited and started on the corresponding controller.

[0102] Finally, the file metadata service and file sharing protocol service within the container operate on the disk space managed by the block service through shared memory.

[0103] Although the file metadata service and file sharing protocol service run within the container, a portion of memory space is allocated within the block service as shared memory to ensure high data access performance. File data read or written by the user host first reaches the file sharing protocol service and file metadata service through the file interface. The file sharing protocol service then notifies the block service to write the data to disk. The block service then utilizes the shared memory address space to directly write the data from memory to disk, eliminating the need for a memory copy from the file service to the block service. Because of the use of shared memory, file and block data can be compared and deduplicated within the same process space, reducing the amount of data actually written to disk. Furthermore, adding version numbers to the data enables unified snapshot management.

[0104] That is, the container images on each controller of the optional embodiment of the present application are symmetrical, and the file metadata and file sharing protocol service components within the container operate as a whole. The file sharing protocol service components within each controller container interact with the current file metadata service component as much as possible. The file metadata and file sharing protocol service components within the container only operate on the disk space managed by the current controller, reducing data forwarding between controllers. The file metadata and file sharing protocol services within the container operate on the disk space managed by the block service through shared memory. The disk array cluster manages the MDS status map to ensure consistency in the file service management of each controller.

[0105] In summary, the optional embodiment of the present application shortens the data access path between block services and file services by sharing memory between file services and block services, can give full play to the high performance of disk arrays, and achieve high performance of file services; through containers and unified cluster management, it not only maintains the relative independence of file services and block services, but also facilitates cluster consistency management, and there will be no situation where the block service cluster and the file service cluster are out of sync.

[0106] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0107] This embodiment also provides a data access device for implementing the above-mentioned embodiments and preferred implementations. Details already described will not be repeated here. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. While the devices described in the following embodiments are preferably implemented using software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0108] Figure 7 is a structural block diagram of a data access device according to an embodiment of the present application, such as Figure 7 As shown, the device includes:

[0109] A startup module 72 is configured to start a container service in each controller, and to add a metadata service component and a file sharing protocol service component to the container image managed by each container service, wherein each controller is a controller in a disk array;

[0110] The access module 74 is configured to, in response to receiving a data access request sent by a target device, access target file data by a first container service in the first controller to a first block service in the first controller, wherein the first container service and the first block service perform data interaction through a first shared memory in the first controller, and the data access request is a request for accessing target file data corresponding to a first disk space managed by the first controller.

[0111] By the data access apparatus, the container service is enabled in each controller in the disk array, and the metadata service component and the file sharing protocol service component are added in the container image managed by each container service; in response to receiving a data access request sent by a target device for accessing target file data corresponding to a first disk space managed by the first controller, the target file data is accessed by a first container service in the first controller to a first block service in the first controller, and the first container service and the first block service perform data interaction through a first shared memory in the first controller. That is, first, the metadata service component and the file sharing protocol service component are added in the container service of each controller in the embodiment of the application, the metadata service component and the file sharing protocol service component operate the disk space managed by the block service through the shared memory, and the path of data access between the block service and the file service is shortened. By adding the metadata service component and the file sharing protocol service component on the corresponding controller, a high-performance file interface is provided, and the high performance of the disk array is exerted, and high-performance file access is provided. Second, the metadata service component and the file sharing protocol service component are added in each container service in the embodiment of the application, and the relative independence of the file service and the block service is maintained. Through the embodiment of the application, the product integrating the block service and the file service in the related art can provide high-performance file access, but cannot guarantee the process space independence of the block service and the file service. Therefore, the process space independence of the block service and the file service can be guaranteed while providing high-performance file access.

[0112] In an example embodiment, the access module 74 is further configured to, in response to determining that the data access request is for reading first file data in the first disk space, determine whether the first file data exists in the first shared memory, wherein the target file data comprises the first file data; in response to determining that the first file data does not exist in the first shared memory, query a first metadata service component in the first container service for first metadata information corresponding to the first file data, wherein the first metadata information comprises at least a first address of the first file data; send, by a first file sharing protocol service component in the first container service, the first metadata information to the first block service, to cause the first block service to read the first file data in the first disk space according to the first metadata information and load the first file data into the first shared memory; and obtain, by the first file sharing protocol service component, the first file data in the first shared memory and send the first file data to the target device, to cause the target device to read the first file data.

[0113] In an example embodiment, the access module 74 is further configured to, in response to determining that the data access request is for writing second file data into the first disk space, create second metadata information of the second file data in a first metadata service component in the first container service, wherein the second metadata information comprises at least a second address of the second file data, and the target file data comprises the second file data; and write, by a first file sharing protocol service component in the first container service and the second metadata information, the second file data into the first shared memory, to cause the first block service to write the second file data in the first shared memory into the second address of the first disk space.

[0114] In one example embodiment, the access module 74 is further configured to split the second file data into a plurality of target data blocks, perform data block comparison between the plurality of target data blocks and a plurality of first data blocks in the first disk space, and determine a comparison result; in a case that the comparison result indicates that the plurality of first data blocks contain the plurality of target data blocks, prohibit writing the second file data to the first disk space; in a case that the comparison result indicates that the plurality of first data blocks contain a second data block of the plurality of target data blocks, write the plurality of target data blocks except the second data block to the first disk space, and prohibit writing the second data block to the first disk space; and in a case that the comparison result indicates that none of the plurality of target data blocks is in the plurality of first data blocks, determine to allow writing the second file data to the first disk space.

[0115] In one example embodiment, the access module 74 is further configured to determine whether the first controller fails; in a case that the first controller is determined to fail, determine a second controller in the disk array that manages the first disk space, and access the target file data through a second container service in the second controller and a second block service in the second controller, wherein the second container service and the second block service perform data interaction through a second shared memory in the second controller; and in a case that the first controller is determined not to fail, access the target file data through the first container service and the first block service.

[0116] In an example embodiment, the access module 74 is further configured to construct a cluster of metadata service components corresponding to the disk array according to each metadata service component in each controller; broadcast a metadata service component map corresponding to each metadata service component to a container service corresponding to each controller according to the cluster of metadata service components, wherein the metadata service component map is used to indicate a running state of each metadata service component; update a first metadata service component map corresponding to the first metadata service component and broadcast the first metadata service component map to the container service corresponding to each controller upon determining that the first controller fails, wherein the first metadata service component map is used to indicate that the running state of the first metadata service component is a failure state, and the first controller comprises the first metadata service component; remap a target disk space in the disk array according to the updated first metadata service component map to map target metadata information of data in the first disk space managed by the first metadata service component to a healthy metadata service component in the cluster of metadata service components, wherein the healthy metadata service component is a metadata service component indicated as normal in the metadata service component map; and determine a controller corresponding to the healthy metadata service component as the second controller.

[0117] In an example embodiment, the enabling module 72 is further configured to establish a service map corresponding to the disk array through an interconnection link between each controller in the disk array and another controller, wherein the service map is used to indicate a first running state of each container service and a second running state of each block service in each controller; perform failure repair on any container service and / or any block service in any controller in the disk array through a distributed consistency protocol upon determining that the any container service and / or the any block service fails according to the service map, wherein a failure repair strategy of each container service in each controller in the disk array is the same, and a failure repair strategy of each block service in each controller in the disk array is the same.

[0118] Embodiments of the present application also provide a computer readable storage medium having a computer program stored therein, wherein the computer program is configured to execute the steps in the above method embodiments when running.

[0119] In an example embodiment, the computer readable storage medium described above can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a flash hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0120] Optionally, in the embodiment, the storage medium described above can be configured to store program codes for performing the following steps:

[0121] S1, starting a container service in each controller, and adding a metadata service component and a file sharing protocol service component in a container image managed by each container service, wherein each controller is a controller in a disk array;

[0122] S2, in response to receiving a data access request sent by a target device, accessing target file data by a first block service in the first controller through a first container service in the first controller, wherein the first container service and the first block service perform data interaction through a first shared memory in the first controller, and the data access request is a request for accessing target file data corresponding to a first disk space managed by the first controller.

[0123] Embodiments of the present application also provide an electronic device including a memory and a processor, the memory storing a computer program, and the processor being configured to run the computer program to perform the steps in the above method embodiments.

[0124] In an example embodiment, the electronic device described above can further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0125] Optionally, in the embodiment, the processor described above can be configured to perform the following steps through the computer program:

[0126] S1, starting a container service in each controller, and adding a metadata service component and a file sharing protocol service component in a container image managed by each container service, wherein each controller is a controller in a disk array;

[0127] S2, in response to receiving a data access request sent by a target device, accessing target file data by a first block service in the first controller through a first container service in the first controller, wherein the first container service and the first block service perform data interaction through a first shared memory in the first controller, and the data access request is a request for accessing target file data corresponding to a first disk space managed by the first controller.

[0128] The embodiment of the present application also provides a computer program product comprising a computer program which, when executed by a processor, implements the steps in the above method embodiment.

[0129] Optionally, in the embodiment, the computer program product can be executed by the processor to perform the following steps:

[0130] S1, enabling a container service in each controller, and adding a metadata service component and a file sharing protocol service component in a container image managed by each container service, wherein the controller is a controller in a disk array;

[0131] S2, in response to receiving a data access request sent by a target device, accessing target file data by a first block service in the first controller through a first container service in the first controller, wherein the first container service and the first block service perform data interaction through a first shared memory in the first controller, and the data access request is a request for accessing target file data corresponding to a first disk space managed by the first controller.

[0132] The specific examples in the embodiment can refer to the examples described in the above embodiments and exemplary embodiments, and the embodiment will not be described here again.

[0133] Obviously, those skilled in the art should understand that each module or each step of the present application described above can be realized by a general computing device, which can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and can be realized by program codes executable by the computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in different order, or they can be manufactured into each integrated circuit module, or multiple modules or steps can be manufactured into a single integrated circuit module. Thus, the present application is not limited to any specific combination of hardware and software.

[0134] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for accessing data, characterized in that: include: Start a container service in each controller, and add a metadata service component and a file sharing protocol service component to the container image managed by each container service, wherein each controller is a controller in a disk array; Upon receiving a data access request sent by a target device, target file data is accessed through a first container service in a first controller and a first block service in the first controller, wherein the first container service and the first block service exchange data through a first shared memory in the first controller, and the data access request is a request for accessing target file data corresponding to a first disk space managed by the first controller.

2. The method according to claim 1, characterized in that Accessing target file data through a first container service in a first controller and a first block service in the first controller includes: In a case where it is determined that the data access request is for instructing to read first file data in the first disk space, determining whether the first file data exists in the first shared memory, wherein the target file data includes: the first file data; When it is determined that the first file data does not exist in the first shared memory, querying a first metadata service component in the first container service for first metadata information corresponding to the first file data, wherein the first metadata information at least includes: a first address of the first file data; sending the first metadata information to the first block service through the first file sharing protocol service component in the first container service, so that the first block service reads the first file data in the first disk space according to the first metadata information and loads the first file data into the first shared memory; The first file data is obtained from the first shared memory through the first file sharing protocol service component, and the first file data is sent to the target device, so that the target device reads the first file data.

3. The method according to claim 1, characterized in that Accessing target file data through a first container service in a first controller and a first block service in the first controller includes: When it is determined that the data access request is for instructing to write second file data into the first disk space, second metadata information of the second file data is created in a first metadata service component in the first container service, wherein the second metadata information at least includes: a second address of the second file data, and the target file data includes: the second file data; The second file data is written into the first shared memory through the first file sharing protocol service component in the first container service and the second metadata information, so that the first block service writes the second file data in the first shared memory to the second address of the first disk space.

4. The method according to claim 3, characterized in that Before creating second metadata information of the second file data in the first metadata service component in the first container service, the method further includes: dividing the second file data into a plurality of target data blocks; Performing data block comparison on the plurality of target data blocks and the plurality of first data blocks in the first disk space, and determining a comparison result; If the comparison result indicates that the plurality of first data blocks include the plurality of target data blocks, prohibiting writing the second file data to the first disk space; If the comparison result indicates that the plurality of first data blocks include the second data block among the plurality of target data blocks, writing the other data blocks among the plurality of target data blocks except the second data block to the first disk space, and prohibiting the second data block from being written to the first disk space; If the comparison result indicates that any data block of the plurality of target data blocks is not within the plurality of first data blocks, it is determined that the second file data is allowed to be written to the first disk space.

5. The method according to claim 1, wherein Accessing target file data through a first container service in a first controller and a first block service in the first controller includes: determining whether the first controller fails; If it is determined that the first controller fails, determining a second controller in the disk array that manages the first disk space, and accessing the target file data through a second container service in the second controller and a second block service in the second controller, wherein the second container service and the second block service exchange data through a second shared memory in the second controller; If it is determined that the first controller has not failed, the target file data is accessed through the first container service and the first block service.

6. The method according to claim 5, characterized in that Determining a second controller in the disk array that manages the first disk space includes: Building a metadata service component cluster corresponding to the disk array according to each metadata service component in each controller; Broadcasting the metadata service component map corresponding to each metadata service component to the container service corresponding to each controller according to the metadata service component cluster, wherein the metadata service component map is used to indicate the running status of each metadata service component; In a case where it is determined that the first controller fails, a first metadata service component map corresponding to the first metadata service component is updated, and the first metadata service component map is broadcast to the container service corresponding to each controller, wherein the first metadata service component map is used to indicate that the operating state of the first metadata service component is a failure state, and the first controller includes: the first metadata service component; Remapping the target disk space in the disk array according to the updated first metadata service component map, so as to map the target metadata information of the data in the first disk space managed by the first metadata service component to the healthy metadata service component in the metadata service component cluster, wherein the healthy metadata service component is a metadata service component whose operating status is indicated to be normal by the metadata service component map; The controller corresponding to the health metadata service component is determined as the second controller.

7. The method according to claim 1, characterized in that After adding the metadata service component and the file sharing protocol service component to each container image managed by the container service, the method further includes: Establishing a service map corresponding to the disk array through interconnection links between each controller in the disk array and other controllers, wherein the service map is used to indicate a first operating state of each container service and a second operating state of each block service in each controller; When it is determined according to the service map that any container service and / or any block service in any controller in the disk array fails, the fault is repaired for the any container service and / or any block service through a distributed consistency protocol, wherein the fault repair strategy corresponding to each container service in each controller in the disk array is the same, and the fault repair strategy for each block service in each controller in the disk array is the same.

8. A data access device, characterized in that: include: A startup module, configured to start a container service in each controller and add a metadata service component and a file sharing protocol service component to the container image managed by each container service, wherein each controller is a controller in a disk array; An access module is configured to, upon receiving a data access request sent by a target device, access target file data through a first container service in a first controller and a first block service in the first controller, wherein the first container service and the first block service exchange data through a first shared memory in the first controller, and the data access request is a request to access target file data corresponding to a first disk space managed by the first controller.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method according to any one of claims 1 to 7 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images