Data access method and device, storage medium and electronic equipment

Abstract

The embodiment of the invention provides a data access method and device, a storage medium and electronic device.The data access method comprises the steps that container service is started in each controller, and a metadata service component and a file sharing protocol service component are added in a container mirror image managed by each container service, each controller is a controller in a disk array; under the condition that a data access request sent by target equipment is received, accessing target file data through a first container service in a first controller and a first block service in the first controller, the first container service and the first block service perform data interaction through a first shared memory in the first controller, and the data access request is a request for accessing target file data corresponding to a first disk space managed by the first controller.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of circuits, and in particular, to a data access method and device, a storage medium, and an electronic device. Background Art

[0002] Disk arrays are usually used in conjunction with existing hardware, and their software technology is relatively mature and has high performance. However, since they can only provide block services, they cannot adapt to the current file service demand scenarios with growing logs.

[0003] The existing products that integrate block services and file services use the superposition of block services and file services. The bottom layer is a complete block service, and the upper layer deploys file-independent file services through virtual machines or containers. However, through the superposition of block services and file services, block services and file services are completely isolated, and the path that users take when accessing data through file interfaces is long, and high-performance file access cannot be provided; the other is the fusion of block services and file services. In one system, there is both the logic of block service implementation and the logic of file service implementation mixed, which affect each other and make the implementation complex and unstable. However, through the fusion of block services and file services, block services and file services are completely integrated, and block interfaces and file interfaces are mixed together, which makes the implementation complex. Modifying one will inevitably affect the other, which is easy to introduce problems and cannot be independently upgraded and maintained.

[0004] Therefore, for products integrating block services and file services in related technologies, there is a problem that they cannot provide high-performance file access while ensuring the independence of the block service and file service process spaces, which has not yet been effectively solved. Summary of the invention

[0005] The embodiments of the present application provide a data access method and device, a storage medium and an electronic device to at least solve the problem that products integrating block services and file services in the related art cannot provide high-performance file access while ensuring the independence of the block service and file service process space.

[0006] According to an embodiment of the present application, a data access method is provided, specifically comprising: starting a container service in each controller, and adding a metadata service component and a file sharing protocol service component in a container image managed by each container service, wherein each controller is a controller in a disk array; upon receiving a data access request sent by a target device, accessing target file data through a first container service in a first controller and a first block service in the first controller, wherein the first container service and the first block service perform data interaction through a first shared memory in the first controller, and the data access request is a request for accessing target file data corresponding to a first disk space managed by the first controller.

[0007] In an exemplary embodiment, accessing target file data through a first container service in a first controller and a first block service in the first controller includes: when it is determined that the data access request is used to indicate reading the first file data in the first disk space, determining whether the first file data exists in the first shared memory, wherein the target file data includes: the first file data; when it is determined that the first file data does not exist in the first shared memory, querying first metadata information corresponding to the first file data in a first metadata service component in the first container service, wherein the first metadata information includes at least: a first address of the first file data; sending the first metadata information to the first block service through a first file sharing protocol service component in the first container service, so that the first block service reads the first file data in the first disk space according to the first metadata information and loads the first file data into the first shared memory; obtaining the first file data in the first shared memory through the first file sharing protocol service component, and sending the first file data to the target device, so that the target device reads the first file data.

[0008] In an exemplary embodiment, accessing target file data through a first container service in a first controller and a first block service in the first controller includes: when determining that the data access request is used to indicate writing the second file data to the first disk space, creating second metadata information of the second file data in a first metadata service component in the first container service, wherein the second metadata information at least includes: a second address of the second file data, and the target file data includes: the second file data; writing the second file data to the first shared memory through a first file sharing protocol service component in the first container service and the second metadata information, so that the first block service writes the second file data in the first shared memory to the second address of the first disk space.

[0009] In an exemplary embodiment, before creating the second metadata information of the second file data in the first metadata service component in the first container service, the method further includes: dividing the second file data into a plurality of target data blocks; performing a data block comparison between the plurality of target data blocks and a plurality of first data blocks in the first disk space, and determining a comparison result; when the comparison result indicates that the plurality of first data blocks include the plurality of target data blocks, prohibiting the second file data from being written to the first disk space; when the comparison result indicates that the plurality of first data blocks include a second data block among the plurality of target data blocks, writing the other data blocks among the plurality of target data blocks except the second data block to the first disk space, and prohibiting the second data block from being written to the first disk space; when the comparison result indicates that any data block of the plurality of target data blocks is not within the plurality of first data blocks, determining that the second file data is allowed to be written to the first disk space.

[0010] In an exemplary embodiment, accessing target file data through a first container service in a first controller and a first block service in the first controller includes: determining whether the first controller fails; if it is determined that the first controller fails, determining a second controller that manages the first disk space in the disk array, and accessing the target file data through a second container service in the second controller and a second block service in the second controller, wherein the second container service and the second block service exchange data through a second shared memory in the second controller; if it is determined that the first controller does not fail, accessing the target file data through the first container service and the first block service.

[0011] In an exemplary embodiment, determining a second controller that manages the first disk space in the disk array includes: constructing a metadata service component cluster corresponding to the disk array according to each metadata service component in each controller; broadcasting a metadata service component map corresponding to each metadata service component to a container service corresponding to each controller according to the metadata service component cluster, wherein the metadata service component map is used to indicate the running status of each metadata service component; when it is determined that the first controller fails, updating a first metadata service component map corresponding to the first metadata service component, and broadcasting the first metadata service component map to a container service corresponding to each controller, wherein the first metadata service component map is used to indicate that the running status of the first metadata service component is a faulty state, and the first controller includes: the first metadata service component; remapping the target disk space in the disk array according to the updated first metadata service component map, so as to map the target metadata information of the data in the first disk space managed by the first metadata service component to a healthy metadata service component in the metadata service component cluster, wherein the healthy metadata service component is a metadata service component whose running status is indicated as normal by the metadata service component map; determining the controller corresponding to the healthy metadata service component as the second controller.

[0012] In an exemplary embodiment, after adding a metadata service component and a file sharing protocol service component to the container image managed by each container service, the method further includes: establishing a service map corresponding to the disk array through the interconnection links between each controller in the disk array and other controllers, wherein the service map is used to indicate a first operating state of each container service and a second operating state of each block service in each controller; when it is determined according to the service map that any container service and / or any block service in any controller in the disk array fails, performing fault repair on any container service and / or any block service through a distributed consistency protocol, wherein the fault repair strategy corresponding to each container service in each controller in the disk array is the same, and the fault repair strategy for each block service in each controller in the disk array is the same.

[0013] According to another embodiment of the present application, a data access device is provided, including: a startup module, used to start a container service in each controller, and add a metadata service component and a file sharing protocol service component in the container image managed by each container service, wherein each controller is a controller in a disk array; an access module, used to access target file data through a first container service in a first controller and a first block service in the first controller when receiving a data access request sent by a target device, wherein the first container service and the first block service perform data interaction through a first shared memory in the first controller, and the data access request is a request to access the target file data corresponding to a first disk space managed by the first controller.

[0014] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps in the above method embodiment when running.

[0015] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in the above method embodiment.

[0016] According to another embodiment of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiment are implemented.

[0017] Through the data access method of the present application, a container service is started in each controller in the disk array, and a metadata service component and a file sharing protocol service component are added to the container image managed by each container service; when a data access request for indicating access to the target file data corresponding to the first disk space managed by the first controller is received from the target device, the target file data is accessed through the first container service in the first controller and the first block service in the first controller, wherein the first container service and the first block service perform data interaction through the first shared memory in the first controller. That is to say, firstly, a metadata service component and a file sharing protocol service component are added to the container service of each controller in the embodiment of the present application, and the metadata service component and the file sharing protocol service component operate the disk space managed by the block service in a shared memory manner, shortening the data access path between the block service and the file service. In addition, by adding the metadata service component and the file sharing protocol service component to the corresponding controller, a high-performance file interface is provided, thereby giving play to the high performance of the disk array and providing high-performance file access. Secondly, the embodiment of the present application adds the metadata service component and the file sharing protocol service component to each container service, maintaining the relative independence of the file service and the block service. Through the embodiments of the present application, the problem that the block service and file service integrated products in the related art cannot guarantee the independence of the block service and file service process spaces while providing high-performance file access can be solved. While providing high-performance file access, the independence of the block service and file service process spaces can be guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a hardware structure block diagram of a computer terminal of a data access method according to an embodiment of the present application;

[0019] Figure 2 is a flow chart of a method for accessing data according to an embodiment of the present application;

[0020] Figure 3 It is a schematic diagram of a product integrating block service and file service in the related art (I);

[0021] Figure 4 It is a product schematic diagram of the integration of block service and file service in the related art (II);

[0022] Figure 5 is a schematic diagram of a symmetrical distributed NAS cluster according to an optional embodiment of the present application;

[0023] Figure 6 is a schematic diagram of an MDS cluster and an MDS map according to an optional embodiment of the present application;

[0024] Figure 7It is a structural block diagram of a data access device according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0027] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 1 is a hardware structure block diagram of a computer terminal of a data access method according to an embodiment of the present application. Figure 1 As shown, the computer terminal may include one or more ( Figure 1 The computer terminal includes a processor 102 (only one is shown) (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA and the like) and a memory 104 for storing data, wherein the computer terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. Figure 1 More or fewer components than shown, or with different Figure 1 Different configurations shown.

[0028] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for determining the interactive state in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory set relative to the processor 102 process, and these process memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0029] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of a computer terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0030] Figure 2 is a flowchart of a data access method according to an embodiment of the present application, which can be applied to Figure 1 In a computer terminal, such as Figure 2 As shown, the process includes the following steps:

[0031] Step S202, starting a container service in each controller, and adding a metadata service component and a file sharing protocol service component in a container image managed by each container service, wherein each controller is a controller in a disk array;

[0032] The metadata service component is a software component used to manage and maintain file system metadata. For a file system, the metadata information contained in the metadata service component includes: file name, file size, file permissions, file creation time, file modification time, file storage location (i.e., physical address on the disk array), file directory structure and other information.

[0033] A metadata service component can have the following responsibilities:

[0034] Metadata storage is not maintained: Store and maintain the metadata information of the file system and ensure the integrity and consistency of this information.

[0035] Metadata query: Provides an application programming interface (API) or interface that allows other components and services to query metadata information, such as the physical storage location of a file.

[0036] Metadata update: When file attributes or file system structure changes, for example, a file is modified, moved, or deleted, the metadata service component is responsible for updating the corresponding metadata.

[0037] Consistency maintenance: In a multi-controller cluster, the metadata service component ensures that metadata information on all controllers is synchronized and consistent through cluster consistency protocols, such as distributed consistent hashing.

[0038] The file sharing protocol service component is a software component that allows different devices or applications to access and share file data through the network. The file sharing protocol service component enables file data to be accessed across network boundaries. The functions of the file sharing protocol service component include:

[0039] File access interface: Provides an interface that complies with a specific file sharing protocol, allowing clients (such as other servers, workstations, or applications) to access file data over the network.

[0040] Data transmission: Responsible for processing the sending and receiving of file data on the network, transferring data from the storage device to the requester on the network.

[0041] Protocol management: manage the status of file sharing protocols, process protocol-related requests and responses, and ensure that access to file data complies with protocol specifications.

[0042] Permission control: Control client access to file data based on the permission information in the file system metadata to prevent unauthorized access or operation.

[0043] Step S204: upon receiving a data access request sent by a target device, access the target file data through a first container service in a first controller and a first block service in the first controller, wherein the first container service and the first block service perform data interaction through a first shared memory in the first controller, and the data access request is a request for accessing the target file data corresponding to a first disk space managed by the first controller.

[0044] Through the data access method of the present application, a container service is started in each controller in the disk array, and a metadata service component and a file sharing protocol service component are added to the container image managed by each container service; when a data access request for indicating access to the target file data corresponding to the first disk space managed by the first controller is received from the target device, the target file data is accessed through the first container service in the first controller and the first block service in the first controller, wherein the first container service and the first block service perform data interaction through the first shared memory in the first controller. That is to say, firstly, a metadata service component and a file sharing protocol service component are added to the container service of each controller in the embodiment of the present application, and the metadata service component and the file sharing protocol service component operate the disk space managed by the block service in a shared memory manner, shortening the data access path between the block service and the file service, thereby exerting the high performance of the disk array, and secondly, the embodiment of the present application adds a metadata service component and a file sharing protocol service component to each container service, maintaining the relative independence of the file service and the block service. Through the embodiments of the present application, the problem that the block service and file service integrated products in the related art cannot guarantee the independence of the block service and file service process spaces while providing high-performance file access can be solved. While providing high-performance file access, the independence of the block service and file service process spaces can be guaranteed.

[0045] Optionally, accessing data includes multiple situations, such as: deleting file data, reading file data, writing file data, etc. The following are specific method steps for reading file data and accessing file data:

[0046] That is, the accessing of the target file data through the first container service in the first controller and the first block service in the first controller in step S204 includes:

[0047] (1) Reading file data: when it is determined that the data access request is used to indicate reading the first file data in the first disk space, determining whether the first file data exists in the first shared memory, wherein the target file data includes: the first file data; when it is determined that the first file data does not exist in the first shared memory, querying the first metadata information corresponding to the first file data in the first metadata service component in the first container service, wherein the first metadata information includes at least: a first address of the first file data; sending the first metadata information to the first block service through the first file sharing protocol service component in the first container service, so that the first block service reads the first file data in the first disk space according to the first metadata information and loads the first file data into the first shared memory; obtaining the first file data in the first shared memory through the first file sharing protocol service component, and sending the first file data to the target device, so that the target device reads the first file data.

[0048] It can be understood that when the first container service in the first controller receives a data read request, the first metadata service component in the first container service queries the metadata information of the first file data, and the metadata information may include the physical address of the first file data in the disk array, that is, the first address. Through this step, the location of the first file data can be located.

[0049] The file sharing protocol component sends the first metadata information to the first block service. The block service is a component in the disk array that directly manages the disk storage space and is responsible for the actual data read and write operations. Through the sharing protocol service component, the block service can obtain the physical location information of the required file data.

[0050] After receiving the first metadata information, the first block service reads the first file data from the first disk space according to the first address, and loads the read first file data into the first shared memory, wherein the shared memory is a memory area accessible to both the container service and the block service contained in each controller.

[0051] Finally, the file sharing protocol service component obtains the first file data from the first shared memory and sends the first file data to the target device. In this way, the file data can be read efficiently without directly interacting with the block service or the disk array, thus simplifying the data access process.

[0052] Through the above technical solution, direct communication between metadata service and block service simplifies the data access path, avoids the additional overhead when transmitting data between different services, and improves data transmission efficiency.

[0053] (2) Writing file data: when it is determined that the data access request is used to indicate that the second file data is to be written into the first disk space, second metadata information of the second file data is created in the first metadata service component in the first container service, wherein the second metadata information at least includes: a second address of the second file data, and the target file data includes: the second file data; the second file data is written into the first shared memory through the first file sharing protocol service component in the first container service and the second metadata information, so that the first block service writes the second file data in the first shared memory to the second address of the first disk space.

[0054] It can be understood that when the first container service in the first controller receives a data write request, the first metadata service component needs to create second metadata information of the second file data, and the second metadata information may include the physical storage address of the second file data, that is, the second address, and the file attributes corresponding to the second file data, such as: file size, creation time, etc. The creation of the second metadata information is the basis for locating and writing the second file data.

[0055] The first file sharing protocol service component will use the second metadata information to write the second file data into the first shared memory. After the second file data is loaded into the first shared memory, the first block service is responsible for writing the second file data from the first shared memory to the second address in the first disk space, wherein the second address is obtained through the second metadata information.

[0056] The above technical solution can improve the efficiency of data writing: by using shared memory as a temporary buffer for data writing, the direct writing operation of the disk is reduced, thereby improving the efficiency and speed of data writing. Ensure data consistency: The metadata service component creates metadata information before writing, ensuring that the physical address of the data is correct, and further ensures the consistency and integrity of the data during storage through the writing operation of the block service. Optimize resource utilization: The use of shared memory makes the allocation of CPU and memory resources more reasonable, avoids resource waste caused by frequent disk I / O operations, and improves resource utilization. Simplify the data management process: The creation and management of metadata information simplifies the management complexity in the data writing process, making the data writing operation more systematic and automatic.

[0057] Among them, before creating the second metadata information of the second file data in the first metadata service component in the first container service, the method also includes: dividing the second file data into multiple target data blocks; performing data block comparison on the multiple target data blocks and the multiple first data blocks in the first disk space, and determining the comparison result; when the comparison result indicates that the multiple first data blocks include the multiple target data blocks, prohibiting the second file data from being written to the first disk space; when the comparison result indicates that the multiple first data blocks include the second data block among the multiple target data blocks, writing the other data blocks among the multiple target data blocks except the second data block to the first disk space, and prohibiting the second data block from being written to the first disk space; when the comparison result indicates that any data block of the multiple target data blocks is not within the multiple first data blocks, determining that the second file data is allowed to be written to the first disk space.

[0058] It is understandable that before creating the second metadata information, data deduplication processing may be performed on the data blocks already stored in the disk array, specifically:

[0059] The second file data is divided into multiple target data blocks to facilitate a more fine-grained comparison with the data blocks in the disk array. The data blocks can be based on fixed-size blocks (e.g., 4kb, 8kb, etc.) or based on an intelligent segmentation strategy.

[0060] Each target data block is compared with multiple first data blocks already in the first disk space. This comparison process can use a hash algorithm (such as message digest algorithm (MD5), secure hash algorithm (SHA-1), etc.) to convert each data block into a unique hash value, and then compare these hash values ​​to determine whether the data blocks are the same.

[0061] If the comparison result shows that all data blocks already exist in the first disk space, it means that the data blocks corresponding to the second data file are duplicated with the data blocks in the disk array, and there is no need to write the second data file to the first disk space, thereby avoiding unnecessary disk writing and waste of storage space.

[0062] If the comparison result shows that some data blocks (i.e., the second data blocks) in the target data blocks already exist in the first disk space, while other data blocks do not exist in the first disk space, the writing process of the second data blocks can be skipped, and only the other data blocks can be written to the first disk space, thereby improving the efficiency of data writing.

[0063] When the comparison result shows that none of the target data blocks is identical to the data blocks stored in the first disk space, it is determined that the second file data is allowed to be completely written into the first disk space.

[0064] Through data deduplication and intelligent data block comparison, storage space usage is significantly reduced and storage efficiency is improved. By avoiding unnecessary disk write operations, the system can complete the data write process faster, improving the write speed and system response time.

[0065] Optionally, the accessing of the target file data through the first container service in the first controller and the first block service in the first controller in the above step S204 includes: determining whether the first controller fails; when it is determined that the first controller fails, determining a second controller that manages the first disk space in the disk array, and accessing the target file data through the second container service in the second controller and the second block service in the second controller, wherein the second container service and the second block service exchange data through the second shared memory in the second controller; when it is determined that the first controller does not fail, accessing the target file data through the first container service and the first block service.

[0066] It is understandable that the specific steps of accessing the target file data may include:

[0067] Continuously monitor the health status of the first controller, and immediately trigger the failover mechanism once the first controller fails. Automatically find and determine the second controller in the disk array that can take over the management responsibility of the first disk space. The second controller is usually a healthy controller that serves as a backup for the first controller in a cluster configuration or is in the same fault domain.

[0068] After the second controller is determined, the target file data is accessed through the second container service and the second block service in the second controller. In this process, the second container service and the second block service perform data interaction through the second shared memory in the second controller to achieve efficient data transmission and caching.

[0069] Through the second shared memory, the second container service can pass file metadata information (such as the physical address of the file, etc.) to the second block service. When processing data reading and writing, the second block service can directly use the data cached in the second shared memory without frequently accessing the disk, thereby improving data access performance.

[0070] When the first controller does not fail, that is, it is in normal operation, the system will continue to access the target file data through the first container service and the first block service without the need for complex failover operations, thereby maintaining the consistency and efficiency of data access.

[0071] Wherein, determining the second controller that manages the first disk space in the disk array includes: constructing a metadata service component cluster corresponding to the disk array according to each metadata service component in each controller; broadcasting the metadata service component map corresponding to each metadata service component to the container service corresponding to each controller according to the metadata service component cluster, wherein the metadata service component map is used to indicate the running status of each metadata service component; when it is determined that the first controller fails, updating the first metadata service component map corresponding to the first metadata service component, and broadcasting the first metadata service component map to the container service corresponding to each controller, wherein the first metadata service component map is used to indicate that the running status of the first metadata service component is a faulty state, and the first controller includes: the first metadata service component; remapping the target disk space in the disk array according to the updated first metadata service component map, so as to map the target metadata information of the data in the first disk space managed by the first metadata service component to the healthy metadata service component in the metadata service component cluster, wherein the healthy metadata service component is a metadata service component whose running status is indicated as normal by the metadata service component map; determining the controller corresponding to the healthy metadata service component as the second controller.

[0072] It is understandable that, in the case where the first controller fails, the specific steps of determining the second controller that manages the first disk space in the disk array may be:

[0073] First, you can build a metadata service component cluster. After building the metadata service component cluster, the metadata service components on all controllers can cooperate with each other and share metadata information to provide a unified file system service. The construction of the metadata service component cluster ensures high efficiency of metadata processing and data consistency.

[0074] Then, the metadata service component map can be broadcasted to the container service corresponding to each controller regularly. The metadata service component map contains the operating status information of each metadata service component, such as normal operation, failure, maintenance, etc. Through the metadata service component map, all controllers can understand the health status of each metadata service component in the cluster in real time.

[0075] When a failure of the first controller is detected, the first metadata service component map in the first controller can be updated to indicate that its running state is a failure state, and the updated first metadata service component map is broadcast to container services corresponding to all controllers.

[0076] According to the updated first metadata service component map, the target disk space in the disk array is remapped. The target metadata information of the data in the first disk space originally managed by the failed metadata service component is remapped to the healthy metadata service component in the metadata service component cluster. Healthy metadata service components are those metadata service components that are in a "normal" operating state. The remapping process can ensure the consistency of metadata and ensure that data access is not affected by the failure.

[0077] The controller corresponding to the healthy metadata service component takes over the management responsibilities corresponding to the first controller.

[0078] The metadata service component cluster and dynamic remapping mechanism can automatically handle controller failures, avoid single point failures, and enhance the overall reliability of the system. In addition, the automated fault detection and metadata remapping mechanism simplify the fault recovery process and improve the efficiency of system maintenance.

[0079] Optionally, after adding a metadata service component and a file sharing protocol service component to the container image managed by each container service in the above step S202, the method further includes: establishing a service map corresponding to the disk array through the interconnection links between each controller in the disk array and other controllers, wherein the service map is used to indicate a first operating state of each container service and a second operating state of each block service in each controller; when it is determined according to the service map that any container service and / or any block service in any controller in the disk array fails, performing fault repair on any container service and / or any block service through a distributed consistency protocol, wherein the fault repair strategy corresponding to each container service in each controller in the disk array is the same, and the fault repair strategy for each block service in each controller in the disk array is the same.

[0080] It can be understood that the file metadata components and file sharing protocol service components between different controllers reuse the interconnection links between disk array controllers to achieve the same fault domain of the file service cluster and block service cluster between controllers, and then the fault repair strategies corresponding to the container service in each controller are the same.

[0081] In order to better understand the process of the above-mentioned data access method, the implementation process of the above-mentioned data access method is described below in combination with an optional embodiment, but it is not used to limit the technical solution of the embodiment of the present application.

[0082] Ceph is a distributed storage system that provides high-performance, reliable, and scalable solutions for modern large-scale data storage needs. The Ceph File System (CephFS) provides a Portable Operating System Interface (POSIX)-compatible file system through Meta Data Services (MDS) and Client, and supports access through Network File System (NFS) or direct mounting. However, the deployment and maintenance of Ceph usually requires certain industry knowledge and technical support because it involves complex cluster management and configuration.

[0083] MDS initially served a single node, which had a single point of failure problem. Even if a single node failed, the entire cluster would be unable to provide file system services. MDS clusters can be supported later, that is, multiple nodes provide services at the same time, but there are more parameters, the configuration is complex, and the performance cannot be guaranteed.

[0084] Disk arrays are usually used in conjunction with existing hardware, and their software technology is relatively mature and has high performance. However, since they can only provide block services, they cannot adapt to the current file service demand scenarios with growing logs.

[0085] There are existing products that integrate block services and file services. One is to use the superposition of block services and file services. Figure 3 It is a schematic diagram of a product integrating block service and file service in related technology (I), such as Figure 3 As shown in the figure, the bottom layer is a complete block service in the disk array, and the upper layer deploys file-independent file services through virtual machines or containers. The block service is not connected to the block interface, and the file service is not connected to the file interface. The other is a fusion of block service and file service, that is, the disk array, metadata management block service and file service disk array are not connected to the block interface and the file interface. Figure 4 This is a product diagram of the integration of block service and file service in the related technology (II), such as Figure 4 As shown, in one system, there is both block service implementation logic and file service implementation logic mixed together, which influence each other and make the implementation complex and unstable.

[0086] The above two methods have the problem that they cannot provide high-performance file access while ensuring the independence of block services and file service process spaces. To address the above problem, an optional embodiment of the present application provides a symmetrical distributed network file system (Network File System, abbreviated as NAS) cluster. The optional embodiment of the present application utilizes the block interface provided by the existing disk array, and combines the characteristics of symmetrical high reliability and cascading expansion of the disk array, and provides a high-performance file interface by adding file metadata and file sharing protocol service implementations on the corresponding controller; at the same time, the file metadata and file sharing protocol of the optional embodiment of the present application are independent service components as a whole, which are deployed in a lightweight container service, and can share the central processing unit (Central Processing Unit, abbreviated as CPU), memory, and hard disk without the disk array, and can also ensure mutual process space independence.

[0087] Figure 5 is a schematic diagram of a symmetrical distributed NAS cluster according to an optional embodiment of the present application, such as Figure 5 As shown,

[0088] (1) The disk array controller of the optional embodiment of the present application is installed with a Pod manager (Podman) container support software package, which can manage container images and control the start and stop of containers.

[0089] (2) Create a container image, and place the file metadata service (i.e., metadata service component) and the file sharing protocol service component in the container image; when you choose to start the container service, check whether the container image is loaded. If the image is not loaded, first import the container image through the Pod manager import command (podman import); then start the container through the Pod manager run command (podman run), and specify the CPU, memory, hard disk directory, shared memory and other resources used by the container.

[0090] An optional embodiment of the present application utilizes the block interface provided by the existing disk array, and combines the characteristics of symmetrical high reliability and cascade expansion of the disk array, and provides a high-performance file interface by adding file metadata components and file sharing protocol services to the corresponding controller; file metadata and file sharing protocols are independent service components as a whole, deployed in a lightweight container service, which can not only share the CPU, memory, and hard disk with the disk array, but also ensure the independence of each other's process space.

[0091] (3) Figure 6 is a schematic diagram of an MDS cluster and an MDS map according to an optional embodiment of the present application, such as Figure 6As shown in the figure, after the file system is created, the MDS (metadata service) in each controller node container together forms an MDS cluster. The MDS status of the current controller is reported to the current disk array node. According to the disk array distributed consistency protocol, the disk array broadcasts the running status map of each controller MDS to the containers of all nodes, that is, the container of each controller node knows the status of the MDS that needs to be started on this node and the MDS running on other nodes.

[0092] Based on the above definition of the optional embodiments of the present application:

[0093] First, deploy a lightweight container service in each controller of the disk array.

[0094] The container service can be started by the end user based on the specific product usage scenario. If the container service is not started, only the block interface is provided, and all controller CPU, memory, and hard disk resources are only used for block services.

[0095] When the container service is started, both block and file interfaces can be provided. The controller CPU, memory, and hard disk resources can be allocated to different services according to business scenarios. After the container service is started, the health status of the container is managed by the disk array. That is, when the block service on the controller is normal, the container service will continue to be provided. When the block service on the controller is abnormal, the container does not need to continue to run.

[0096] Second, symmetrical file metadata and file sharing protocol service components are deployed in containers within each controller.

[0097] The file metadata service component and the file sharing protocol service component run as a whole program. In this way, the same cache (i.e., shared memory) can be used between the metadata service and the sharing protocol service, which improves the cache utilization, reduces the copying of cache between services, and improves the performance of data access. The file sharing protocol service component on each controller interacts only with the metadata service component on the controller as much as possible, and operates the disk space managed by the controller first.

[0098] Third, the file metadata and file sharing protocol service components between different controllers form a cluster.

[0099] The metadata service components and file sharing protocol service components between different controllers reuse the interconnection links between disk array controllers to achieve the same fault domain of the file service cluster and block service cluster between controllers. The distributed consistency protocol based on message passing of the disk array ensures data consistency. Similarly, the file service cluster also ensures the consistency of service start and stop in each node container by sending service map messages through the same distributed consistency protocol on the disk array, that is, a service in the file service cluster is only started on a certain controller at the same time.

[0100] Fourth, the metadata service and file sharing protocol service within the container can be dynamically started and exited.

[0101] By default, the file metadata service and file sharing service in the container only operate the disk space that is managed by this controller first, avoiding data forwarding between controllers. However, when a controller fails, the disk space it manages needs to be taken over by other controllers (i.e., the second controller). Similarly, the file metadata service and file sharing protocol service in the container on it also need to be taken over by the containers on other controllers. Therefore, the file metadata service component and file sharing protocol service component in the container need to be dynamically started in the container. Conversely, when the failed controller recovers, in order to ensure the pressure balance of each controller, the file metadata service and file sharing protocol service that have been taken over need to be exited and started on the corresponding controller.

[0102] Finally, the in-container file metadata service and file sharing protocol service operate on the disk space managed by the block service through shared memory.

[0103] Although the file metadata service and file sharing protocol service run in the container, a portion of memory space is applied in the block service as shared memory for high performance of data access. The file data read and written by the user host first reaches the file sharing protocol service and file metadata service through the file interface, and then the file sharing protocol service notifies the block service to write the data to the disk. At this time, the block service can use the shared memory address space to directly write the data in the memory to the disk, avoiding the memory copy from the file service to the block service. Due to the use of shared memory, file data and block data can be compared and deleted in the same process space, reducing the amount of data actually written to the disk; at the same time, adding version numbers to the data can achieve unified snapshot management.

[0104] That is to say, the container image on each controller of the optional embodiment of the present application is symmetrical, the file metadata and file sharing protocol service components in the container run as a whole, and the file sharing protocol service components in each controller container interact with the current file metadata service component as much as possible. The file metadata and file sharing protocol service components in the container only operate the disk space managed by the current controller, reducing data forwarding between controllers. The file metadata and file sharing protocol services in the container operate the disk space managed by the block service through shared memory. The disk array cluster manages the MDS status map to ensure the consistency of file service management of each controller.

[0105] In summary, the optional embodiments of the present application share memory between file services and block services, shorten the data access path between block services and file services, and can give full play to the high performance of disk arrays to achieve high performance of file services; through containers and unified cluster management, the relative independence of file services and block services is maintained, and cluster consistency management is also facilitated, and there will be no situation where the block service cluster and the file service cluster are out of sync.

[0106] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of commands to enable a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0107] In this embodiment, a data access device is also provided, which is used to implement the above embodiments and preferred implementations, and will not be repeated for those that have been described. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0108] Figure 7 is a structural block diagram of a data access device according to an embodiment of the present application, such as Figure 7 As shown, the device comprises:

[0109] A startup module 72, used to start a container service in each controller, and add a metadata service component and a file sharing protocol service component in a container image managed by each container service, wherein each controller is a controller in a disk array;

[0110] The access module 74 is used to access the target file data through the first container service in the first controller and the first block service in the first controller when receiving a data access request sent by the target device, wherein the first container service and the first block service exchange data through the first shared memory in the first controller, and the data access request is a request for accessing the target file data corresponding to the first disk space managed by the first controller.

[0111] Through the data access device of the present application, a container service is started in each controller in the disk array, and a metadata service component and a file sharing protocol service component are added to the container image managed by each container service; when a data access request for indicating access to the target file data corresponding to the first disk space managed by the first controller is received from the target device, the target file data is accessed through the first container service in the first controller and the first block service in the first controller, wherein the first container service and the first block service perform data interaction through the first shared memory in the first controller. That is to say, firstly, a metadata service component and a file sharing protocol service component are added to the container service of each controller in the embodiment of the present application, and the metadata service component and the file sharing protocol service component operate the disk space managed by the block service in a shared memory manner, shortening the data access path between the block service and the file service. In addition, by adding the metadata service component and the file sharing protocol service component to the corresponding controller, a high-performance file interface is provided, thereby giving play to the high performance of the disk array and providing high-performance file access. Secondly, the embodiment of the present application adds the metadata service component and the file sharing protocol service component to each container service, maintaining the relative independence of the file service and the block service. Through the embodiments of the present application, the problem that the block service and file service integrated products in the related art cannot guarantee the independence of the block service and file service process spaces while providing high-performance file access can be solved. While providing high-performance file access, the independence of the block service and file service process spaces can be guaranteed.

[0112] In an exemplary embodiment, the access module 74 is further used to determine whether the first file data exists in the first shared memory when it is determined that the data access request is used to indicate reading the first file data in the first disk space, wherein the target file data includes: the first file data; when it is determined that the first file data does not exist in the first shared memory, query the first metadata information corresponding to the first file data in the first metadata service component in the first container service, wherein the first metadata information includes at least: the first address of the first file data; send the first metadata information to the first block service through the first file sharing protocol service component in the first container service, so that the first block service reads the first file data in the first disk space according to the first metadata information, and loads the first file data into the first shared memory; obtain the first file data in the first shared memory through the first file sharing protocol service component, and send the first file data to the target device, so that the target device reads the first file data.

[0113] In an exemplary embodiment, the access module 74 is also used to create second metadata information of the second file data in the first metadata service component in the first container service when it is determined that the data access request is used to indicate that the second file data is to be written to the first disk space, wherein the second metadata information at least includes: a second address of the second file data, and the target file data includes: the second file data; the second file data is written to the first shared memory through the first file sharing protocol service component in the first container service and the second metadata information, so that the first block service writes the second file data in the first shared memory to the second address of the first disk space.

[0114] In an exemplary embodiment, the access module 74 is also used to divide the second file data into multiple target data blocks; perform data block comparison on the multiple target data blocks and the multiple first data blocks in the first disk space, and determine the comparison result; when the comparison result indicates that the multiple first data blocks include the multiple target data blocks, prohibit the second file data from being written to the first disk space; when the comparison result indicates that the multiple first data blocks include the second data block among the multiple target data blocks, write the other data blocks among the multiple target data blocks except the second data block to the first disk space, and prohibit the second data block from being written to the first disk space; when the comparison result indicates that any data block of the multiple target data blocks is not within the multiple first data blocks, determine that the second file data is allowed to be written to the first disk space.

[0115] In an exemplary embodiment, the access module 74 is also used to determine whether the first controller fails; when it is determined that the first controller fails, determine a second controller that manages the first disk space in the disk array, and access the target file data through a second container service in the second controller and a second block service in the second controller, wherein the second container service and the second block service exchange data through a second shared memory in the second controller; when it is determined that the first controller does not fail, access the target file data through the first container service and the first block service.

[0116] In an exemplary embodiment, the access module 74 is also used to build a metadata service component cluster corresponding to the disk array according to each metadata service component in each controller; broadcast the metadata service component map corresponding to each metadata service component to the container service corresponding to each controller according to the metadata service component cluster, wherein the metadata service component map is used to indicate the running status of each metadata service component; when it is determined that the first controller fails, update the first metadata service component map corresponding to the first metadata service component, and broadcast the first metadata service component map to the container service corresponding to each controller, wherein the first metadata service component map is used to indicate that the running status of the first metadata service component is a faulty state, and the first controller includes: the first metadata service component; remap the target disk space in the disk array according to the updated first metadata service component map to map the target metadata information of the data in the first disk space managed by the first metadata service component to the healthy metadata service component in the metadata service component cluster, wherein the healthy metadata service component is a metadata service component whose running status is indicated as normal by the metadata service component map; determine the controller corresponding to the healthy metadata service component as the second controller.

[0117] In an exemplary embodiment, the startup module 72 is further used to establish a service map corresponding to the disk array through the interconnection links between each controller in the disk array and other controllers, wherein the service map is used to indicate a first operating state of each container service and a second operating state of each block service in each controller; when it is determined according to the service map that any container service and / or any block service in any controller in the disk array fails, the any container service and / or any block service is repaired through a distributed consistency protocol, wherein the fault repair strategy corresponding to each container service in each controller in the disk array is the same, and the fault repair strategy for each block service in each controller in the disk array is the same.

[0118] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps in the above method embodiment when running.

[0119] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0120] Optionally, in this embodiment, the storage medium may be configured to store program codes for executing the following steps:

[0121] S1, starting a container service in each controller, and adding a metadata service component and a file sharing protocol service component in a container image managed by each container service, wherein each controller is a controller in a disk array;

[0122] S2. Upon receiving a data access request sent by a target device, access target file data through a first container service in a first controller and a first block service in the first controller, wherein the first container service and the first block service interact with each other through a first shared memory in the first controller, and the data access request is a request for accessing target file data corresponding to a first disk space managed by the first controller.

[0123] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in the above method embodiment.

[0124] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0125] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0126] S1, starting a container service in each controller, and adding a metadata service component and a file sharing protocol service component in a container image managed by each container service, wherein each controller is a controller in a disk array;

[0127] S2. Upon receiving a data access request sent by a target device, access target file data through a first container service in a first controller and a first block service in the first controller, wherein the first container service and the first block service interact with each other through a first shared memory in the first controller, and the data access request is a request for accessing target file data corresponding to a first disk space managed by the first controller.

[0128] An embodiment of the present application further provides a computer program product, including a computer program, which is implemented to perform the steps in the above method embodiment when executed by a processor.

[0129] Optionally, in this embodiment, the computer program product may be executed by a processor to perform the following steps:

[0130] S1, starting a container service in each controller, and adding a metadata service component and a file sharing protocol service component in a container image managed by each container service, wherein each controller is a controller in a disk array;

[0131] S2. Upon receiving a data access request sent by a target device, access target file data through a first container service in a first controller and a first block service in the first controller, wherein the first container service and the first block service interact with each other through a first shared memory in the first controller, and the data access request is a request for accessing target file data corresponding to a first disk space managed by the first controller.

[0132] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0133] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the present application is not limited to any specific combination of hardware and software.

[0134] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for accessing data, characterized in that: include: A container service is started in each controller, and a metadata service component and a file sharing protocol service component are added to the container image managed by each container service, wherein each controller is a controller in a disk array; upon receiving a data access request sent by a target device, the target file data is accessed through a first container service in a first controller and a first block service in the first controller, wherein the first container service and the first block service interact with each other through a first shared memory in the first controller, and the data access request is a request for accessing the target file data corresponding to a first disk space managed by the first controller.

2. The method according to claim 1, characterized in that Accessing target file data through a first container service in a first controller and a first block service in the first controller includes: In the case where it is determined that the data access request is used to instruct to read the first file data in the first disk space, determining whether the first file data exists in the first shared memory, wherein the target file data includes: the first file data; When it is determined that the first file data does not exist in the first shared memory, querying first metadata information corresponding to the first file data in a first metadata service component in the first container service, wherein the first metadata information at least includes: a first address of the first file data; sending the first metadata information to the first block service through a first file sharing protocol service component in the first container service, so that the first block service reads the first file data in the first disk space according to the first metadata information and loads the first file data into the first shared memory; The first file data is obtained in the first shared memory through the first file sharing protocol service component, and the first file data is sent to the target device, so that the target device reads the first file data.

3. The method according to claim 1, characterized in that Accessing target file data through a first container service in a first controller and a first block service in the first controller includes: In a case where it is determined that the data access request is used to instruct to write the second file data into the first disk space, second metadata information of the second file data is created in a first metadata service component in the first container service, wherein the second metadata information at least includes: a second address of the second file data, and the target file data includes: the second file data; The second file data is written into the first shared memory through the first file sharing protocol service component in the first container service and the second metadata information, so that the first block service writes the second file data in the first shared memory to the second address of the first disk space.

4. The method according to claim 3, characterized in that Before creating second metadata information of the second file data in the first metadata service component in the first container service, the method further includes: dividing the second file data into a plurality of target data blocks; Performing data block comparison on the plurality of target data blocks and the plurality of first data blocks in the first disk space, and determining a comparison result; If the comparison result indicates that the plurality of first data blocks include the plurality of target data blocks, prohibiting the second file data from being written to the first disk space; When the comparison result indicates that the plurality of first data blocks include the second data block among the plurality of target data blocks, write the other data blocks among the plurality of target data blocks except the second data block to the first disk space, and prohibit writing the second data block to the first disk space; When the comparison result indicates that any data block of the plurality of target data blocks is not within the plurality of first data blocks, it is determined that the second file data is allowed to be written to the first disk space.

5. The method according to claim 1, characterized in that Accessing target file data through a first container service in a first controller and a first block service in the first controller includes: determining whether the first controller fails; In the case where it is determined that the first controller fails, determining a second controller that manages the first disk space in the disk array, and accessing the target file data through a second container service in the second controller and a second block service in the second controller, wherein the second container service and the second block service perform data exchange through a second shared memory in the second controller; When it is determined that the first controller has not failed, the target file data is accessed through the first container service and the first block service.

6. The method according to claim 5, characterized in that Determining a second controller in the disk array that manages the first disk space includes: Building a metadata service component cluster corresponding to the disk array according to each metadata service component in each controller; Broadcasting the metadata service component map corresponding to each metadata service component to the container service corresponding to each controller according to the metadata service component cluster, wherein the metadata service component map is used to indicate the running status of each metadata service component; In the case of determining that the first controller fails, updating a first metadata service component map corresponding to the first metadata service component, and broadcasting the first metadata service component map to the container service corresponding to each controller, wherein the first metadata service component map is used to indicate that the operating state of the first metadata service component is a failure state, and the first controller includes: the first metadata service component; Remap the target disk space in the disk array according to the updated first metadata service component map, so as to map the target metadata information of the data in the first disk space managed by the first metadata service component to the healthy metadata service component in the metadata service component cluster, wherein the healthy metadata service component is a metadata service component whose running status is indicated to be normal by the metadata service component map; The controller corresponding to the health metadata service component is determined as the second controller.

7. The method according to claim 1, characterized in that After adding the metadata service component and the file sharing protocol service component to each container image managed by the container service, the method further includes: Establishing a service map corresponding to the disk array through interconnection links between each controller in the disk array and other controllers, wherein the service map is used to indicate a first operating state of each container service and a second operating state of each block service in each controller; When it is determined according to the service map that any container service and / or any block service in any controller in the disk array fails, the fault is repaired for the any container service and / or any block service through a distributed consistency protocol, wherein the fault repair strategy corresponding to each container service in each controller in the disk array is the same, and the fault repair strategy for each block service in each controller in the disk array is the same.

8. A data access device, characterized in that: include: A startup module, used to start a container service in each controller, and add a metadata service component and a file sharing protocol service component in the container image managed by each container service, wherein each controller is a controller in a disk array; an access module, used to access target file data through a first container service in a first controller and a first block service in the first controller when a data access request sent by a target device is received, wherein the first container service and the first block service exchange data through a first shared memory in the first controller, and the data access request is a request to access the target file data corresponding to a first disk space managed by the first controller.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method according to any one of claims 1 to 7 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images