Method, device and readable storage medium for expanding capacity by combining multiple different or identical file systems

Abstract

The present invention proposes a capacity expansion method, device and readable storage medium for combining multiple different or identical file systems, including: communicating with the operating system through the FUSE protocol, and forwarding the IO request to the user state for processing and returning through the kernel state FUSE module. User state file system related operations, such as mounting through the mount command, storing files by taking the remainder of the file path hash and the original storage total to determine the location and store it in a database (such as SQLite), first taking the remainder of the hash to determine the location when accessing, and obtaining and updating from the database when changing. The original storage permissions are used for obtaining and modifying permissions, and the capacity is the sum of the original storage. Also disclosed are a capacity expansion device, an electronic device using the method and a readable storage medium. The problem of insufficient storage system capacity and difficulty in integration is solved, the utilization rate is improved, and the cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer storage technology, and in particular to a capacity expansion method, a device and a readable storage medium for combining a plurality of different or identical file systems. Background Art

[0002] With the continuous development of hospital pathology scanning business, the amount of data is showing a rapid growth trend. In the hospital storage environment, there are various storage systems, including storage based on SMB protocol, storage based on FTP protocol, and hard disk storage. However, the capacity provided by these single storage systems is gradually unable to meet the growing storage needs.

[0003] On the one hand, the hospital's storage systems are complex and diverse, and different storage systems may be based on different protocols and technical architectures, which makes it difficult to integrate and efficiently use storage resources. On the other hand, how to fully tap and utilize the remaining capacity space in each storage system without changing the existing application service usage has become an urgent problem to be solved.

[0004] This situation not only leads to a waste of storage resources and increases the storage costs of hospitals, but also may affect the storage and management efficiency of pathological data, which in turn has an adverse impact on the hospital's pathological diagnosis and other businesses. Therefore, a method that can effectively combine multiple different or identical file systems for expansion is needed to solve these problems faced by hospital storage systems. Summary of the invention

[0005] The embodiments of the present invention provide a capacity expansion method, device and readable storage medium for combining multiple different or identical file systems, which aims to solve the problems that the existing single storage systems of multiple different protocols have insufficient capacity and are difficult to integrate and utilize, resulting in waste of storage resources and increased costs, and affecting the efficiency of pathology data management.

[0006] The core technology of the present invention is mainly to realize the fuse protocol communication of the operating system, and manage different original storages in combination with the user-mode file system, including the hash remainder algorithm for file storage reading and the permission capacity acquisition method, so as to realize the expansion of combining multiple different or the same file systems.

[0007] In a first aspect, the present invention provides a capacity expansion method for combining multiple different or identical file systems, the method comprising the following steps:

[0008] Step 1: Implement FUSE protocol communication with the operating system:

[0009] When performing a read / write operation, the IO request first enters the kernel state and is then passed to the kernel state FUSE file system module. The FUSE file system module then forwards the request to the user state file system for processing according to the situation. After the processing is completed, the response is returned along the original path;

[0010] Step 2: Implement user-mode file system related operations:

[0011] Mount the user-mode file system to a folder;

[0012] When storing a file, the storage location is determined by performing a hash operation on the file path and taking the remainder of the original storage total number, and the file path and storage location information are stored in the database;

[0013] When accessing a file, first perform a hash modulus to determine the original storage location. If the total number of original storage changes, obtain the actual storage location from the database, return the user's request first, then move the file and update the database;

[0014] The permissions for obtaining and modifying are directly based on the original storage permissions, and the capacity is obtained by obtaining each original storage and then adding them up.

[0015] Furthermore, in step 2, the user-mode file system is mounted to the folder through the mount command.

[0016] Furthermore, in step 2, the database is SQLite.

[0017] Furthermore, in step 2, the remainder of the hash is hash (file path) / the total number of original storages.

[0018] Furthermore, in step 2, Ftp is mounted on the system ftpData directory, smb is mounted on the system smbData directory, and the hard disk is mounted on the system diskData directory.

[0019] In a second aspect, the present invention provides a device for expanding capacity by combining multiple different or identical file systems, comprising:

[0020] The FUSE protocol communication module is used to implement FUSE protocol communication with the operating system. When performing read / write operations, the IO request first enters the kernel state and is then passed to the kernel state FUSE file system module. The FUSE file system module then forwards the request to the user state file system for processing according to the situation. After the processing is completed, the response is returned along the original path.

[0021] The user-mode file system related operation module is used to implement user-mode file system related operations; mount the user-mode file system to a folder; when storing files, determine the storage location based on the hash of the file path and the modulus of the original storage total, and store the file path and storage location information in the database; when accessing files, first perform a hash modulus to determine the original storage location, if the original storage total changes, obtain the actual storage location from the database, return the user's needs first, then move the file and update the database; obtain and modify permissions directly using the original storage permissions, and the capacity is obtained by obtaining each original storage and then adding them up.

[0022] In a third aspect, the present invention provides an electronic device including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the above-mentioned expansion method combining multiple different or identical file systems.

[0023] In a fourth aspect, the present invention provides a readable storage medium, in which a computer program is stored. The computer program includes a program code for controlling a process to execute the process. The process includes a method for expanding the capacity of a plurality of different or identical file systems according to the above combination.

[0024] The main contributions and innovations of the present invention are as follows:

[0025] 1. Implement FUSE protocol communication with the operating system: Enable IO requests to be efficiently transmitted between kernel-mode and user-mode file systems. When performing read / write operations, IO requests first enter kernel-mode and are then passed to the kernel-mode FUSE file system module. The FUSE file system module then forwards the request to the user-mode file system for processing based on the situation. After processing, the response is returned along the original path, thereby improving the processing efficiency of the file system.

[0026] 2. The present invention solves some drawbacks in the traditional medical information construction model through innovative methods, such as limited storage capacity, poor scalability, and difficult operation and maintenance, improves the utilization rate and management efficiency of storage resources, reduces costs, and meets the needs of large capacity, high performance, easy scalability, and easy management of file systems in scenarios such as hospitals. At the same time, the invention can also be applied to other scenarios that require the integration of multiple file systems, and has a wide range of applicability.

[0027] The details of one or more embodiments of the invention are set forth in the following drawings and description so that other features, objects, and advantages of the invention are more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0029] Figure 1 is a general architecture diagram of performing FUSE protocol communication with an operating system according to an embodiment of the present invention;

[0030] Figure 2 It is a custom user-mode file system according to an embodiment of the present invention that combines various original storages;

[0031] Figure 3 is a schematic diagram of an implementation method of storage according to an embodiment of the present invention;

[0032] Figure 4 4 is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0034] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0035] Embodiment 1

[0036] The present invention aims to propose a capacity expansion method combining multiple different or identical file systems, specifically, referring to Figure 1 and Figure 2 The steps of applying the method in the hospital pathology data storage system are as follows:

[0037] Step 1: System Setup

[0038] In the hospital's pathology data storage environment, there is a storage server based on the SMB protocol for storing some pathology report documents. It is configured with a certain amount of storage space and connected to various terminals in the hospital through the network, which is convenient for doctors and related staff to access. The storage device based on the FTP protocol is used to store pathology image data. The device has high-speed data transmission capabilities to meet the storage and reading needs of a large number of pathology images. In addition, there are several hard disk storages for backing up important data. These hard disks are stored in the local server to ensure data security.

[0039] According to the method of the present invention, the FUSE protocol communication between the operating system and the user-mode file system is first implemented. When the system is started, the relevant kernel modules and user-mode programs are loaded. The FUSE file system module in the kernel is responsible for handling the protocol conversion and request forwarding related to the kernel mode. The user-mode file system is responsible for the specific file operation logic. After the system is started, the FUSE protocol communication link is established to ensure that the IO request can be correctly transmitted between different states.

[0040] For example Figure 1 In the process of executing read / write, the IO request is advanced into the kernel and passed to the kernel FUSE file system module. The kernel FUSE module sends the request to the user state, which is handled by the user state file system ( Figure 1 The A system fs) program receives and processes it. After the processing is completed, the response is returned by the original path. The kernel FUSE module performs protocol packaging and protocol parsing in the kernel state. It accepts the next request and forwards it to the user state according to the FUSE protocol, then receives the response from the user state and replies to the user. For example, the kernel FUSE is used to accept the io request from the operating system, and then encapsulates it into a FUSE data packet and forwards it to the user state. At this time, the user-state file system receives the FUSE data packet. If it wants to understand this data packet, it must implement the FUSE protocol code. This is the first step of this system. After implementation, calling the system standard IO call is like calling the user-state file system ( Figure 1 In A system fs), for example, calling File.Read(path) in other systems is like calling the Read(stringpath) function in the user-mode file system.

[0041] Among them, FUSE (Filesystem in Userspace) is a mechanism for implementing a file system in user space. It allows developers to create custom file systems in user space without making extensive modifications to the kernel. By establishing a communication mechanism between user space and kernel space, the file system in user space can interact with the kernel.

[0042] IO request (Input / Output Request) is a request for data transfer between a process or application and an external device (such as a storage device, network device, etc.) in a computer system. It is classified as follows:

[0043] 1. Read Request

[0044] When an application needs to obtain data from a storage device (such as a hard disk, memory, etc.), a read request is issued. For example, in the present invention, when an application service needs to access a file, a read request is issued, and the request will eventually obtain data from the original storage and return it according to a certain process (first passing through the kernel FUSE module, etc.).

[0045] 2. Write Request

[0046] When an application needs to store data in a storage device, a write request is issued. For example, in the present invention, when storing a file, the application issues a write request, determines the storage location according to a related algorithm (such as a hash modulus algorithm), and stores the file in the corresponding original storage (each real storage).

[0047] Step 2: File storage process

[0048] When a doctor uploads a new pathology report document, the user-mode file system receives a storage request. Assuming that there are currently three original storages (smb storage, ftp storage, and hard disk storage), a hash calculation is performed based on the file path. For example, the file path is " / path / to / report.docx". First, a specific hash algorithm is performed on the path string. This hash algorithm can be a common hash function, such as a simplified version of the SHA-256 algorithm, which converts the file path into a fixed-length hash value. Then divide this hash value by the total number of original storages (3) and take the remainder. Assuming the remainder result is 1, and the agreed remainder 0 corresponds to smb storage, 1 corresponds to ftp storage, and 2 corresponds to hard disk storage, then the file will be stored in the ftp storage.

[0049] When storing the file, the file path " / path / to / report.docx" and the storage location information (ftp storage) will be stored in the SQLite database. The SQLite database runs on the local server and has a special table for storing the mapping relationship between the file path and the storage location. When the storage operation is completed, a new record will be added to the database to record the relevant information of the file.

[0050] Among them, SQLite is a lightweight database management system, which exists in the form of files and does not require an independent database server process. In the present invention, it is used to store information such as file paths and corresponding original storage locations.

[0051] Step 3: File access process

[0052] When the doctor needs to view the pathology report document, the user-state file system first performs a hash modulus algorithm on the file path. If the original storage total has not changed, the previous algorithm determines that the file is in the FTP storage. At this time, the user-state file system sends a read request to the FTP storage, and the FTP storage searches for the file in its own storage area according to the file path and returns the file data to the user-state file system, which then returns the file to the doctor.

[0053] If the total number of original storage becomes 4 because the hospital adds a new storage device, the result of the hash modulus algorithm may change. At this time, the user-state file system first obtains the actual storage location of the file from the SQLite database (still ftp storage). In the process of obtaining the actual location of the file, the user-state file system will perform a query operation in the SQLite database to find the corresponding storage location record according to the file path. After finding the record, the file is returned to the doctor first, and then the file is moved to the new location specified by the hash modulus algorithm (if there is a change) and the information in the SQLite database is updated. In the process of moving the file, the user-state file system will read the file data from the ftp storage, and then calculate the result based on the new storage location, write the file to the corresponding original storage, and update the record in the database to ensure that the information in the database is consistent with the actual storage location of the file.

[0054] Step 4: Permission and capacity management

[0055] When the doctor needs to modify the permissions of the pathology report document, for example, to set it to read-only permissions, the user-state file system directly sends the permission modification request to the ftp storage (original storage), because the file is stored in the ftp storage and uses the original storage permissions. In the process of modifying permissions, the user-state file system will send permission modification instructions to the ftp storage based on the file storage location information. After receiving the instructions, the ftp storage will modify the file permissions in its own permission management module, which can simplify the permission management process.

[0056] For capacity management, the system can obtain SMB storage in real time and has a detailed acquisition process, such as obtaining the available capacity of SMB storage through relevant commands in the SMB protocol, obtaining the available capacity of FTP storage through specific instructions in the FTP protocol, and obtaining the available capacity of hard disk storage through the disk management tool of the local server, and adding them together to obtain the total available capacity, which is displayed to the system administrator so that the administrator can understand the usage of storage resources. The system administrator can reasonably arrange new data storage tasks based on the total available capacity and the usage of each storage to avoid overloading of a certain storage device.

[0057] like Figure 3 As shown, if there is a folder A in the ftp mounted folder containing (A1.txt, A2.txt, A3.txt), there is a folder B in smb containing (B1.txt, B2.txt, B3.txt), and there is also a folder A in the folder mounted on the hard disk containing (A4.txt, A5.txt, A6.txt), then there are only two folders A and B in the user-mode file system mounted folder, and folder A merges the files in smb and hard disk. This is also the final appearance seen by the application, and the capacity is the sum of each original storage.

[0058] Embodiment 2

[0059] Based on the same concept, the present invention also proposes a file storage integration application in an enterprise data center, including:

[0060] Step 1: System Setup

[0061] Enterprise data centers have a variety of storage systems, including storage based on the Network File System (NFS) protocol for sharing office documents, storage based on the iSCSI protocol for database backup, and some local hard disk storage for temporary file storage.

[0062] Similarly, FUSE protocol communication between the operating system and the user-mode file system is implemented.

[0063] Step 2: File storage process

[0064] When an employee uploads a new office document, assuming there are currently three original storages (NFS storage, iSCSI storage, and hard disk storage), the file path is hashed and then modulo 3 to determine the storage location. For example, if the file path is " / company / docs / project.docx", the modulo result is 2, which corresponds to hard disk storage, then the file will be stored in the hard disk storage, and the file path and storage location information will be stored in the SQLite database.

[0065] Step 3: File access process

[0066] When an employee needs to access the office document, if the original storage total has not changed, the hash remainder algorithm is used to determine that the document is in the hard disk storage, and the document is obtained directly from the hard disk storage. If the enterprise adds a new storage device, the original storage total changes, and the actual storage location is obtained from the SQLite database to meet the employee's needs first, then move the file (if necessary) and update the database.

[0067] Step 4: Permission and capacity management

[0068] For permission management, when an employee needs to modify the permissions of an office document, such as setting it to editable permissions, the user-mode file system uses the original storage permissions to make modifications based on the file storage location (hard disk storage).

[0069] For capacity management, the system obtains the capacity of NFS storage, iSCSI storage, and hard disk storage, adds them together, and provides them to the system administrator to reasonably plan storage resources.

[0070] Embodiment 3

[0071] This embodiment also provides an electronic device, referring to Figure 4 , comprises a memory 404 and a processor 402, wherein the memory 404 stores a computer program, and the processor 402 is configured to run the computer program to execute the steps in any of the above method embodiments.

[0072] Specifically, the processor 402 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiment of the present invention.

[0073] Among them, the memory 404 may include a large capacity memory 404 for data or instructions. For example, but not limitation, the memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. In appropriate cases, the memory 404 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 404 may be inside or outside the data processing device. In a specific embodiment, the memory 404 is a non-volatile memory. In a specific embodiment, the memory 404 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable Programmable Read-Only Memory, EPROM for short), an electrically erasable PROM (Electrically Erasable Programmable Read-Only Memory, EEPROM for short), an electrically alterable ROM (Electrically Alterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. In appropriate circumstances, the RAM may be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM may be a fast page mode dynamic random access memory 404 (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0074] The memory 404 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 402 .

[0075] The processor 402 reads and executes computer program instructions stored in the memory 404 to implement any one of the capacity expansion methods for combining multiple different or identical file systems in the above embodiments.

[0076] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408 , wherein the transmission device 406 is connected to the processor 402 , and the input / output device 408 is connected to the processor 402 .

[0077] The transmission device 406 can be used to receive or send data via a network. Specific examples of the above-mentioned network may include a wired or wireless network provided by a communication provider of the electronic device. In one example, the transmission device includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 406 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0078] Input / output devices 408 are used to input or output information.

[0079] Embodiment 4

[0080] This embodiment also provides a readable storage medium, in which a computer program is stored. The computer program includes a program code for controlling a process to execute the process. The process includes a capacity expansion method for combining multiple different or identical file systems according to the first embodiment.

[0081] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0082] In general, various embodiments may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects of the invention may be implemented in hardware, while other aspects may be implemented in firmware or software executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flow charts, or using some other graphical representation, it should be understood that, as non-limiting examples, the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0083] Embodiments of the present invention can be implemented by computer software, which is executable by a data processor of a mobile device, such as in a processor entity, or implemented by hardware, or implemented by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets and / or macros can be stored in any device readable data storage medium, and they include program instructions for performing specific tasks. Computer program products can include one or more computer executable components configured to perform embodiments when the program is running. One or more computer executable components can be at least one software code or a part thereof. In addition, at this point, it should be noted that any box of the logic flow can represent a program step, or interconnected logic circuits, boxes and functions, or a combination of program steps and logic circuits, boxes and functions. Software can be stored in physical media such as memory chips or storage blocks implemented in processors, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and data variants thereof, CDs. Physical media are non-transient media.

[0084] Those skilled in the art should understand that the technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0085] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A capacity expansion method combining multiple different or identical file systems, characterized in that: The following steps are involved: Step 1: Implement FUSE protocol communication with the operating system: When performing a read / write operation, the IO request first enters the kernel state and is then passed to the kernel state FUSE file system module. The FUSE file system module then forwards the request to the user state file system for processing according to the situation. After the processing is completed, the response is returned along the original path; Step 2: Implement user-mode file system related operations: Mount the user-mode file system to a folder using the mount command; mount Ftp to the system ftpData directory, mount smb to the system smbData directory, and mount the hard disk to the system diskData directory; When storing a file, the storage location is determined by performing a hash operation on the file path and taking the remainder of the original storage total number, and the file path and storage location information are stored in the database; When accessing a file, first perform a hash modulus to determine the original storage location. If the total number of original storage changes, obtain the actual storage location from the database, return the user's request first, then move the file and update the database; The permissions for obtaining and modifying are directly based on the original storage permissions, and the capacity is obtained by obtaining each original storage and then adding them up.

2. A capacity expansion method combining multiple different or identical file systems as claimed in claim 1, characterized in that: In step 2, the database is SQLite.

3. A capacity expansion method combining multiple different or identical file systems as claimed in claim 1, characterized in that: In step 2, the remainder of the hash is: The path string of the file path is processed by hash algorithm and converted into a hash value of fixed length; Divide the fixed-length hash value by the total number of original storage and take the remainder.

4. A device for expanding capacity by combining multiple different or identical file systems, characterized in that: include: FUSE protocol communication module, used to implement FUSE protocol communication with the operating system; When performing a read / write operation, the IO request first enters the kernel state and is then passed to the kernel state FUSE file system module. The FUSE file system module then forwards the request to the user state file system for processing according to the situation. After the processing is completed, the response is returned along the original path; The user-mode file system operation module is used to implement user-mode file system operations. The user-mode file system is mounted on a folder through the mount command. When storing files, the storage location is determined based on the hash of the file path and the remainder of the original storage total, and the file path and storage location information are stored in the database. When accessing files, the hash remainder is first performed to determine the original storage location. If the original storage total changes, the actual storage location is obtained from the database, and the user's needs are returned first, and then the file is moved and the database is updated. The original storage permissions are directly used for obtaining and modifying permissions, and the capacity is obtained by obtaining each original storage and then adding them up. Among them, Ftp is mounted to the system ftpData directory, smb is mounted to the system smbData directory, and the hard disk is mounted to the system diskData directory.

5. An electronic device comprising a memory and a processor, characterized in that: The memory stores a computer program, and the processor is configured to run the computer program to execute the capacity expansion method for combining multiple different or identical file systems as described in any one of claims 1 to 3.

6. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which includes a program code for controlling a process to execute a process, and the process includes the capacity expansion method for combining multiple different or identical file systems according to any one of claims 1 to 3.

Images