Computer systems and file storage

Abstract

To enable easy and appropriate file searching across multiple file storage systems. [Solution] In a computer system 1 having a search server 20 and a plurality of file storage 10, the search server 20 receives a search request, sends a file storage search request corresponding to the search request to the file storage 10, receives search results corresponding to the file storage search request from the file storage 10, and returns the search results to the requester. The file storage 10 stores metadata files T3 that store metadata about files in snapshots of the file system managed by the file storage 10, receives a file storage search request from the search server 20, searches for files from the metadata files T3 based on the file storage search request, and returns the search results obtained from the search to the search server 20.

Description

Technical Field

[0001] The present invention relates to data search technology.

Background Art

[0002] In recent years, the use of generative AI (Artificial Intelligence) has attracted attention. In generative AI, additional learning is performed on files / databases. The learning data may exist at multiple sites, and there is a need for a technology to discover learning data distributed across multiple sites.

[0003] Patent Document 1 discloses a technique for aggregating file metadata in snapshots of multiple sites or multiple file storages into one database to enable global data search.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In data search using the technique described in Patent Document 1, since it is necessary to aggregate metadata into one DB, there is a problem that the search time increases as the number of file storages or files increases. Also, since only the metadata of the snapshot is transferred, there is a problem that files created or updated in the file system after the snapshot is acquired cannot be searched.

[0006] The present invention has been made in view of the above circumstances, and an object thereof is to provide a technology capable of easily and appropriately searching for files from among a plurality of file storages.

Means for Solving the Problems

[0007] To achieve the above objective, a computer system relating to one perspective includes a search server and a plurality of file storages that manage a file system, wherein the search server receives a search request for a file to be searched, sends a file storage search request corresponding to the search request to the file storage, receives search results corresponding to the file storage search request from the file storage, and returns the search results to the source of the search request, and the file storage stores a first metadata file that stores metadata about files in snapshots of the file system managed by the file storage, receives a file storage search request from the search server, searches for files from the first metadata file based on the file storage search request, and returns the search results obtained by the search to the search server. [Effects of the Invention]

[0008] According to the present invention, files can be easily and appropriately searched from among multiple file storage devices. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 is a diagram illustrating the outline of Example 1. [Figure 2] Figure 2 is an overall configuration diagram of the computer system according to Example 1. [Figure 3] Figure 3 is a diagram illustrating an example of a file storage system according to Example 1. [Figure 4] Figure 4 is a diagram showing an example of the configuration of a search server according to Embodiment 1. [Figure 5] Figure 5 is a diagram showing an example of a metadata file list related to Example 1. [Figure 6] Figure 6 is a diagram showing an example of a metadata file related to Example 1. [Figure 7]FIG. 7 is a configuration diagram of an example of an operation log list according to Embodiment 1. [Figure 8] FIG. 8 is a configuration diagram of an example of a file storage list according to Embodiment 1. [Figure 9] FIG. 9 is a configuration diagram of an example of a data search result according to Embodiment 1. [Figure 10] FIG. 10 is a flowchart of an example of a search process according to Embodiment 1. [Figure 11] FIG. 11 is a flowchart of an example of a file I / O process according to Embodiment 1. [Figure 12] FIG. 12 is a flowchart of an example of a metadata file update process according to Embodiment 1. [Figure 13] FIG. 13 is a flowchart showing an example of a snapshot acquisition process according to Embodiment 1. [Figure 14] FIG. 14 is a flowchart showing an example of a snapshot deletion process according to Embodiment 1. [Figure 15] FIG. 15 is a diagram for explaining the outline of Embodiment 2. [Figure 16] FIG. 16 is a configuration diagram of an example of a file storage according to Embodiment 2. [Figure 17] FIG. 17 is a flowchart of an example of a search process according to Embodiment 2. [Figure 18] FIG. 18 is a flowchart of an example of a local search process according to Embodiment 2. [Figure 19] FIG. 19 is a configuration diagram of an example of a file storage according to Embodiment 3. [Figure 20] FIG. 20 is a configuration diagram of an example of a search server according to Embodiment 3. [Figure 21] FIG. 21 is a flowchart of an example of a search process according to Embodiment 3. [Figure 22] FIG. 22 is a flowchart of an example of a metadata sending process according to Embodiment 3.

MODE FOR CARRYING OUT THE INVENTION

[0010] For some embodiments, an explanation will be given with reference to the drawings. Note that the embodiments described below do not limit the invention according to the claims, and not all of the elements and combinations thereof described in the embodiments are essential for the solution means of the invention.

[0011] In the following description, the data structure of information may be described as a table, but the information may be represented in any data structure.

[0012] Also, in the following description, the "program" may be used as the main body of processing for explanation. However, the program is executed by a processor to perform defined processing while appropriately using at least one of the storage unit and the interface unit. Therefore, the main body of the processing may be a processor (or a computer or computer system having a processor). The program may be installed in the computer from a program source. The program source may be, for example, a program distribution server or a computer-readable recording medium. Also, in the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs. Also, at least a part of the processing realized by the execution of the program may be realized by a hardware circuit (for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array)).

Embodiment

[0013] In the computer system of Embodiment 1, a search server that provides global data search (that is, data search for file storage at multiple sites) performs data search on the metadata files on each file storage using an interface (single file search request: for example, S3select in Amazon (registered trademark) S3) that searches by SQL for a single file, thereby realizing global data search.

[0014] Figure 1 is a diagram illustrating the outline of Example 1.

[0015] Client 40 requests a global data search from search server 20. Search criteria can include file name, directory name, file size, file metadata such as creation date and modification date, and / or location and file storage. Figure 1 shows an example where client 40 requests a search for a file named "f1".

[0016] The search server 20 receives a search request from the client 40, and based on the received search request, performs a search using S3select against the metadata files of each file system in each file storage 10 (second metadata files) and metadata files corresponding to snapshots of the file systems (first metadata files). The server then receives the search results from each file storage 10, aggregates the search results, and returns them to the client 40.

[0017] In the example shown in Figure 1, the search server 20 uses S3select to perform a search on the file system FS1 of file storage 10 (file storage #1), and on the metadata files T3 of FS1_0714 and FS1_0713, which are snapshots of file system FS1.

[0018] The file storage 10 has an S3 gateway program P3, which can process S3select requests. The file storage 10 also maintains a log list T5 of file I / O operations to the file system (FS1, etc.) and creates a metadata file T3 based on the log list T5. The metadata file T3 stores the metadata of the files in the file system in a format that can be searched by an S3select request. Furthermore, if the file storage 10 has taken a snapshot, it stores the metadata difference between the metadata file T3 of the previous snapshot and the metadata file T3 of subsequent snapshots (incremental snapshots) and the file system from which the snapshot was taken (parent file system) in the metadata file T3.

[0019] In the example in Figure 1, the metadata file T3 of the FS1_0713 snapshot contains metadata for the created files f1 and f2. The snapshot of FS1_0714, which is an incremental snapshot of FS1_0713, contains only the metadata for file f1, which is the difference from the snapshot of FS1_0713. In addition, the metadata file T3 of the file system FS1 contains the metadata for files f1 and f2, which are the differences from the snapshot of FS1_0714.

[0020] Figure 2 is an overall configuration diagram of the computer system according to Example 1.

[0021] Computer system 1 includes one or more clients 40, a search server 20, and multiple file storage devices 10. The clients 40, the search server 20, and the file storage devices 10 are connected via a network 30.

[0022] Network 30 is, for example, a WAN (Wide Area Network) or a LAN (Local Area Network).

[0023] Client 40 is a device used by a user to search for a desired file from multiple file storages 10. It sends search conditions to the search server 20, receives search results from the search server 20, and displays them. Client 40 is composed of a computer such as a PC (Personal Computer).

[0024] File storage 10 is a device that manages a file system that manages data as files (content). File storage 10 may consist of, for example, a single storage device, or it may be a distributed file storage system consisting of multiple storage devices. The storage device may be, for example, a dedicated storage device, or it may be a storage device consisting of, for example, a PC or a general-purpose server.

[0025] The search server 20 receives a search request from the client 40, controls the search process for files managed in multiple file storages 10 based on the search request, and returns the search results to the client 40.

[0026] Figure 3 is a diagram illustrating an example of a file storage system according to Example 1.

[0027] File storage 10 is a file storage system that manages data as files (content). File storage 10 is composed of, for example, a computer and has a CPU (Central Processing Unit) 100 as an example of a processor, memory 110, cache 120, network I / F 130, and storage device 150. The CPU 100, memory 110, cache 120, network I / F 130, and storage device 150 are interconnected via a communication path such as a bus.

[0028] The CPU 100 controls the operation of the file storage by executing a program stored in the memory 110.

[0029] Memory 110 is, for example, RAM (Random Access Memory) and temporarily stores programs and data necessary for controlling the operation of the CPU 100. Memory 110 stores the network file system program P1, the S3 gateway program P3, the local file system program P7, and the metadata file update program P9. Note that the programs stored in memory 110 may also be stored in the storage device 150.

[0030] The network file system program P1, executed on the CPU 100, receives various operation requests such as Read / Write requests to the file system from clients 40 and other devices, and processes the protocols included in these operation requests. For example, the network file system program P1 processes protocols such as Native-Client, FUSE (Filesystem in Userapace), NFS (Network File System), and SMB (Server Message Block).

[0031] The S3 gateway program P3 runs on CPU 100 to receive various S3 API (Application Programming Interface) requests from clients 40, search servers 20, etc., and executes processing according to the received requests. The S3 API also includes S3select.

[0032] The local file system program P7, when executed on CPU 100, provides content storage such as file systems and object storage to the network file system program P1 and the S3 gateway program P3. Additionally, when executed on CPU 100, the local file system program P7 appends details of operations performed on the content storage to the operation log list T5.

[0033] The metadata file update program P9, executed on CPU100, creates or updates metadata file T3 based on the operation log list T5.

[0034] The cache 120 is, for example, RAM, and temporarily stores data written from the client 40 and data read from the memory device 150.

[0035] The network interface 130 is, for example, an interface such as a wired LAN card or a wireless LAN card, and communicates with other devices (for example, a client 40 or a search server 20) via the network 30.

[0036] The storage device 150 is, for example, a hard disk or flash memory, and stores various types of content, including content used by the user of client 40. The storage device 150 stores a metadata file list T1, a metadata file T3, and an operation log list T5. In addition to or instead of the storage device 150, block storage connected to the file storage 10 may be used. The block storage may provide the file storage 10 with block-type storage functionality, such as FC-SAN (Fibre Channel Storage Area Network). Furthermore, the storage device 150 may tier the data between object storage and block storage.

[0037] Figure 4 is a diagram showing an example of the configuration of a search server according to Embodiment 1.

[0038] The search server 20 provides a user interface (UI) for data retrieval to the client 40 and searches for data stored in multiple file storage devices 10 in response to requests from the client 40. The search server 20 is composed of a computer such as a PC or a general-purpose server. The search server 20 has a CPU 200 as an example of a processor, memory 210, cache 220, network interface 230, and storage device 250. The CPU 200, memory 210, cache 220, network interface 230, and storage device 250 are interconnected via a communication path such as a bus.

[0039] The CPU 200 controls the operation of the search server 20 by executing a program stored in the memory 210.

[0040] Memory 210 is, for example, RAM, and temporarily stores programs and data necessary for controlling the operation of the CPU 200. Memory 210 stores the data retrieval UI program P11, the data retrieval program P13, and the data search result T9. Note that the programs stored in memory 210 may also be stored in the storage device 250.

[0041] The data retrieval UI program P11 is executed on the CPU 200 to provide a data retrieval UI to the client 40 and to receive data retrieval requests from the client 40.

[0042] The data retrieval program P13, when executed by the CPU 200, issues S3select commands to each file storage 10 in response to data retrieval requests from the client 40.

[0043] The cache 220 is, for example, RAM, and temporarily stores search results for data retrieval requests from the client 40 and data read from the storage device 250.

[0044] The network interface 230 is, for example, an interface such as a wired LAN card or a wireless LAN card, and communicates with other devices (e.g., client 40, file storage 10) via the network 30.

[0045] The storage device 250 is, for example, a hard disk or flash memory, and stores the operating system of the search server 20, the file storage list T7, and so on.

[0046] Figure 5 shows an example of a metadata file list related to Example 1.

[0047] The metadata file list T1 is a table that stores information about metadata files corresponding to file systems and snapshots of file systems within file storage 10. The metadata file list T1 stores entries corresponding to each metadata file T3 of a file system or snapshot. Each entry in the metadata file list T1 includes fields for the file system name C11, type C13, and snapshot name C15.

[0048] The filesystem name C11 stores the filesystem name corresponding to the metadata file associated with the entry. For example, if the metadata file corresponding to the entry is a snapshot, then the filesystem name C11 stores the filesystem name of the filesystem on which the snapshot is based (the parent filesystem).

[0049] Type C13 stores a type name that indicates whether the metadata file corresponding to the entry corresponds to a file system or a snapshot of the file system.

[0050] The snapshot name C15 contains the snapshot name corresponding to the metadata file that corresponds to the entry. If the metadata file corresponding to the entry is a metadata file about the file system, the snapshot name C15 will contain "-" to indicate that there is no data.

[0051] In this embodiment, the path to the metadata file corresponding to each entry can be identified, for example, according to predetermined rules or entry values. For example, the metadata file for the first line may be FS1 / metadata file, and the metadata file for the second line may be FS1 / Snap / FS1_20240822_001 / metadata file. Furthermore, each entry may include a field to store the path name of the metadata file.

[0052] Figure 6 shows an example of a metadata file related to Example 1.

[0053] A metadata file T3 is created for each file system and snapshot. The metadata file T3 stores metadata about files and directories as entries. In this embodiment, the metadata file T3 is configured in a format searchable by S3select, such as CSV, JSON, or Apache Parquet.

[0054] The entries in metadata file T3 include the fields: path C31, type C32, size C33, creation date C35, modification date C37, and status change C39. Note that entries may also include other metadata such as access rights.

[0055] The path C31 stores the path to the file or directory corresponding to the entry. The type C32 stores a flag indicating whether the entry corresponds to a file or a directory. The size C33 stores the size of the file or directory corresponding to the entry. The creation date and time C35 stores the date and time the file or directory corresponding to the entry was created.

[0056] The modification date and time C37 stores the date and time the file or directory corresponding to the entry was last modified. The state change C39 stores the state change of the file or directory corresponding to the entry from the state of the previous snapshot. The state change C39 stores creation, update, or deletion. Creation indicates that a file or directory that did not exist in the previous snapshot was created. Update indicates that a file or directory that existed in the previous snapshot was updated. Deletion indicates that a file or directory that existed in the previous snapshot was deleted.

[0057] Figure 7 shows an example of an operation log list related to Example 1.

[0058] The operation log list T5 stores entries (logs) for each file I / O operation. File I / O operations include operations on files and directories. Each entry in the operation log list T5 includes fields for operation type C51, file system name C52, file handler C53, type C55, and timestamp C57.

[0059] The operation type C51 stores the type of file I / O operation corresponding to the entry. Examples of operation types include Create, Write, and Delete. The file system name C52 stores the name of the file system to which the target of the operation (file or directory) corresponding to the entry belongs. The file handler C53 stores the file handler for the target of the operation corresponding to the entry. The type C55 stores the type name of the target of the operation corresponding to the entry, i.e., a value indicating whether it is a file or a directory. The timestamp C57 stores information indicating the time when the operation corresponding to the entry was performed.

[0060] Figure 8 shows an example of a file storage list according to Example 1.

[0061] File storage list T7 is a table that stores access information for accessing each file storage 10 located at multiple locations. File storage list T7 stores an entry for each file storage 10. Each entry in file storage list T7 includes fields for location C71, file storage name C73, and access information C75.

[0062] Location C71 stores the name of the location where the file storage 10 corresponding to the entry resides. File storage name C73 stores the name of the file storage 10 corresponding to the entry. Access information C75 stores information for accessing the file storage 10 corresponding to the entry. Access information includes, for example, an IP address, username, and password.

[0063] Figure 9 shows an example of the data search results related to Example 1.

[0064] The data search result T9 is the search result for client 40's data search request. The data search result T9 includes, for example, an entry about the search result (file or directory). The entries in the data search result T9 include fields for name C90, path C91, location C92, type C93, size C94, creation date C95, and modification date C97.

[0065] Name C90 stores the name of the result corresponding to the entry. Path C91 stores the full path of the result corresponding to the entry. Location C92 stores the location name of the file storage 10 where the result corresponding to the entry is stored. Type C93 stores the type name of the result corresponding to the entry. Size C94 stores the size of the result corresponding to the entry. Creation Date & Time C95 stores the date and time the result corresponding to the entry was created. Modification Date & Time C97 stores the date and time the result corresponding to the entry was last modified.

[0066] Next, we will explain in detail the processing operations in computer system 1.

[0067] Figure 10 is a flowchart showing an example of the search process according to Example 1.

[0068] The search process S100 is performed in the search server 20 by the CPU 200 executing the data search UI program P11 and the data search program P13.

[0069] The data search UI program P11 (more precisely, the CPU 200 that executes the data search UI program P11) displays a data search UI to the client 40, receives a global data search request from the client 40 via the data search UI that can target multiple file storages 10, and requests the data search program P13 to perform a data search in accordance with the global data search request (S101). Here, the global data search request may include conditions to filter the locations and file storages 10 to be searched.

[0070] When the data retrieval program P13 receives a data retrieval request, it retrieves one unprocessed entry from the file storage list T7 (S102).

[0071] The data retrieval program P13 checks the location name of location C71 and the file storage name C73 of the entry obtained from the file storage list T7, and confirms whether the file storage 10 corresponding to the entry is included in the scope of the data retrieval request (S103).

[0072] As a result, if the file storage corresponding to the entry is included in the search scope (S103:Yes), the process proceeds to step S104; otherwise, the process proceeds to step S102. In this process, it is possible to avoid making a search request to file storage 10 at locations that are not included in the search scope in the global data search request.

[0073] In step S104, the data retrieval program P13 refers to the access information C75 of the entry obtained from the file storage list T7, and uses the access information to obtain the metadata file list T1 of the target file storage from the file storage 10 (referred to as the target file storage in this description) corresponding to this entry (S104).

[0074] The data retrieval program P13 retrieves one unprocessed entry from the acquired metadata file list T1 (S105).

[0075] The data retrieval program P13 issues an S3select (an example of a search request for file storage) to the target file storage in response to the search request for the metadata file T3 corresponding to the retrieved entry (S106).

[0076] As a result, in the target file storage, the S3 gateway program P3 will obtain S3select, search for the metadata file T3 according to the obtained S3select, and send the search results to the search server 20.

[0077] The data search program P13 retrieves search results from the target file storage using S3select and reflects the retrieved search results in the data search results T9 (S107). Specifically, the data search program P13 adds an entry to the data search results T9 and stores the retrieved search results, file name, and location information in that entry. For example, if the search results include multiple versions of a file, an entry for each version with the same path is added to the data search results T9.

[0078] Next, the data retrieval program P13 checks whether the entry obtained from the metadata file list T1 in step S105 is the last entry (S108). If the entry is the last entry (S108: Yes), the process proceeds to step S109. If the entry is not the last entry (S108: No), the process proceeds to step S105 to process the next entry.

[0079] In step S109, the data retrieval program P13 checks whether the entry obtained from the file storage list T7 in step S102 is the last entry. If the entry is the last entry (S109: Yes), the process proceeds to step S110. If the entry is not the last entry (S109: No), the process proceeds to step S102 to process the next entry.

[0080] In step S110, the data search program P13 returns the data search result T9 to the data search UI program P11, and the data search UI program P11 displays the data search result T9 to the requesting client 40 in a user-friendly format.

[0081] This search process reduces the processing load on the search server 20 because it does not directly search for data on the server 20, but instead has each file storage 10 perform the search. For example, if the search server 20 is configured on a public cloud, the cost required to perform the processing on the public cloud can be reduced. In addition, since the search process is performed on multiple file storage 10, the search processing time can be reduced. Furthermore, each file storage 10 only needs to execute SQL (S3select in this example) on a single file, resulting in a low processing load and reduced processing time for the search.

[0082] Next, we will explain in detail the processing operations in file storage 10.

[0083] Figure 11 is a flowchart showing an example of file I / O processing according to Example 1.

[0084] File I / O processing S200 is performed in the file storage 10 by the CPU 100 executing the network file system program P1 and the local file system program P7.

[0085] When a file I / O request is sent from client 40, the network file system program P1 performs protocol processing to accept the file I / O request and requests the local file system program P7 to perform the file I / O operation corresponding to the file I / O request (S201).

[0086] The local file system program P7 performs the requested file I / O operation on the file system (S204).

[0087] Next, the local file system program P7 appends the details of the executed file I / O operation to the operation log list T5 (S205).

[0088] Next, the local file system program P7 returns the operation result to the network file system program P1, and the network file system program P1 responds to the client 40, i.e., returns the operation result (S206).

[0089] Figure 12 is a flowchart showing an example of the metadata file update process according to Example 1.

[0090] The metadata file update process S300 is executed periodically, for example, at intervals shorter than the interval at which snapshots are taken (e.g., every 1 minute, every 5 minutes, etc.), or it is executed in the snapshot acquisition process S400 described later. The metadata file update process S300 is performed in the file storage 10 by the CPU 100 executing the metadata file update program P9.

[0091] The metadata file update program P9 retrieves the portion of the operation log list T5 from the entry processed in the previous metadata file update process S300 onwards (S301). In the following description of the process, the retrieved portion of the operation log list T5 will be referred to as the retrieved log list. Note that the portion from the entry processed in the previous metadata file update process onwards may be identified by, for example, storing the timestamp of when the metadata file update process was executed and using that timestamp.

[0092] Next, the metadata file update program P9 retrieves one unprocessed entry from the retrieved log list (S302).

[0093] Next, the metadata file update program P9 obtains a path from the file system name C52 of the acquired entry and the file handler C53, and checks whether the acquired path exists in the metadata file T3 (referred to as the target metadata file in this explanation of the process) for the file system with the file system name C52 (or, if executed in the snapshot acquisition process S400, the snapshot to be created) (S303). If the acquired path exists in the target metadata file (S303: Yes), the process proceeds to step S304; otherwise, the process proceeds to step S305.

[0094] In step S304, the metadata file update program P9 retrieves metadata for the file corresponding to the acquired path (referred to as the target file in this description) and updates the information of the target file's entry in the target metadata file. For example, if the operation type C51 of the entry is Create, the status change C39 of the target file's entry in the target metadata file is updated to Create. Note that if the operation type C51 of the entry is Create, the only possible status change C39 for the existing target file's entry is Delete. Also, if the operation type C51 is Write, the status change C39 of the target file's entry in the target metadata file is not updated. This is because if the status change C39 of the existing entry is Create, overwriting it with Update would make it impossible to determine whether the target file was created at this version. Note that if the processing in step S304 or step S305 has already been performed on the target file within the metadata file update process S300, the processing in step S304 may be skipped. Additionally, if the operation type C51 is Delete, the size C33, creation date C35, and modification date C37 information of the target file's entry are deleted, and the status change C39 is updated to "Deleted".

[0095] In step S305, the metadata file update program P9 retrieves the metadata of the target file and adds an entry for the target file to the target metadata file. Here, if the operation type C51 is Create, the status change C39 of the entry is set to "Created", if it is Write, the status change C39 is set to "Updated", and if it is Delete, the status change C39 is set to "Deleted".

[0096] After performing step S304 or step S305, the metadata file update program P9 checks whether the entry obtained in step S302 is the last entry in the retrieved log list (S306). If it is the last entry (S306: Yes), the metadata file update program P9 terminates processing; otherwise, if it is not the last entry (S306: No), the metadata file update program P9 proceeds to step S302 to process the next entry.

[0097] This metadata file update process allows the metadata file T3 to store the latest metadata about the file.

[0098] Figure 13 is a flowchart showing an example of the snapshot acquisition process according to Example 1.

[0099] The snapshot acquisition process S400 is executed when a snapshot acquisition request is received from the client 40, or when it is time to perform snapshot acquisition according to the snapshot acquisition settings (for example, settings to acquire snapshots periodically, such as every day). The snapshot acquisition process S400 is performed in the file storage 10 by the CPU 100 executing the network file system program P1.

[0100] The network file system program P1 performs the metadata file update process S300 (S401).

[0101] Next, the network file system program P1 takes a snapshot of the file system that is the target of the snapshot (target file system) (S402).

[0102] Next, the network file system program P1 deletes the contents of the target file system's metadata file T3 (S403). This is because the metadata file T3 only stores metadata that represents the differences from the state of the previous version's snapshot.

[0103] Note that, regarding steps S401 and S402, step S402 may be executed first, followed by step S401. However, in this case, step S401 refers to the log in the operation log list T5 up to the time of snapshot acquisition in step S402, and updates the metadata file T3 corresponding to the snapshot.

[0104] This snapshot acquisition process allows metadata such as files that represent the differences from the state of the previous snapshot to be stored in the metadata file T3 corresponding to the snapshot being acquired.

[0105] Figure 14 is a flowchart showing an example of the snapshot deletion process according to Example 1.

[0106] The snapshot deletion process S500 is executed when a snapshot deletion request is received from client 40, or when it is time to follow the snapshot deletion settings (for example, settings to delete snapshots periodically). The snapshot deletion process S500 is performed in the file storage 10 by the CPU 100 executing the network file system program P1.

[0107] The network file system program P1 obtains a metadata file T3 corresponding to the snapshot immediately following the snapshot to be deleted (called an incremental snapshot) (S501).

[0108] Next, the network file system program P1 retrieves one unprocessed entry from the acquired metadata file T3 (S502).

[0109] Next, the network file system program P1 checks whether the path C31 of the acquired entry exists in the metadata file T3 of the snapshot to be deleted (S503). If the result is that the path C31 of the acquired entry exists in the metadata file T3 of the snapshot to be deleted (S503: Yes), the network file system program P1 proceeds to step S504 to update the entry in the metadata file T3 of the snapshot to be deleted. On the other hand, if the path C31 of the acquired entry does not exist in the metadata file T3 of the snapshot to be deleted (S503: No), the network file system program P1 proceeds to step S510.

[0110] In step S504, the network file system program P1 checks the state change of the entry C39 obtained in step S502. If the state change of C39 is "created" (S504: created), the network file system program P1 proceeds to step S505. If the state change of C39 is "deleted" (S504: deleted), the network file system program P1 proceeds to step S506. If the state change of C39 is "updated" (S504: updated), the network file system program P1 proceeds to step S509.

[0111] In step S505, the network file system program P1 overwrites the corresponding entry in the metadata file T3 of the snapshot to be deleted with the entry obtained in step S502. At this time, the entry's state change C39 is overwritten as "updated". This is because if the state change C39 of the entry obtained in step S502 is "created", the only possible state change C39 of the entry of the snapshot to be deleted is "deleted", and deleting the snapshot to be deleted will be as if the deletion never happened.

[0112] In step S506, the network file system program P1 checks the state change C39 of the entry in the metadata file T3 of the snapshot to be deleted, which corresponds to the entry in the metadata file T3 of the incremental snapshot. If the state change C39 is "created" (S506: created), the network file system program P1 proceeds to step S507. If it is "updated" (S506: updated), the network file system program P1 proceeds to S508. In this case, the state change C39 can never be "deleted".

[0113] In step S507, the network file system program P1 deletes the entry in the metadata file T3 of the snapshot to be deleted that corresponds to the entry in the metadata file T3 of the incremental snapshot.

[0114] In step S508, the network file system program P1 overwrites the entry in the metadata file T3 of the snapshot to be deleted with the entry in the metadata file T3 of the incremental snapshot.

[0115] In step S509, the network file system program P1 overwrites the entry in the metadata file T3 of the snapshot to be deleted with the entry in the metadata file T3 of the incremental snapshot. However, it does not overwrite the state change C39 of the entry.

[0116] In step S510, the network file system program P1 copies the entries from the incremental snapshot's metadata file T3 to the metadata file T3 of the snapshot to be deleted.

[0117] After performing steps S505, S507-S510, the network file system program P1 checks whether the entry obtained in step S502 is the last entry in the incremental snapshot metadata file T3 (S511).

[0118] As a result, if the entry is the last entry in the metadata file T3 (S511:Yes), the network file system program P1 proceeds to step S512, while if the entry is not the last entry in the metadata file T3 (S511:No), the network file system program P1 proceeds to step S502 and performs processing for the next entry.

[0119] In step S512, the network file system program P1 makes the updated metadata file T3 of the snapshot to be deleted the new metadata file T3 of the incremental snapshot (S512).

[0120] Next, the network file system program P1 deletes the snapshot to be deleted (S513).

[0121] According to the snapshot deletion process S500, when deleting a snapshot, the metadata file T3 of the incremental snapshot can be put into an appropriate state. [Examples]

[0122] Next, Example 2 will be described. In the computer system 1A according to Example 2, the local search program P5 of the file storage 10A issues an S3select within the file storage. This reduces cross-location communication between the search server 20 and each file storage 10A, thereby reducing the load on the search server 20. In Example 2, parts similar to those in Example 1 are denoted by the same reference numerals, and redundant explanations may be omitted.

[0123] Figure 15 is a diagram illustrating the overview of Example 2. In Example 2, parts similar to those in Example 1 are denoted by the same reference numerals, and redundant explanations may be omitted.

[0124] In Example 2, the difference from Example 1 shown in Figure 1 is that the search server 20 does not issue an S3select, but instead forwards the search request based on the client 40's search request to each file storage 10A, and the local search program P5 in each file storage 10A issues an S3select for each metadata file T3 in that file storage 10A to perform the search.

[0125] Figure 16 is a configuration diagram showing an example of file storage according to Example 2.

[0126] The file storage 10A in Example 2 further stores the local search program P5 and the data search results T9 in the memory 110 of the file storage 10 in Example 1.

[0127] The local search program P5 is executed on the CPU 100, receives data retrieval requests from the search server 20 and other sources, and performs search processing according to these data retrieval requests.

[0128] Figure 17 is a flowchart showing an example of the search process according to Example 2.

[0129] The search process S1000 is performed by the CPU 200 executing the data search UI program P11 and the data search program P13 on the search server 20, and the CPU 100 executing the local search program P5 on the file storage 10A.

[0130] The processing in steps S1001 to S1003 is the same as steps S101 to S103 in Figure 10.

[0131] In step S1004, the data retrieval program P13 refers to the access information C75 of the entry obtained in step S1002 and sends a search request (file storage search request) to the file storage 10A corresponding to this entry, in response to the data retrieval request from the client 40.

[0132] Next, the local search program P5 of the file storage 10A executes the local search process S1100 (see Figure 18) and returns the search results (S1005).

[0133] Next, the data retrieval program P13 receives the search results from the file storage 10A and reflects them in the data search results T9 (S1006).

[0134] Steps S1009 to S1010 are the same as steps S109 to S110 in Figure 10.

[0135] According to the search process S1000, the search server 20 does not directly search for data, but instead has each file storage 10 perform the search, thereby reducing the processing load on the search server 20. For example, if the search server 20 is configured on a public cloud, the cost required to perform the processing on the public cloud can be reduced. In addition, since the search process is performed on multiple file storage 10A, the search processing time can be reduced.

[0136] Figure 18 is a flowchart showing an example of the local search process in Example 2.

[0137] The local search process S1100 is performed in the file storage 10A by the CPU 100 executing the local search program P5.

[0138] In step S1101, the local search program P5 receives a search request from the search server 20.

[0139] Steps S1104 to S1108 differ in that the local search program P5 performs the processing, but the processing content is the same as in steps S104 to S108.

[0140] In the local search process S1100, when the local search program P5 issues an S3select in step S1106, the S3 gateway program P3 receives the S3select, searches the metadata file T3 according to the received S3select, and returns the search results to the local search program P5. [Examples]

[0141] Next, Example 3 will be described. In the computer system according to Example 3, the file storage 10B extracts the differential metadata, transfers the metadata to the search server 20B, and aggregates and manages the metadata in the DB (database) on the search server 20B. In Example 3, the same reference numerals are used for parts that are the same as in Example 1, and redundant explanations may be omitted.

[0142] Figure 19 is a configuration diagram showing an example of file storage according to Example 3.

[0143] In the file storage 10B according to Example 3, the memory 110 stores the metadata sending program P10, as in the file storage 10 according to Example 1. The file storage 10B does not necessarily have to include the S3 gateway program P3, the metadata file update program P9, the metadata file list T1, and the metadata file T3.

[0144] The metadata sending program P10 is executed on the CPU 100, which detects metadata updates from the operation log list T5 and sends the metadata to the search server 20B.

[0145] Figure 20 is a configuration diagram showing an example of a search server according to Embodiment 3.

[0146] The search server 20B according to Example 3 has a metadata management program P15 and a metadata database T11 in addition to the search server 20 according to Example 1. Note that the search server 20B does not necessarily have a data search program P13.

[0147] The metadata management program P15 is executed on CPU200 to manage the metadata database T11.

[0148] The metadata database T11 is a database that manages metadata collected from each file storage 10. Entries in the metadata database T11 have fields similar to those in the data search result T9.

[0149] Figure 21 is a flowchart showing an example of the search process according to Example 3.

[0150] The search process S2100 is performed in the search server 20B by the CPU 200 executing the data search UI program P11 and the metadata management program P15.

[0151] The data search UI program P11 displays a data search UI to the client 40, receives a global data search request from the client 40 via the data search UI that can target multiple file storages 10, and requests the metadata management program P15 to perform a data search in accordance with the global data search request (S2101).

[0152] Next, the metadata management program P15 refers to the metadata database T11, performs a search in response to the search request, and returns the search results to the data search UI program P11 (S2102).

[0153] Next, the data search UI program P11 displays the data search results T9 to the client 40 in a user-friendly format (S2110).

[0154] Figure 22 is a flowchart showing an example of the metadata transmission process according to Example 3.

[0155] The metadata transmission process S2200 is the process of sending and managing metadata about the file system in each file storage 10. The metadata transmission process S2200 is executed periodically, for example, at intervals shorter than the interval at which snapshots are taken.

[0156] The metadata transmission process S2200 is performed by having the CPU 100 execute the metadata transmission program P10 in each file storage 10, and by having the CPU 200 execute the metadata management program P15 in the search server 20B.

[0157] The processing in steps S2201 to S2202 is similar to that in steps S301 to S302 in Figure 12, although it is executed by the metadata sending program P10.

[0158] In step S2203, the metadata sending program P10 checks whether the metadata for the file indicated by the acquired entry has already been acquired by the metadata sending process S2200.

[0159] As a result, if metadata has already been obtained in metadata sending process S2200 (S2203: Yes), the metadata sending program P10 proceeds to step S2202. On the other hand, if metadata has not already been obtained in metadata sending process S2200 (S2203: No), the metadata sending program P10 proceeds to step S2204.

[0160] In step S2204, the metadata sending program P10 retrieves the metadata for the file indicated by the entry.

[0161] Next, the metadata sending program P10 checks whether the acquired entry is the last entry in the operation log list T5 (S2206).

[0162] As a result, if the acquired entry is the last entry in the operation log list T5 (S2206:Yes), the metadata sending program P10 proceeds to step S2207, while if the acquired entry is not the last entry in the operation log list T5 (S2206:No), it proceeds to step S2202.

[0163] In S2207, the metadata sending program P10 sends the acquired metadata to the search server 20B. Then, the metadata management program P15 on the search server 20B updates the metadata database T11 based on the sent metadata (S2208).

[0164] According to the metadata transmission process S2200, metadata such as files in the file system after the snapshot was taken can be appropriately reflected in the metadata database T11 of the search server 20B. This allows the search server 20B to properly search for data in the file system after the snapshot was taken.

[0165] It should be noted that the present invention is not limited to the embodiments described above, and can be implemented with appropriate modifications without departing from the spirit of the invention. [Explanation of Symbols]

[0166] 1,1A…Computer system, 10,10A,10B…File storage, 20,20B…Search server, 30…Network, 40…Client, 100…CPU, 110…Memory, 120…Cache, 130…Network I / F, 150…Storage device, 200…CPU, 210…Memory, 220…Cache, 230…Network I / F, 250…Storage device

Claims

1. A computer system having a search server and multiple file storages that manage a file system, The aforementioned search server, Accepting a search request for the file to be searched, A search request for file storage corresponding to the aforementioned search request is sent to the file storage, The search results corresponding to the search request for the file storage are received from the aforementioned file storage, The search results are returned to the source of the search request. The aforementioned file storage is The file storage stores a first metadata file which stores metadata about files in a snapshot of the file system managed by the aforementioned file storage, The search server receives a search request for the file storage, Based on the aforementioned search request for file storage, the first metadata file is searched for, The search results obtained through the search are returned to the search server. Computer system.

2. The file storage creates the first metadata file as the difference between it and the first metadata file of the previous snapshot. The computer system according to claim 1.

3. The aforementioned file storage is A second metadata file is created to store metadata related to file operations after the most recent snapshot was taken in the aforementioned file system. When the aforementioned file storage search request is received, the system searches for files from the first metadata file and the second metadata file. The computer system according to claim 1.

4. The aforementioned file storage is The system stores logs of file I / O operations on the aforementioned file system and creates the second metadata file based on the logs. The computer system according to claim 3.

5. The aforementioned search server, The search request is converted into one or more single file search requests for a single metadata file in the file storage. As a search request for the file storage, the single file search request is sent to the file storage having the corresponding metadata file. The aforementioned file storage is The system receives the single-file search request and searches for the file from the corresponding metadata file in accordance with the single-file search request. The computer system according to claim 1.

6. A file storage that manages the file system, It comprises a memory device and a processor, The storage device stores a first metadata file which stores metadata about files in a snapshot of the file system managed by the file storage, The aforementioned processor, Receive a search request for file storage from the search server, Based on the aforementioned search request for file storage, the first metadata file is searched for, The search results obtained through the search are returned to the search server. File storage.

7. A computer system having a search server and multiple file storages that manage a file system, The aforementioned file storage is The file storage manages the file system and sequentially transmits metadata related to the manipulated files to the search server. The aforementioned search server, The metadata is received from the file storage and reflected in the metadata database. Accepting a search request for the file to be searched, Using the metadata database, search for the file corresponding to the search request. The search results obtained through the search are returned to the requester of the search request. Computer system.

Images