Management system, how the management system performs its actions

Abstract

This increases the likelihood that storage at a tier relatively close to the computing resources is holding the data at the time it is used to execute a job. [Solution] Before starting the execution of job 102, the job analysis unit 300 analyzes the workflow of job 102 to identify the data that the computing resource 210 will read when executing job 102. The pre-read management unit 1200 controls the pre-reading from secondary storage 240 to primary storage 230 for data identified by the job analysis unit 300 for job 102 where the actual data 920 does not exist in primary storage 230 but does exist in secondary storage 240.

Description

Technical Field

[0001] The present disclosure relates to a technique for prefetching data between storages.

Background Art

[0002] For computing resources that execute job processing, the storage that stores data read during job execution may be composed of multiple layers. For example, in order to optimize model parameters (perform big data analysis processing) using learning data such as big data to train a model for realizing artificial intelligence (AI), or to perform preprocessing on learning data before the optimization processing (analysis processing), an integrated platform may be provided that includes a compute server forming computing resources and a storage that stores data read during job execution. This integrated platform may include file storage as primary storage, which is the first layer of storage (the layer relatively close when viewed from the computing resources (compute server)), and object storage as secondary storage, which is the next layer of storage (the layer relatively far when viewed from the computing resources (compute server)). Alternatively, the integrated platform may transfer data between a secondary storage outside the integrated platform while including computing resources (compute server) and primary storage. When the storage has multiple layers as described above, if the actual data read by computing resources (e.g., a compute server) during job execution does not exist in the primary storage but exists in the secondary storage, the actual data needs to be transferred from the secondary storage to the primary storage. When the above transfer is performed, the computing resources (e.g., a compute server) may perform a rendezvous until the actual data read during job execution exists in the primary storage (i.e., an input / output bottleneck may occur).

[0003] To minimize waiting times when computing resources (e.g., compute servers) execute jobs, it is useful to control the system so that the actual data that the computing resources will read during job execution already exists in the storage tier relatively close to the computing resources (primary storage) at the time the computing resources execute the job. To achieve this, one possible approach is to pre-transfer the actual data that the computing resources will read during job execution from the storage tier relatively farther from the computing resources (secondary storage) to the storage tier relatively close to the computing resources (primary storage) (pre-fetching).

[0004] A prior art document relating to data lookup is Patent Document 1 (U.S. Patent No. 10084877). Patent Document 1 discloses a technology that records a graph showing the order of access between accessed data based on past access history, and then, when a certain data is accessed, uses the information in the graph to target data that is estimated to be highly likely to be accessed subsequently for lookup control. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] U.S. Patent No. 10084877 [Overview of the project] [Problems that the invention aims to solve]

[0006] Even assuming that the prior art relating to pre-fetch control disclosed in Patent Document 1 is applied to a system including computing resources (e.g., a compute server) and multi-tiered storage, it is still possible that the computing resources may experience many delays when executing jobs. Specifically, in the above scenario, when the computing resources (compute server) start executing a job and accesses actual data, it initiates a process of transferring (pre-fetching) the actual data that is estimated to be likely to be accessed subsequently from storage at a tier relatively far from the computing resources (secondary storage) to storage at a tier relatively close to the computing resources (primary storage). At this time, depending on the processing power of the computing resources (compute server) itself, the data transfer speed from primary storage to the computing resources (compute server), and the data transfer speed from secondary storage to primary storage, it is possible that the computing resources (compute server) may read and process the data before the actual data targeted for pre-fetching is present in primary storage, resulting in the computing resources (compute server) experiencing delays. As described above, when a queue occurs for job execution by computing resources (compute servers), the time required for job execution increases (job execution performance decreases). For example, if computing resources (compute servers) execute a job to perform optimization processing of model parameters using training data consisting of big data (big data analysis processing), or to perform preprocessing on the training data before such optimization processing (analysis processing), the time required for model parameter optimization processing (big data analysis processing) and preprocessing of the training data will increase.

[0007] Based on the above, one of the purposes of this disclosure may be to increase the likelihood that storage at a tier relatively close to the computing resources holds the data at the time the data is used to execute the job. [Means for solving the problem]

[0008] To achieve at least one of the above objectives, the features that this disclosure may have include, for example, the following: One of the disclosures is a management system. The management system is for managing computing resources and storage. The management system has a job analysis unit and a pre-fetch management unit. Before starting the execution of a job, the job analysis unit analyzes the workflow of the job to identify the data that computing resources will read when executing the job. The pre-fetch management unit controls the system to start pre-fetching from secondary storage to primary storage for data identified by the job analysis unit for a job, where the actual data does not exist in primary storage, which has relatively high access performance from computing resources, but the actual data exists in secondary storage, which has relatively low access performance from computing resources. [Effects of the Invention]

[0009] Based on the above, this disclosure can increase the likelihood that, at the time the data is used to execute a job, the storage at a tier relatively close to the computing resources is holding that data.

[0010] Methods and programs that achieve the same results as the management system described above can also achieve the same effects. In the case of a program, costs are often reduced. Furthermore, design changes related to processing are more easily implemented in a program. Any other features that this disclosure may possess, and the effects corresponding to such features, are disclosed in this specification, claims, or drawings. [Brief explanation of the drawing]

[0011] [Figure 1] The basic functional configuration of the embodiments disclosed herein is shown. [Figure 2] The overall configuration of the first embodiment is shown. [Figure 3] This shows the process performed by the job analysis unit. [Figure 4] Shows an example of a job workflow. [Figure 5] Shows an example of a command to call a job. [Figure 6] Shows an example of a job configuration file. [Figure 7] Shows job analysis information. [Figure 8] Shows the processing performed by the investigation department. [Figure 9] Shows the state of data (files). [Figure 10] Shows the processing performed by the file and object management department. [Figure 11] Shows the processing performed by the prefetch request department. [Figure 12] Shows the processing performed by the prefetch management department. [Figure 13] Shows the processing performed by the job assignment department. [Figure 14] Shows a scheduling policy configuration file. [Figure 15] Shows the processing performed by the sorting department. [Figure 16] Shows the processing performed by a modified example of the sorting department. [Figure 17] Shows the overall configuration of the second embodiment. [Figure 18] Shows the processing performed by the file and object management department of the second embodiment. [Figure 19] Shows the computer architecture of various servers.

Modes for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present disclosure will be described while referring to the drawings. Note that the embodiments described below do not limit the disclosure according to the claims, and not all of the elements and combinations thereof described in the embodiments are essential for the solution means of the present disclosure. Each of the systems, devices, or functional units of this disclosure may be a single, integrated hardware unit, or it may be divided into multiple parts that work together to perform their respective functions. Several systems, devices, or functional units may be integrated hardware-wise. Each system, device, or functional unit may be implemented by having a computer execute a program (as shown in Figure 19). Some of the functions of the system, device, or functional unit may be implemented in hardware (e.g., hardwired logic or field-programmable gate array (FPGA)), and the remaining functions may be implemented by program execution. All of the functions of each system, device, or functional unit may be implemented in hardware. Each of the systems, devices, or functional units of this disclosure may be implemented virtually. For example, virtual computer or virtual container techniques may be used. The program is not limited to any particular type or form. Furthermore, the program may initially be recorded in a compressed format. When a system, device, functional unit, or part of the functionality of a functional unit is implemented by having a computer execute a program, the implemented system, device, functional unit, or part of the functionality of the functional unit does not need to be implemented at all times. In other words, it is sufficient for the system, device, functional unit, or part of the functionality of the functional unit to be implemented only when the processing provided by that system, device, functional unit, or part of the functionality of the functional unit is needed. Reference numbers used in multiple drawings are considered equivalent. In flowcharts, rectangular boxes indicate processing steps, and hexagonal boxes indicate conditional branching steps. In flowcharts, "step" is abbreviated as "S". Also, in flowcharts, points with the same number circled in a circle are linked in terms of control.

[0013] 1. Basic Functional Configuration (Figure 1) Figure 1 shows the basic functional configuration 100 (and the information handled) of the management system 101 according to an embodiment of this disclosure. Not all functional configurations shown in Figure 1 are mandatory. Furthermore, it is not prohibited for functional configurations other than those shown in Figure 1 to exist. In Figure 1 (and Figures 2 and 17), solid rectangles with "department" appended to their names indicate functional units. Furthermore, Figure 1 consists of an upper, middle, and lower section. As time progresses, the situation changes from the state shown in the upper section of Figure 1 to the state shown in the middle section, and then to the state shown in the lower section of Figure 1. The management system 101 is for managing computing resources 210 and storage (primary storage 230 and secondary storage 240). As shown in Figure 1, the management system 101 has a job analysis unit 300 and a pre-read management unit 1200.

[0014] The top of Figure 1 shows the situation before computing resource 210 started executing job 102. In the situation shown at the top of Figure 1, the job analysis unit 300 identifies the data that the computing resource 210 reads when executing job 102. The job analysis unit 300 may identify the data that the computing resource 210 reads when executing job 102 by analyzing the workflow of job 102. The identified data entity 920 may reside in the primary storage 230, which has relatively high access performance from the computing resource 210, but not in the secondary storage 240, which has relatively low access performance from the computing resource 210; it may reside in both the primary storage 230 and the secondary storage 240; or it may reside in the secondary storage 240 but not in the primary storage 230. The top of Figure 1 shows the case where the identified data entity 920 resides in the secondary storage 240 but not in the primary storage 230. In a case where the actual data 920 exists in secondary storage 240 but not in primary storage 230, and the primary storage 230 (or computing resource 210) holds the management information for that data, the state of the data is said to be in stub state 903.

[0015] The middle section of Figure 1 shows the situation at a later time than the upper section of Figure 1, and before the computing resource 210 starts executing job 102. In the situation shown in the middle of Figure 1, the pre-read management unit 1200 controls the system to start pre-reading from secondary storage 240 to primary storage 230 for data identified by the job analysis unit 300 for job 102 where the actual data 920 does not exist in primary storage 230 but does exist in secondary storage 240. In the case shown at the top of Figure 1, where the actual data 920 exists in secondary storage 240 but not in primary storage 230, the pre-read management unit 1200 controls the system to start transferring (pre-reading) the actual data 920 from secondary storage 240 to primary storage 230.

[0016] The lower part of Figure 1 shows the situation at a later time than the middle part of Figure 1, and also shows the situation after the computing resource 210 has started executing job 102. As shown in the middle of Figure 1, pre-fetching of the data entity 920 that the computing resource 210 will read during the execution of job 102 begins before the computing resource 210 starts executing job 102. Therefore, as shown in the lower part of Figure 1, there is a high probability that the data entity 920 is held in the primary storage 230 at the time when the computing resource 210 is executing job 102 and attempting to actually read the data. At the very least, compared to starting pre-fetching of the data entity 920 to be read for the execution of job 102 while the computing resource 210 is executing job 102, the series of processes shown in Figure 1 increase the likelihood that the data entity 920 is present in primary storage 230 at the time the computing resource 210 attempts to read the data entity 920 from primary storage 230.

[0017] Although only one job 102 is shown in Figure 1, in reality, the management system 101 may handle multiple jobs 102 simultaneously. In that case, the management system 101 will perform the series of processes shown in Figure 1 for each job 102.

[0018] Since the management system 101 in the embodiment of this disclosure has the functional configuration described above, it can provide the aforementioned [effects of the invention] (the effects shown in paragraphs

[0010] to

[0011] ).

[0019] 2. Overall configuration, functional configuration, information, and processing of individual embodiments As embodiments of this disclosure, a first embodiment is described below in which the computing resources 210, primary storage 230, and secondary storage 240 are located at the same site, and a second embodiment is described in which the computing resources 210 and primary storage 230 are located at the same site, while the secondary storage 240 is located at a different site (another site 1799).

[0020] 2.1. First Embodiment The first embodiment described below assumes that the computing resource 210, primary storage 230, and secondary storage 240 are located at the same site. However, by appropriately adjusting how the data read by the computing resource 210 when executing job 102 is managed in a data space or file system that the computing resource 210 can recognize, this disclosure can be made less limited to the case where the computing resource 210, primary storage 230, and secondary storage 240 are located at the same site. (The case in which the secondary storage 240 is located at a different site 1799 can be said to be the second embodiment described later.)

[0021] 2.1.1. Overall configuration of the first embodiment (Figure 2) Figure 2 shows the overall configuration 200 of the first embodiment of this disclosure. Note that not all of the functional configurations (and the information handled) shown in Figure 2 are essential. Furthermore, the existence of functional configurations other than those shown in Figure 2 is not prohibited. This section provides an overview of each component of the configuration shown in Figure 2. Detailed processing and information will be explained later in section 2-1-3, "Functional Configuration, Processing, and Information of the First Embodiment."

[0022] In a first embodiment of this disclosure, the management system 101 (which may also be called an analysis platform) may include computing resources 210, primary storage 230, secondary storage 240, a compute management server 220, and a storage management server 250. The computing resources 210 may consist of one or more (N in Figure 1) compute servers 211. The primary storage 230 may consist of one or more (P in Figure 1) storage servers 231. The secondary storage 240 may consist of one or more (S in Figure 1) storage servers 241. Each of the above types of servers may be implemented using the computer architecture shown in "2.1.2. Computer Architecture for Realizing the First Embodiment of the Disclosure" and Figure 19, as described below. Depending on the type (role) of the server, the performance or capacity of each component shown in Figure 19 may be determined.

[0023] Each of the compute servers 211 forming the computing resources 210 may have a GPU. Each compute server 211 may cooperate to realize an execution infrastructure unit 212 as a functional unit by executing a program for realizing the execution infrastructure. The execution infrastructure unit 212 may cause a virtual computing resource, a container, to execute a job 102 that has been assigned to one of the compute servers 211 by the scheduler unit 221, which is a functional unit of the compute management server 220. Alternatively, the execution infrastructure unit 212 may manage computing resources other than containers. Furthermore, the processing performed by Job 102 here may be arbitrary. For example, Job 102 may be for processing models or data related to artificial intelligence (AI). Alternatively, Job 102 may perform optimization processing of model parameters using training data consisting of big data, etc., in order to train a model for realizing artificial intelligence (AI) (big data analysis processing, model training processing), or it may perform preprocessing on the training data before said optimization processing (analysis processing, model training processing). In this case, Job 102 may be called an analysis job. Moreover, Job 102 may be for performing estimation or inference using a trained (pre-learned) model for realizing artificial intelligence (AI). In this case, Job 102 may be called an inference job. By having the management system 101 handle the above-mentioned analysis jobs or inference jobs related to artificial intelligence (AI), the time required to perform analysis or inference processing related to artificial intelligence (AI) can be reduced.

[0024] Each of the storage servers 231 forming the primary storage 230 may collaborate to realize a file and object management unit 1000 as a functional unit by executing a program for file and object management. Similarly, each of the storage servers 241 forming the secondary storage 240 may collaborate to realize a file and object management unit 243 as a functional unit by executing a program for file and object management. Here, the program code for realizing file and object management may be common to both the primary storage 230 and the secondary storage 240, or it may differ. Furthermore, the file and object management unit 1000 in the primary storage 230 and the file and object management unit 243 in the secondary storage 240 may each provide functions corresponding to the role that the file and object management unit should play with respect to data (files). The file and object management unit 1000 in the primary storage 230 and the file and object management unit 243 in the secondary storage 240 may cooperate to present a file system for managing data (files) to the compute server 211 that forms the computing resources 210. The file and object management unit 1000 in the primary storage 230 may proactively determine the processing content for managing the file system, while the file and object management unit 243 in the secondary storage 240 may operate passively according to the decisions of the primary storage 230. The file system presented may be of any form. For example, as shown in Figure 2, it may be a system in which access to a file is made possible by specifying the file path (including the specification of the directory (Dir)) of the file corresponding to the data. By presenting this file system, virtualization and hierarchical control of data and files stored on the recording media of each storage server 231 and each storage server 241 are realized. To illustrate this, Figure 2 shows the virtualization and hierarchical control unit 232 as a functional unit within the file and object management unit 1000.

[0025] When primary storage 230 and secondary storage 240 are provided at the same location, the recording medium of the storage server 231 forming the primary storage 230 may be a recording medium with relatively high read / write performance (e.g., a solid-state drive (SSD)), while the recording medium of the storage server 241 forming the secondary storage 240 may be a recording medium with relatively low read / write performance (e.g., a hard disk drive (HHD)). In this way, the primary storage 230 and secondary storage 240 can offer the compute server 211 high-speed read / write performance and large storage capacity while keeping costs down. The size of the data (file) storage areas provided by the primary storage 230 and secondary storage 240 is arbitrary, but for example, the size of the storage area in the primary storage 230 may be approximately 1 petabyte, and the size of the storage area in the secondary storage 240 may be approximately 5 petabytes.

[0026] Furthermore, the recording medium of the storage server 231 forming the primary storage 230 and the recording medium of the storage server 241 forming the secondary storage 240 may be file storage, object storage, or file object storage, or any other type of storage. Here, file storage is storage that enables access by file path to the accessing entity. Object storage is storage that behaves as storage having a bucket-based recording area that is treated as a flat space for management purposes to the accessing entity. File object storage is storage that can behave as either file storage or object storage. For example, each of the storage servers 231 forming the primary storage 230 may be treated as having file storage, and the entire storage server 231 forming the primary storage 230 may present a high-speed distributed file system to each of the compute servers 211 forming the computing resources 210. On the other hand, the recording medium of the storage server 241 forming the secondary storage 240 may be treated as object storage as a backup destination in hierarchical control. With such a storage configuration, it becomes possible to provide the compute servers 211 with a file system that can be accessed via file paths, while also implementing hierarchically structured or virtualized storage. Alternatively, for example, the primary storage 230 and secondary storage 240 may be arranged to provide tiered control over the data lake.

[0027] The compute management server 220 is primarily for controlling each of the compute servers 211 that make up the computing resources 210. As shown in Figure 2, the compute management server 220 has a job analysis unit 300 and a scheduler unit 221 as functional units. The scheduler unit 221 has a pre-read request unit 1100, a job assignment unit 1300, and a sorting unit 1500 (or 1600) as internal functional units. The compute management server 220 handles job analysis information 700 and a scheduling policy setting file 1400. Details of the above-mentioned functional units and information in the compute management server 220 will be described later in "2.1.3. Functional Configuration, Processing, and Information of the First Embodiment". Furthermore, the job analysis unit 300 and the scheduler unit 221 in the compute management server 220 may be collectively referred to as the compute management unit. Furthermore, the inclusion relationships between the job analysis unit 300, the pre-read request unit 1100, the job assignment unit 1300, the sorting unit 1500 (or 1600), and the scheduler unit 221, as shown in Figure 2, are arbitrary. For example, the scheduler unit 221 may also include the job analysis unit 300. Alternatively, the scheduler unit 221 may not exist, and the job analysis unit 300, the pre-read request unit 1100, the job assignment unit 1300, and the sorting unit 1500 (or 1600) may each exist as separate functional units.

[0028] The storage management server 250 is primarily for controlling each of the storage servers 231 that form the primary storage 230, and each of the storage servers 241 that form the secondary storage 240. As shown in Figure 2, the storage management server 250 has a storage management unit 251 as a functional unit. The storage management unit 251 has an investigation unit 800 and a pre-read management unit 1200 as internal functional units. The storage management server 250 has job analysis information 700 as information to handle. Details of the above-mentioned functional units and information in the storage management server 250 will be described later in "2.1.3. Functional Configuration, Processing, and Information of the First Embodiment".

[0029] A data network 260 exists for mutual communication between each of the compute servers 211 that form the computing resources 210 and each of the storage servers 231 that form the primary storage 230. A data network 261 also exists for mutual communication between each of the storage servers 231 that form the primary storage 230 and each of the storage servers 241 that form the secondary storage 240. Furthermore, a management network 280 exists to send and receive control information between the various servers shown in Figure 2. If the computing resources 210, primary storage 230, and secondary storage 240 are located at the same site, then the data network 260, data network 270, and management network 280 may be what is known as a local area network. For example, the data network 260 may be InfiniBand compliant, and the data network 270 may be Ethernet compliant, but are not limited to these. Furthermore, if the computing resources 210, primary storage 230, and secondary storage 240 are located at the same site, the communication speed of the data network 260 may be faster than the communication speed of the data network 270. For example, the communication speed of the data network 260 may be several hundred gigabits per second, and the communication speed of the data network 270 may be approximately 10 gigabits per second to approximately 100 gigabits per second. Even in such a case, the embodiments of this disclosure can reduce the possibility of generating input / output bottlenecks. Furthermore, some or all of the data network 260, data network 270, and management network 280 may be integrated.

[0030] 2-1-2. Computer architecture for realizing various servers (Figure 19) Figure 19 shows a computer architecture 1900 for realizing various servers constituting the management system 101 of the embodiments of this disclosure (the first embodiment and the second embodiment described later). The computer architecture 1900 shown in Figure 19 may also be called an information processing device or information processing system. (Furthermore, the computer architecture 1900, which is an information processing device or information processing system, may be understood to be an implementer of a method.) To realize the various servers constituting the management system 101 of the embodiment of this disclosure, some or all of the following may be interconnected at the interconnection unit 1911: the arithmetic processing unit 1901, the storage device 1902, the non-volatile recording medium (recording device) 1903, the external recording medium drive 1904, the input device 1906, the display or output device 1907, the communication device 1908, the external input / output port 1909, and the reading device 1910. (Note that some or all of the interconnection unit 1911 may be a network. In that case, the various servers will be realized by multiple devices connected via the network.) The arithmetic processing unit 1901 may be, for example, a processor. Examples of such processors include a CPU, MPU, or GPU. Alternatively, the processor referred to herein may be any other semiconductor device that performs a predetermined process. Furthermore, the arithmetic processing unit 1901 may be one or more (micro)processors. The storage device 1902 may be, for example, memory. The non-volatile recording medium (recording device) 1903 may be, for example, non-volatile memory (e.g., flash memory) or a non-volatile disk device. The external recording medium drive 1904 may be, for example, a disk drive. The input device 1906 may be, for example, a mouse, keyboard, etc. The display or output device 1907 may be, for example, a display, printer, or speaker. The communication device 1908 may be, for example, a wired communication device or a wireless communication device. The communication device 1908 may be a network interface device (NIC). The interconnection unit 1911 may be, for example, a bus or a crossbar switch. (As mentioned above, part or all of the interconnection unit 1911 may be a network.)

[0031] The non-volatile recording medium (recording device) 1903 may record various programs included in the program group 1931, various data groups included in the data group 1932, or information included in the various information 1933. Program group 1931 may include various programs for realizing each of the functional units designated as "units" in the functional configuration diagrams of Figures 1, 2, and 17. Some of the above programs may be integrated into a single program. Alternatively, any of the above programs may be split into multiple programs. The data group 1932 may include information (data, etc.) handled by the above-mentioned functional unit. For example, the data group 1932 may include information that constitutes each of the information groups or data groups shown in the functional configuration diagrams or overall configuration diagrams in Figures 1, 2, and 17. (Note that some or all of the information included in the information group or data group may be stored in the storage device 1902 (memory).) Alternatively, some or all of the various programs included in the program group 1931, the various information groups or data groups included in the data group 1932, or the information included in the various information 1933 may be obtained from outside the configuration shown in Figure 19.

[0032] The external storage medium drive 1904 can connect to an external storage medium 1905. The external storage medium 1905 may be, for example, a portable recording disk, non-volatile memory (e.g., flash memory), etc. Alternatively, information similar to the various programs included in the program group 1931, the various information groups or data groups included in the data group 1932, or the information included in the various information 1933 may be transferred and stored from this external storage medium 1905 to the non-volatile storage medium (recording device) 1903 or the storage device 1902. Various programs included in program group 1931, various information groups or data groups included in data group 1932, or information included in various information 1933 may be provided via communication device 1908, external input / output port 1909, input device 1906, and reading device 1910, and recorded or stored in non-volatile recording medium (recording device) 1903 or storage device 1902.

[0033] In order for the architecture in Figure 19 to function as the management system 101, each functional unit within the management system 101, or a part of each functional unit (execute one or a series of processes (steps)), various programs included in the program group 1931 may be loaded into the storage device 1902 (for example, from the non-volatile recording medium (recording device) 1903). The loaded program is shown as 1921 in Figure 19. The arithmetic processing unit 1901 may then execute program 1921 (using, if necessary, various information groups or data groups included in the data group 1932 present in the non-volatile recording medium (recording device) 1903, etc., or information included in various information 1933). The execution of program 1921 realizes the functions of the management system 101, each functional unit within the management system 101, or a part of each functional unit (one or a series of processes (steps) are executed). Various buffers 1923 temporarily formed in the storage device 1902 may also be used as appropriate.

[0034] 2.1.3. Functional configuration, processing, and information of the first embodiment The following describes the functional configuration, processing, and information of the management system 101 in the first embodiment. Note that not all of the functional configurations (and information handled) described below are mandatory. Furthermore, the existence of functional configurations other than those described below is not prohibited.

[0035] 2-1-3-1. Job Analysis Department and Investigation Department (Figures 3-10) This section describes the functional configuration (processing) realized through the collaboration of the job analysis unit 300, a functional unit implemented on the compute management server 220; the investigation unit 800, a functional unit implemented on the storage management server 250; and the file and object management unit 1000, a functional unit implemented on the primary storage 230. The information used in this functional configuration (processing) is also described. The functional configuration, processing, and information described below allow for the analysis of a job before its execution begins, enabling the identification of the data to be loaded into the computing resources (compute server) during the job's execution. As explained below, when the data to be read during the execution of a job is identified by analyzing the job's workflow, the data can be identified more accurately or in greater quantity than when the data is identified based on past access patterns (historical analysis). Furthermore, the status of each piece of data loaded into the computing resources (compute server) during job execution can be monitored. This monitoring may include the data (file) status, indicating whether the actual data resides in primary or secondary storage, and the stub file size, which is the data size of the portion of the data where the actual data does not reside in primary storage but resides in secondary storage. As a result, more accurate information will be collected for use in data pre-fetching control, as described in section 2-1-3-2, "Pre-fetch Request Unit and Pre-fetch Management Unit," and in job allocation control to computing resources (compute servers), as described in section 2-1-3-3, "Job Assignment Unit and Sorting Unit."

[0036] Figure 3 shows a flowchart of the processing of the job analysis unit 300, which is a functional unit implemented in the compute management server 220. Each step shown in Figure 3 may be considered to constitute a "job analysis step". In step 301 of Figure 3, the job analysis unit 300 determines whether there are any jobs 102 that the job analysis unit 300 has not yet analyzed (unanalyzed jobs) among the jobs 102 recognized by the scheduler unit 221. Here, the jobs 102 recognized by the scheduler unit 221 are jobs 102 that the scheduler unit 221 plans to assign to one of the compute servers 211 that make up the computing resources 210. Furthermore, the jobs 102 that the job analysis unit 300 has not yet analyzed are jobs 102 for which the data (files) to be read during the execution of the job 102 has not yet been identified, and the status of the data (files) has not yet been investigated. If the result of the determination in step 301 is positive, control proceeds to step 302. If the result of the determination in step 301 is negative, step 301 is repeated. In step 302 of Figure 3, the job analysis unit 300 selects one of the unanalyzed jobs.

[0037] In step 303 of Figure 3, the job analysis unit 300 identifies the data (file) that one of the compute servers 211 forming the computing resources will read when executing job 102, which was selected in the most recent step 302. The data may be in the form of a file. When the job analysis unit 300 identifies the data (file) that one of the compute servers 211 will read when executing job 102, the job analysis unit 300 may identify the data (file) by analyzing the workflow of job 102.

[0038] Figure 4 shows an example workflow 400 for job 102 (referred to as job W in Figures 4, 5, and 6). Here, Figure 4 shows the case where the primary storage 230 can be accessed as file storage, and the data access path is set to file path 705. If the primary storage 230 can be accessed as object storage via a URL starting with http or https, the data access path may also be a URL starting with http or https. Figure 4 illustrates the user interface configuration in low-code development. In the example in Figure 4, the processes executed in job W consist of processes X, Y, and Z. Process X takes two data files as input: one with file path 705 being "C:\\dirJ\fileJ.csv" (where file path 705 refers to the CSV file fileJ.csv located under the drive dirJ, which is under the root of the C drive; the same applies hereafter), and another with file path 705 being "C:\\dirK\fileK.csv". The output of process X is a data file with file path 705 being "C:\\dirL\fileL.csv". Process Y takes data (file) with file path 705 "C:\\dirM\fileM.csv" as input, and outputs data (file) with file path 705 "C:\\dirN\fileN.csv". Process Z takes as input data (file) with file path 705 "C:\\dirP\fileP.csv", data (file) with file path 705 "C:\\dirL\fileL.csv" which is the output of Process X, and data (file) with file path 705 "C:\\dirN\fileN.csv" which is the output of Process Y, and sets the output of Process Z to data (file) with file path 705 "C:\\dirQ\fileQ.csv". In the example in Figure 4, the data (files) that computing resource 210 (compute server) reads when executing job W are the data (file) with file path 705 "C:\\dirJ\fileJ.csv", the data (file) with file path "C:\\dirK\fileK.csv", the data (file) with file path 705 "C:\\dirM\fileM.csv", and the data (file) with file path 705 "C:\\dirP\fileP.csv".

[0039] Within the information describing the example workflow 400 for job W shown in Figure 4, one or more objects (object X, object Y, and object Z in Figure 4) may be used. In the example in Figure 4, object X may include information that identifies the type of process X, information that identifies the file path 705 of the data (file) that is the input for process X, and information that identifies the file path 705 of the data (file) that is the output, which is the result of process X. Similarly, object Y may include information that identifies the type of process Y, information that identifies the file path 705 of the data (file) that is the input for process Y, and information that identifies the file path 705 of the data (file) that is the output, which is the result of process Y. Similarly, object Z may include information that identifies the type of process Z, information that identifies the file path 705 of the data (file) that is the input for process Z, and information that identifies the file path 705 of the data (file) that is the output, which is the result of process Z.

[0040] In step 302, the job analysis unit 300 may identify the data (file) that one of the compute servers 211 forming the computing resources will read when executing job 102 by understanding the information contained within the information describing the job workflow, as shown in Figure 4. When the job analysis unit 300 analyzes information describing the job's workflow and identifies data (files) that one of the compute servers 211 forming the computing resources will read when executing job 102, the job analysis unit 300 can accurately identify the data (files) in detail by understanding the workflow of job 102.

[0041] Alternatively, the job analysis unit 300 may identify the data that one of the compute servers 211 forming the computing resources will read when executing job 102 by referring to the arguments of the command for calling the job. Figure 5 shows an example command 500 for calling job 102. Here, Figure 5 shows the case where the primary storage 230 can be accessed as file storage and the access path to the data is the file path 705. If the primary storage 230 can be accessed as object storage via a URL starting with http or https, the access path to the data may also be a URL starting with http or https. Figure 5 shows an example of a command to invoke Job W having the workflow shown in Figure 4. As shown in Figure 5, the command to invoke Job W includes "JobW", which indicates the name of Job W itself. The command may also include as arguments "C:\\dirJ\fileJ.csv", "C:\\dirK\fileK.csv", "C:\\dirM\fileM.csv", and "C:\\dirP\fileP.csv", which are information that identifies the file path 705 of the data (file) to be read when Job W is executed. Furthermore, the command may include as a return value "C:\\dirQ\fileQ.csv", which is information that identifies the file path 705 of the output data (file) that shows the processing result of Job W.

[0042] When the job analysis unit 300 identifies the data (file) that one of the compute servers 211 forming the computing resources will read when executing job 102 by referring to the arguments of the command for calling the job, the job analysis unit 300 can generally understand the data (file) without having a detailed understanding of the inside of the workflow.

[0043] Alternatively, the job analysis unit 300 may obtain specific information about the data (files) that one of the compute servers 211 forming the computing resources will read when executing job 102, by referring to the configuration file for job 102. Figure 6 shows an example 600 of a configuration file for job 102. The configuration file is what the compute server 211, which forms the computing resource 210, reads when job 102 is executed. Here, Figure 6 shows the case where the primary storage 230 can be accessed as file storage, and the access path to the data is set to the file path 705. If the primary storage 230 can be accessed as object storage via a URL starting with http or https, the access path to the data may also be a URL starting with http or https. Figure 6 shows an example of a configuration file for job W having the workflow shown in Figure 4. The configuration file for job W may include "job W" which indicates the name of job W itself, "C:\\dirJ\fileJ.csv", "C:\\dirK\fileK.csv", "C:\\dirM\fileM.csv", and "C:\\dirP\fileP.csv", which are information that identifies the file path 705 of the data (file) that is the output data (file) showing the processing result of job W, and "C:\\dirQ\fileQ.csv". The job analysis unit 300 identifies the data (file) that one of the compute servers 211 forming the computing resources will read when executing job 102 by extracting the information portion that identifies the file path 705 of the data (file) to be read when job W is executed from the configuration file shown in Figure 6.

[0044] The job analysis unit 300 can identify the data (file) that one of the compute servers 211 that make up the computing resources will read when executing job 102, by referring to the configuration file for the job. In such cases, the job analysis unit 300 can generally understand the data (file) without having a detailed understanding of the workflow.

[0045] In step 304 of Figure 3, the job analysis unit 300 creates a record in the job analysis information 700 for each of the data (files) identified in the most recent step 303. Figure 7 shows the job analysis information 700. Figure 7 is shown in table form. Each row in this table corresponds to a record. Each record may correspond to one of the data (files) that any of the compute servers 211 forming the computing resources read when executing job 102. In the example of job W shown in Figures 4, 5, and 6, records for job analysis information 700 may be provided for each of the data (files) "C:\\dirJ\fileJ.csv", "C:\\dirK\fileK.csv", "C:\\dirM\fileM.csv", and "C:\\dirP\fileP.csv".

[0046] As shown in Figure 7, each record of the job analysis information 700 may have the following items: job number 701, job registration time 702, job execution time 703, file identifier 704, file path 705, data (file) status 900, total file size 706, stub file size 707, pre-read time 708, and pre-read execution status 709. Job number 701 indicates information that identifies job 102, which uses the data (file) corresponding to the record. Note that job number 701 may also be a general identifier other than a number. The job registration time 702 may be the time when the scheduler unit 221 recognizes that job 102, which uses the data (file) corresponding to the record, is to be assigned to one of the compute servers 211, or the time when the record is registered. The job execution time 703 represents an estimated time required for job 102, which uses data (files) corresponding to records, to be executed on the compute server 211. The job execution time 703 may be a historical actual time or a statistical value thereof, or it may be a value derived from some model formula. The file identifier 704 is information that identifies the data (file) corresponding to the record. Note that if the file path 705 (described later) is always set, this file identifier 704 does not need to exist. The file path 705 is information indicating the location of the data (file) corresponding to the record in the file system managed by the file and object management unit 1000. Here, Figure 7 shows the case where the primary storage 230 can access files as file storage, and the access path to the data is set to the file path 705. If the primary storage 230 can access the data as object storage using a URL starting with http or https, the access path to the data may also be a URL starting with http or https. The data (file) status 900 indicates whether the actual data (file) corresponding to the record (data entity 920) exists in primary storage 230 or secondary storage 240. Further details will be provided later with reference to Figure 9. The total file size of 706 indicates the size of the data (file) corresponding to the record. This total file size of 706 represents the overall size of the data (file), regardless of which storage the actual data (920) resides in. The stub file size 707 indicates the size of the uncached portion of the data (file) in primary storage 230 when the actual data (file) corresponding to a record does not exist in primary storage 230, but does exist in secondary storage 240. For example, in the example of the data (file) in Figure 7 where file path 705 is "C:\\dir2\file2-1", the stub file size 707 is 100 gigabytes out of a total file size of 200 gigabytes (706). In other words, for 100 gigabytes of the 200 gigabytes of data, the actual data (file) is not cached in primary storage 230, but exists in secondary storage 240. The prefetch time 708 indicates the estimated time required to transfer (prefetch) the actual data 920, represented by the stub file size 707, from secondary storage 240 to primary storage 230. For example, in the example of data (file) where the file path 705 in Figure 7 is "C:\\dir2\file2-1", the estimated time required to transfer (prefetch) the actual data 920 with a stub file size 707 of 100 gigabytes is 100 seconds. Note that in Figure 7, the item for prefetch time 708 is shown as "100 / 200", where the " / 200" part indicates the estimated time required to prefetch if the actual data 920 with a total file size of 706 is to be transferred (prefetched). This " / 200" part is optional. The method of utilizing the look-ahead time 708 is arbitrary. The look-ahead time 708 may be used for the control of look-ahead. For example, the look-ahead time 708 may be used to adjust the time between when look-ahead of data (files) from secondary storage 240 to primary storage 230 begins and when computing resource 210 (compute server) uses the look-ahead data (files). If the look-ahead time 708 is used in the manner described above, the likelihood that computing resource 210 (compute server) will be able to read the data (files) at the time it uses the data (files) increases. The prefetch execution status 709 indicates the execution status of the transfer (prefetch) of data (files) corresponding to records from secondary storage 240 to primary storage 230. If this prefetch execution status 709 is "Completed," it means that the transfer (prefetch) is complete or that the transfer (prefetch) was not required in the first place. If it is "In Progress," it means that the transfer (prefetch) is in progress. If it is "Waiting," it means that the required transfer (prefetch) has not started or that the transfer (prefetch) has been interrupted for some reason.

[0047] As shown in Figure 7, the job analysis information 700 is provided, which allows the computing resource (compute server) to identify each of the data (files) it reads when executing job 102, and to understand the status of that data (file). Furthermore, at step 304 in Figure 3, when the record for job analysis information 700 is created, the following items in the record are not necessarily set: data (file) status 900, total file size 706, stub file size 707, pre-fetch time 708, and pre-fetch execution status 709. These items may be set by the investigation unit 800 and the file and object management unit 1000, as described later.

[0048] In step 305 of Figure 3, the job analysis unit 300 requests the investigation unit 800, a functional unit implemented in the storage management server 250, to investigate the status of each of the data (files) identified in the most recent step 303.

[0049] Figure 8 shows a flowchart of the processing of the investigation unit 800, which is a functional unit implemented in the storage management server 250. Each step shown in Figure 8 can be considered to constitute an "investigation step". In step 801 of Figure 8, the investigation unit 800 determines whether there is a request from the compute management server 220 (or its job analysis unit 300) to investigate the status of the data (files). This request is the same as the request in step 305 of Figure 3. If the result of the determination in step 801 is positive, control proceeds to step 802. If the result of the determination in step 801 is negative, step 801 is repeated. In step 802 of Figure 8, the investigation unit 800 creates a record of job analysis information 700 for the data (files) that are the subject of the situation investigation request. In step 304 of Figure 3, a record of job analysis information 700 accessed from the compute management server 220 was created, but in step 802, a record of job analysis information 700 accessed from the storage management server 250 is created. In other words, with respect to job analysis information 700, there may be records accessed from the compute management server 220 and records accessed from the storage management server 250, and they may be controlled to hold generally similar information. In step 803 of Figure 8, the investigation unit 800 queries the file and object management unit 1000, which is implemented by each of the storage servers 231 forming the primary storage 230, for the status of the data (file) that is the subject of the status investigation request. The status to be queried may include the data (file) status 900, total file size 706, or stub file size 707 in the record of the job analysis information 700 in Figure 7.

[0050] Figure 9 illustrates the data (file) state 900. In the file system presented by the file and object management unit 1000 to each of the compute servers 211, the state 900 of the data (file) read during the execution of job 102 may be a non-hierarchical state 901, a cached state 902, or a stub state 903. In any state, the management information 910 for the data (file) may reside in the primary storage 230, and this management information 910 may be accessible from the file and object management unit 1000.

[0051] The non-hierarchical state 901 indicates that the data entity 920 exists only in the primary storage 230. For example, when the compute server 211 forming the computing resources 210 first creates a data entity 920 and stores it in the primary storage 230, the data may be in the non-hierarchical state 901. By using the non-hierarchical state 901, it is possible to avoid having to manage all data (files) hierarchically at all times. Cache state 902 is a state in which the data entity 920 exists in both primary storage 230 and secondary storage 240. In this case, the data entity 920 in primary storage 230 and the data entity 920 in secondary storage 240 can be said to be copies of each other. For example, by utilizing the available communication bandwidth between the primary storage 230 and the secondary storage 240 (the communication bandwidth of the data network 270), the actual data 920 of data (files) in a non-hierarchical state 901 can be transferred (destaged, backed up) from the primary storage 230 to the secondary storage 240, thereby setting the state of the data (files) to a cached state 902. At this time, information regarding directories on the file system can also be transferred (destaged, backed up) from the primary storage 230 to the secondary storage 240. If the secondary storage 240 is object storage, information regarding directories on the file system will also be recorded in a bucket-based recording area. By using cache state 902, the compute server 211 can access the actual data 920, while a backup of the actual data 920 is available, and it is possible to transition to the stub state 903 described later at any time. Stub state 903 is a state in which the actual data 920 does not exist in primary storage 230, but the actual data 920 exists in secondary storage 240. Alternatively, stub state 903 may be a state in which the actual data 920 does not exist in primary storage 230, but is treated as if the data (file) exists for file system management purposes. In the file system presented by the file and object management unit 1000, such data (file) may be referred to as "stub data (stub file)". Alternatively, invalid data 930 may be considered to exist in primary storage 230. Even in such a case, the management information 910 for the data exists in primary storage 230 and is accessible from the file and object management unit 1000. For example, if there is a long period of time during which the compute server 211 has not accessed data (files) that are in cache state 902, or if the frequency of access from the compute server 211 decreases, the file and object management unit 1000 may invalidate the actual data 920 in the primary storage 230 to make it invalid data 930, and change the state 900 of the data (files) to stub state 903. Alternatively, if the free space in the primary storage 230 falls below a predetermined threshold, the file and object management unit 1000 may invalidate the actual data 920 in the primary storage 230 to make it invalid data 930, and set the data (file) state 900 to stub state 903. While the use of the stub state 903 may not fully accommodate immediate access from the compute server 211 to the actual data 920, it still allows the compute server 211 to recognize the existence of the data (file). In order to transition the data (file) state 900 from stub state 903 to cache state 902, enabling the compute server 211 to read the actual data 920, it is necessary to transfer (stage, prefetch) the actual data 920 from secondary storage 240 to primary storage 230. This transfer time is called the prefetch time 708. In some cases, when starting to manage a certain piece of data (file) in the file system provided by the file and object management unit 1000, the state 900 of the data (file) may be set to a stub state.

[0052] The data (file) may consist of multiple parts, and a data (file) state 900 may be set for each part. For example, one part of a single data (file) may be in a cache state 902, and the remaining part in a stub state 903. In this case, the state of the data (file) as a single data (file) may be referred to as "partially stub state". For example, a "partially stub state" may occur when a data (file) that was in a stub state 903 is being transferred (staged, pre-fetched) from secondary storage 240 to primary storage 230. The stub file size 707 for data (file) in a "partially stub state" may be defined by the actual data 920 of the part of the data (file) that is in a stub state 903.

[0053] In step 803 of Figure 8, the investigation unit 800 inquires with the file and object management unit 1000 about the status of the data (file) that is the subject of the status investigation request. In response to this inquiry, the file and object management unit 1000 recognizes the inquiry in step 1801 of Figure 10. Figure 10 shows a flowchart of the processing of the file and object management unit 1000, which is a functional unit implemented by each of the storage servers 231 that form the primary storage 230. Each step shown in Figure 10 may be considered to constitute a "file and object management step". In step 1801 of Figure 10, the file and object management unit 1000 determines whether there is an inquiry from the storage management server 250 (or its investigation unit 800) regarding the status of data (files). If an inquiry is generated from the investigation unit 800 in step 803 of Figure 8, the file and object management unit 1000 recognizes the inquiry in step 1801. If the determination result in step 1801 is positive, control proceeds to step 1802. If the determination result in step 1801 is negative, step 1801 is repeated. In step 1802 of Figure 10, the file and object management unit 1000 grasps the status of the data (file) that was queried in step 1801. To grasp the status of the data (file), the file and object management unit 1000 may, for example, investigate configuration files managed by the file and object management unit 1000. The status of the data (file) investigated in step 1802 may be, for example, the data (file) status 900, the total file size 706, or the stub file size 707 in Figure 7. The status may also include the pre-fetch execution status 709. After step 1802, control of the file and object management unit 1000 transitions to step 1807. In step 1807 of Figure 10, the file and object management unit 1000 sends (responds to) the storage management server 250 (or its investigation unit 800) information regarding the status of the data (files) obtained in step 1802. After step 1807, control of the file and object management unit 1000 is returned to step 1801.

[0054] In step 1807 of Figure 10, the File and Object Management Unit 1000 transmits information regarding the status of the data (file) that is the subject of the status investigation request to the Investigation Unit 800. In response to this, the Investigation Unit 800 recognizes the receipt of the information regarding the status in step 804 of Figure 8. In step 804 of Figure 8, the investigation unit 800, which is a functional unit of the storage management server 250, determines whether it has received status information, which is the answer to the query in the most recent step 803, from the storage server 231 (which is a functional unit of the file and object management unit 1000) that forms the primary storage 230. If the result of the determination in step 804 is positive, control proceeds to step 805. If the result of the determination in step 804 is negative, step 804 is repeated. In step 805 of Figure 8, the investigation unit 800 obtains performance information regarding the transfer of data (files) from the secondary storage 240 to the primary storage 230. The performance information may be, for example, the catalog transfer speed of the line used for the transfer (data network 270 in Figure 2; in the second embodiment described later, the wide area network 1770 (WAN) in Figure 17), or the transfer speed of the line measured most recently. The processing in step 805 does not have to be performed every time the control reaches step 805, but may be performed once every few times, or only once at the beginning. In step 806 of Figure 8, the investigation unit 800 selects one of the data files that were the subject of the investigation request in the most recent step 803 and that correspond to the data files that received information about the situation in the most recent step 804. In step 807 of Figure 8, the investigation unit 800 determines whether the state 900 of the data (file) selected in the most recent step 806 is such that even a portion of the data (file) does not exist in the primary storage 230, but the actual data 920 exists in the secondary storage 240. The investigation unit 800 may, for example, determine whether the state 900 of the data (file) is in a "partially stub state" or a "stub state". If the determination result in step 807 is positive, control proceeds to step 808. If the determination result in step 807 is negative, control proceeds to step 809 (skipping step 808). In step 808 of Figure 8, the investigation unit 800 determines the stub file size 707, which is the size of the actual data 920 in the portion of the data (file) selected in the most recent step 806 where the actual data 920 does not exist in the primary storage 230, but does exist in the secondary storage 240. The investigation unit 800 then calculates or determines the pre-fetch time 708, which is the time required to transfer (stage, pre-fetch) the actual data 920, which has a stub file size of 707, from the secondary storage 240 to the primary storage 230. The investigation unit 800 may, for example, use the value obtained by dividing the stub file size 707 by the transfer speed of the line used for data transfer from the secondary storage 240 to the primary storage 230 (data network 270 in Figure 2; in the second embodiment described later, the wide area network 1770 (WAN) in Figure 17) which was determined in step 805, as the pre-fetch time 708. After step 808, control transitions to step 809. In step 809 of Figure 8, the investigation unit 800 determines whether all of the data (files) that were the subject of the situation investigation request in the most recent step 803 and that correspond to the situation information received in the most recent step 804 have been selected in step 806. If the result of the determination in step 809 is positive, control proceeds to step 810. If the result of the determination in step 809 is negative, control returns to step 806, and one of the data (files) that has not yet been selected is newly selected.

[0055] In step 810 of Figure 8, the investigation unit 800 reflects the status information received in the most recent step 804 and the lookup time 708 calculated or determined in step 808 into the job analysis information 700 record accessible from the storage management server 250. In step 811 of Figure 8, the investigation unit 800 transmits the status information received in the most recent step 804 and the lookup time 708 calculated or determined in step 808 to the compute management server 220 (its job analysis unit 300) as investigation results information corresponding to the inquiry received in the most recent step 801. After step 811, control of the investigation unit 800 is returned to step 801.

[0056] In step 811 of Figure 8, the investigation unit 800 transmits information regarding the status of the data (file) to the job analysis unit 300. In response to this transmission, the job analysis unit 300 recognizes the receipt of the investigation results in step 306 of Figure 3. In step 306 of Figure 3, the job analysis unit 300 determines whether it has received information from the storage management server 250 (or its investigation unit 800) regarding the results of the investigation into the status of each data (file), which is the response to the request for investigation into the status of each data (file) performed in the most recent step 305. If the determination result in step 306 is positive, control proceeds to step 307. If the determination result in step 306 is negative, step 306 is repeated. In step 307 of Figure 3, the job analysis unit 300 reflects the information on the data (file) status received in the most recent step 306 into the record of the job analysis information 700 that the compute management server 220 can access. Steps 810 in Figure 8 and 307 in Figure 3 ensure that the two job analysis information records 700 contain generally similar information. After step 307, control of the job analysis unit 300 is returned to step 301.

[0057] 2-1-3-2. Look-ahead request unit and look-ahead management unit (Figures 11-12) This section describes the functional configuration (processing) realized through the collaboration of the pre-fetch request unit 1100, a functional unit implemented in the compute management server 220; the pre-fetch management unit 1200, a functional unit implemented in the storage management server 250; and the file and object management unit 1000, a functional unit implemented in the primary storage 230. The information used in this functional configuration (processing) is also described. The functional configuration, processing, and information described below allow the Job Analysis Unit 300 to pre-fetch from secondary storage 240 to primary storage 230 for the portion of the data (files) identified by the Job Analysis Unit 300 for the Job 102 that does not exist in primary storage 230 but does exist in secondary storage 240. In this way, pre-fetching of data (files) is started before the execution of the Job 102 begins, which increases the likelihood that the actual data (files) will be present in primary storage 230 when the computing resource 210 (compute server) actually uses the data (files) when executing the Job 102, compared to when pre-fetching is started during the execution of the Job 102. In other words, it is expected that the likelihood of waiting occurring when the computing resource 210 (compute server) executes the Job 102 will decrease, and waiting times will be reduced.

[0058] Figure 11 shows a flowchart of the processing of the pre-fetch request unit 1100, which is a functional unit implemented in the compute management server 220. Each step shown in Figure 11 may be considered to constitute a "pre-fetch request step". In step 1101 of Figure 11, the pre-read request unit 1100 determines whether there is a job 102 recognized by the scheduler unit 221 that has not yet been assigned to any of the compute servers 211 that make up the computing resources 210 (an unassigned job). If the result of the determination in step 1101 is positive, control proceeds to step 1102. If the result of the determination in step 1101 is negative, step 1101 is repeated. In step 1102 of Figure 11, the pre-read request unit 1100 determines whether there is a job 102 among the unassigned jobs recognized in the most recent step 1101 that has not been the subject of a pre-read request in step 1104 described below. If the result of the determination in step 1102 is positive, control transitions to step 1103. If the result of the determination in step 1102 is negative, control of the pre-read request unit 1100 returns to step 1101. In step 1103 of Figure 11, the pre-read request unit 1100 identifies one job 102 from among the jobs 102 recognized in step 1102 that have never been the subject of a pre-read request. The pre-read request unit 1100 may, for example, identify a job 102 with a relatively high priority based on the order of unassigned jobs according to their assignment priority to the compute server 211. Alternatively, the pre-read request unit 1100 may identify multiple jobs 102 in step 1103. In step 1104 of Figure 11, the pre-fetch request unit 1100 requests the pre-fetch management unit 1200 of the storage management server 250 to perform pre-fetching for each of the data (files) that will be read during the execution of the job 102 identified in the most recent step 1103. Pre-fetching means that for each of the data (files) that will be read during the execution of the identified job 102, the data 920 that does not exist in the primary storage 230 but does exist in the secondary storage 240 is transferred (staged, pre-fetched) from the secondary storage 240 to the primary storage 230.

[0059] In step 1104 of Figure 11, a request for execution of a pre-read request is made from the pre-read request unit 1100 to the pre-read management unit 1200. In response to this request, the pre-read management unit 1200 recognizes the pre-read execution request in step 1201 of Figure 12. Figure 12 shows a flowchart of the processing of the pre-read management unit 1200, which is a functional unit implemented in the storage management server 250. Each step shown in Figure 12 may be considered to constitute a "pre-read management step". In step 1201 of Figure 12, the pre-read management unit 1200 determines whether it has received a request to perform a pre-read from the compute management server 220 (or its pre-read request unit 1100). If the result of the determination in step 1201 is positive, control proceeds to step 1202. If the result of the determination in step 1201 is negative, step 1201 is repeated. In step 1202 of Figure 12, the pre-read management unit 1200 determines the available storage space in each of the storage servers 231 forming the primary storage 230 that can be used to store data (files) that are scheduled to be transferred from the secondary storage 240. The pre-read management unit 1200 then determines the size of the available space (available size (A)). In Figure 9, the area where the actual data 920 is stored is not considered available space, whereas the area where invalid data 930 exists may be considered available space. In step 1203 of Figure 12, the pre-read management unit 1200 identifies data (files) that are in cache state 902 but can be transitioned to stub state 903 in each of the storage servers 231 that make up the primary storage 230. The pre-read management unit 1200 then identifies the data size (stub-able size (B)) that the actual data 920 of such data (files) occupies in the primary storage 230. Here, data (files) that are in cache state 902 but can transition to stub state 903 may be, for example, data (files) that are accessed relatively less frequently from the compute server 211 than other data (files). As data (files) with relatively low access frequency, for example, the least frequently used (LFU) data (file) may be identified. Alternatively, data (files) that are in cache state 902 but can transition to stub state 903 may be, for example, data (files) whose most recent access time from compute server 211 is relatively old. As data (files) with a relatively old most recent access time, for example, the data (file) with the oldest most recent access time (Least Recently Used (LRU)) may be identified. In executing step 1203, the pre-read management unit 1200 may send and receive necessary information with the file and object management unit 1800 of the primary storage 230.

[0060] In step 1204 of Figure 12, the pre-read management unit 1200 compares the size of the data (file) that was the target of the pre-read execution request received in step 1201 with the size of the area in the primary storage 230 that can store the pre-read data (file). The pre-read management unit 1200 calculates the size of the area (A+B) in which the pre-read data (file) can be stored, for example, as the sum of the free size (A) identified in step 1202 and the stub-able size (B) identified in step 1203. Then, the pre-read management unit 1200 calculates the size of the area to be reserved (S+α), which is the sum of the stub file size 707(S) and the margin size (α) for the data (file) that was the subject of the pre-read execution request received in step 1201. Then, the pre-read management unit 1200 determines the relationship between the size of the area where the pre-read data (file) can be stored (A+B) and the size of the area to be allocated (S+α). If the result of the determination in step 1204 is that the size of the area where the pre-read data (file) can be stored (A+B) is larger than the size of the area to be allocated (S+α), the control proceeds to step 1205. Otherwise, the control proceeds to step 1206.

[0061] In step 1205 of Figure 12, the pre-fetch management unit 1200 controls the transfer (staging, pre-fetch) of all the actual data 920 of the stub state 903 portion of the data (file) that was the target of the pre-fetch execution request in the most recent step 1201 from the secondary storage 240 to the primary storage 230. The pre-fetch management unit 1200 may instruct the file and object management unit 1800 of the primary storage 230 to start the transfer (staging, pre-fetch). In this case, the pre-read management unit 1200 may, if necessary, instruct the file and object management unit 1800 of the primary storage 230 to move data (files) that are in cache state 902 in step 1203 but have been determined to be eligible to move to stub state 903 to stub state 903. The processes described in steps 1202, 1203, 1204, and 1205 confirm that all actual data 920 of the stub state 903 portion of the data (file) targeted for prefetching can be stored in the primary storage 230 before prefetching begins. This enhances the security of prefetching control. After step 1205, control transitions to step 1207.

[0062] In step 1206 of Figure 12, the pre-fetch management unit 1200 controls the transfer (staging, pre-fetching) of the actual data 920 from the secondary storage 240 to the primary storage 230, which is within the size range (A+B) of the area where the pre-fetched data (file) can be stored, as identified in step 1204, from the stub state 903 portion of the data (file) that was the target of the pre-fetch execution request in the most recent step 1201. By controlling it as described above, even if it is not possible to immediately store all the actual data 920 of the stub state 903 portion of the data (file) that is the target of the prefetch request in the primary storage 230, it is possible to ensure that as much of the data (file) as possible has actual data 920 in the primary storage 230. The manner in which the start of the transfer (staging, prefetching) is instructed, and the manner in which the system is instructed to transition to the stub state 903, may be the same as in step 1205 (except for the size of the data (file) to which the prefetching instruction is applied).

[0063] Alternatively, in a modified version of step 1206, if step 1206 is reached, the look-ahead management unit 1200 may abandon all look-ahead operations corresponding to the look-ahead execution requests received in the most recent step 1201. This modification simplifies the control.

[0064] After step 1206, control transitions to step 1207. In step 1207 of Figure 12, the pre-read management unit 1200 reflects the results of the pre-read execution, or the results of the intermediate steps of the pre-read execution, in the record of the job analysis information 700 accessible from the storage management server 250, following the instruction to start the pre-read execution in step 1205 or step 1206. The pre-read management unit 1200 may update one or more of the information in Figure 7, such as the data (file) status 900, stub file size 707, pre-read duration 708, or pre-read execution status 709. Specifically, the stub file size 707 and pre-read duration 708 may decrease as a result of the pre-read execution. In step 1208 of Figure 12, the look-ahead management unit 1200 transmits the information of the result of the look-ahead execution, or the information of the intermediate results of the look-ahead, used in step 1207, to the compute management server 220 (specifically, the look-ahead request unit 1100). After step 1208, control of the look-ahead management unit 1200 is returned to step 1201.

[0065] In step 1208 of Figure 12, the pre-read management unit 1200 transmits information about the result of the pre-read execution, or information about the progress of the pre-read, to the pre-read request unit 1100. In response to this transmission, the pre-read request unit 1100 recognizes the receipt of the said result information in step 1105 of Figure 11. In step 1105 of Figure 11, the pre-read request unit 1100 determines whether it has received information from the storage management server 250 (or its pre-read management unit 1200) regarding the result of the pre-read execution, or information regarding the intermediate results of the pre-read, corresponding to the pre-read execution request in the most recent step 1104. If the determination result in step 1105 is positive, control proceeds to step 1106. If the determination result in step 1105 is negative, step 1105 is repeated. In step 1106 of Figure 11, the look-ahead request unit 1100 reflects the information received in step 1105, either the result of the look-ahead execution or the intermediate result of the look-ahead, into the record of the job analysis information 700 that is accessible from the compute management server 220. Steps 1207 in Figure 12 and 1106 in Figure 11 ensure that the contents of the two job analysis information 700s are generally the same. After step 1106, control of the look-ahead request unit 1100 is returned to step 1101.

[0066] 2-1-3-3. Job assignment and sorting section (Figures 13-15) This section describes the functional configuration (processing) realized by the collaboration of the job assignment unit 1300 and the sorting unit 1500 (or 1600), which are functional units implemented on the compute management server 220. It also describes the information used in this functional configuration (processing). Based on the functional configuration, processing, and information described below, the computing resource 210 (compute server) can preferentially allocate jobs 102 in which the stub file size 707 is relatively small, which is the size of the portion of the actual data 920 that the computing resource 210 (compute server) reads when executing a job 102 that does not exist in the primary storage 230 but exists in the secondary storage 240, or jobs 102 in which the pre-read time 708 for the data of said stub file size 707 is relatively short. By controlling the allocation of job 102 as described above, it is possible to increase the likelihood that the actual data 920 exists in the primary storage 230 at the time the computing resource 210 (compute server) actually uses the data (file) when executing job 102. In other words, it is expected that the possibility of waiting occurring when the computing resource 210 (compute server) executes job 102 will decrease, and a reduction in waiting time can be expected.

[0067] Figure 13 shows a flowchart of the processing of the job assignment unit 1300, which is a functional unit implemented in the compute management server 220. Each step shown in Figure 13 may be considered to constitute a "job assignment step". In step 1301 of Figure 13, the job assignment unit 1300 determines whether there is a job 102 recognized by the scheduler unit 221 that has not yet been assigned to any of the compute servers 211 that make up the computing resources 210 (an unassigned job). If the result of the determination in step 1301 is positive, control proceeds to step 1302. If the result of the determination in step 1301 is negative, step 1301 is repeated.

[0068] In step 1302 of Figure 13, the job assignment unit 1300 checks the scheduling policy, which is a guideline for assigning job 102 to each of the compute servers 211 that make up the computing resources 210. The job assignment unit 1300 may determine the currently applied scheduling policy by, for example, checking the type of scheduling policy described in the scheduling policy setting file 1400. Figure 14 shows a scheduling policy configuration file 1400. As shown in Figure 14, the scheduling policy configuration file 1400 may contain the name of the scheduling policy type itself. (The scheduling policy configuration file 1400 may also contain a number or symbol corresponding to the scheduling policy type.) The top of Figure 14 shows an example where "FIFO" (FIFO policy) is described as the name of the scheduling policy type. The bottom of Figure 14 shows an example where "Cached Job Priority" (Cached Job Priority policy) is described as the name of the scheduling policy type. Other scheduling policy types besides "FIFO" and "Cached Job Priority" may also exist. The "FIFO" scheduling policy type indicates that the scheduler unit 221 will assign jobs 102 to the compute server 211 in the order in which it receives them. On the other hand, the "cached job priority" scheduling policy type indicates that when the compute server 211 reads data (files) for executing jobs 102, jobs 102 in which the size of the data 920 is relatively small in the portion where the actual data 920 exists in the secondary storage 240 but not in the primary storage 230 will be preferentially assigned to the compute server 211. In Figure 13, it appears as though the job assignment unit 1300 determines the type of scheduling policy each time the control transitions to step 1302. However, the job assignment unit 1300 may determine the type of scheduling policy only once when it is started (or when the scheduler unit 221 is started). Alternatively, the job assignment unit 1300 may determine the type of scheduling policy once each time the control transitions to step 1302 a predetermined number of times. Alternatively, the job assignment unit 1300 may determine the type of scheduling policy when it receives explicit instructions from a user of the management system 101. If the result of the determination in step 1302 is that the scheduling policy type is "FIFO", then (skipping steps 1303 and 1304,) control proceeds to step 1305. If the result of the determination in step 1302 is that the scheduling policy type is "Cached Jobs First", then control proceeds to step 1303.

[0069] In step 1303 of Figure 13, the job assignment unit 1300 requests the sorting unit 1500 (or 1600) to sort the unassigned jobs in order to implement the scheduling policy of "cached jobs first". Sorting the unassigned jobs means rearranging each unassigned job in order of priority (or the order in which they will be assigned) to be assigned to one of the compute servers 211 that make up the computing resources 210.

[0070] In step 1303 of Figure 13, the job assignment unit 1300 requests the sorting unit 1500 to sort the unassigned jobs, and in step 1601 of Figure 15 (or step 1601 of Figure 16), the sorting unit 1500 recognizes that a request for job sorting has been made. Figure 15 shows a flowchart of the processing of the sorting unit 1500, which is implemented as a functional unit in the compute management server 220. Each step shown in Figure 15 may be considered to constitute a "sorting step". In step 1601 of Figure 15, the sorting unit 1500 determines whether it has received a request from the job assignment unit 1300 to sort unassigned jobs. If the result of the determination in step 1601 is positive, control proceeds to step 1602. If the result of the determination in step 1601 is negative, step 1601 is repeated. In step 1602 of Figure 15, the sorting unit 1500 determines, for each unassigned job that is subject to sorting, the stub file size 707, which is the data size of the portion of the data (file) to be read when the unassigned job is executed that does not exist in the primary storage 230 but does exist in the secondary storage 240. If there are multiple data (files) to be read for a single unassigned job, the sorting unit 1500 may determine the sum of the stub file sizes 707 for each of the data (files) to be read. The sorting unit 1500 then sorts the unassigned jobs in descending order of the total stub file size of each unassigned job (707). In other words, the sorting unit 1600 assigns a higher priority (earlier order) to unassigned jobs with a smaller total stub file size (707) when assigning them to the compute server 211. Here, the sorting unit 1500 may also sort the records of the job analysis information 700 that are accessible from the compute management server 220, so as to reflect the sorting of unassigned jobs. After step 1602, control transitions to step 1606.

[0071] In step 1606 of Figure 15, the sorting unit 1500 may adjust the priority or order of unassigned jobs after the sorting in step 1602 to take into consideration unassigned jobs that have been waiting for an excessively long time to be allocated to the compute server 211 that forms the computing resources 210. Specifically, the sorting unit 1600 identifies unassigned jobs whose elapsed time from the job registration time 702 (elapsed time waiting for allocation) is greater than or equal to a threshold (T). The sorting unit 1500 adjusts the priority (or order) of the identified unassigned jobs to either increase their allocation priority or move them earlier in the allocation order. Here, the sorting unit 1500 may also sort the records of the job analysis information 700 that are accessible from the compute management server 220, in order to reflect any adjustments to the priority or order of unassigned jobs. The process in step 1606 prevents the occurrence of jobs 102 that would wait for an excessively long time to be allocated to the compute server 211 that forms the computing resources 210 due to some reason (for example, a large total stub file size of 707, a long pre-fetch time of 708, etc.).

[0072] In step 1607 of Figure 15, the sorting unit 1500 reports to the job assignment unit 1300 that the sorting of unassigned jobs is complete. After step 1607, control of the sorting unit 1500 is returned to step 1601. In step 1607 of Figure 15, the sorting unit 1500 reports to the job assignment unit 1300 that the sorting of unassigned jobs is complete. In response, the job assignment unit 1300 recognizes in step 1304 of Figure 13 that the job sorting has been completed. In step 1304 of Figure 13, the job assignment unit 1300 determines whether the sorting unit 1500 (or 1600) has reported the completion of sorting of unassigned jobs. If the result of the determination in step 1304 is positive, control proceeds to step 1305. If the result of the determination in step 1304 is negative, step 1304 is repeated. In step 1305 of Figure 13, the job assignment unit 1300 assigns unassigned jobs to the compute servers 211 that make up the computing resources 210, either those with a high priority or an early order in the priority or sequence for assigning unassigned jobs to each of the compute servers 211 that make up the computing resources 210, and which are in a state where they can be assigned to the compute servers 211. Note that the assignment of unassigned jobs to the compute servers 211 in step 1305 may be performed for one or more unassigned jobs in each step 1305. After step 1305, control of the job assignment unit 1300 is returned to step 1301.

[0073] 2-1-3-4. Modified versions of the rearrangement section (Figures 13, 14, and 16) Of those described in "2-1-3-3. Job Assignment Unit and Sorting Unit" above, the sorting unit 1500 shown in the processing flowchart of Figure 15 may be replaced with the modified sorting unit 1600 shown in the processing flowchart of Figure 16, which will be described below. The modified sorting unit will be described below. The functional configuration, processing, and information described below provide similar effects to those of the sorting unit 1500. In addition, the modified sorting unit 1600 can adjust the order in which jobs 102 are assigned to the computing resource 210 (compute server) such that the pre-read time 708 for each job 102 is less than the sum of the job execution times 703 required to execute a predetermined number of jobs scheduled to run immediately before that job 102. For example, the assignment order of jobs 102 can be rearranged to delay the assignment of jobs 102 with relatively long pre-read times 708 to the computing resource 210 (compute server). Thus, the modified sorting unit 1600, by taking into account the job execution time 703 for each unassigned job, can further reduce the possibility of delays occurring when the compute server 211 reads data (files) when executing job 102, or shorten the time required for delays.

[0074] Figure 16 shows a flowchart of the processing of a modified sorting unit 1600 implemented as a functional unit of the compute management server 220. Each step shown in Figure 16 may be considered to constitute a "sorting step". Of the processes shown in Figure 16, steps 1601, 1602, 1606, and 1607 are the same as the steps with the same names shown in Figure 15 (except that the execution entity is the modified sorting unit 1600), so their explanation will be largely omitted. After step 1602 in Figure 16, the control transitions to step 1603. In step 1603 of Figure 16, the modified sorting unit 1600 determines the job execution time 703 for each unassigned job. The modified sorting unit 1600 may determine the job execution time 703 for each unassigned job based on the job execution time 703 item in the job analysis information 700 record. Alternatively, the modified sorting unit 1600 may obtain information on the job execution time 703 for each unassigned job when it was previously executed from a source other than the job analysis information 700 record. Alternatively, the modified sorting unit 1600 may determine the amount of resources (GPU (time resources), memory, etc.) of the compute server 211 that are scheduled to be allocated to each unassigned job, and then calculate the job execution time 703 using a model equation between the amount of said resources and the job execution time 703.

[0075] In step 1604 of Figure 16, the modified sorting unit 1600 determines, for each unassigned job, the total pre-read time of 708 (P) corresponding to the total stub file size of 707 for each data (file) to be read when the unassigned job is executed. The modified sorting unit 1600 then determines, in the sorting order based on the priority or assignment order of unassigned jobs determined in step 1602, which of the predetermined number of unassigned jobs immediately preceding each unassigned job. If there are multiple compute servers 211 forming the computing resources 210, the system may determine, for each compute server 211, which of the predetermined number of unassigned jobs immediately preceding each unassigned job. Furthermore, the modified sorting unit 1600 determines, for each unassigned job, the total job execution time 703 (E) of a predetermined number of unassigned jobs immediately preceding the unassigned job. In step 1605 of Figure 16, the modified sorting unit 1600 determines, for each unassigned job, whether the total look-ahead time 708 (P) for that unassigned job is less than the total job execution time 703 (E) for a predetermined number of unassigned jobs immediately preceding that unassigned job. If the result of this determination is negative, the modified sorting unit 1600 adjusts the priority or order of the unassigned jobs so that the result is as positive as possible. Here, the modified sorting unit 1600 may also sort the records of the job analysis information 700 that are accessible from the compute management server 220, in order to reflect any adjustments to the priority or order of unassigned jobs. After step 1605, control transitions to step 1606. In the modified sorting section 1600, step 1606 handles the priority or order of the unassigned jobs after the adjustments made in step 1605.

[0076] 2-2. Second Embodiment The second embodiment enables jobs using data (files) located at another site 1799 (for example, jobs that analyze data (files)) to be executed at a site where computing resources 210 and primary storage 1730 (and further, compute management server 220 and storage management server 1750) are located. In the second embodiment, for example, computing resources 210 and primary storage 1730 (and further, compute management server 220 and storage management server 1750) may be located at the same site, while secondary storage 1740 (and storage management server 1790) may be located at a different site (another site 1799). Secondary storage 1740 may be primary storage within the other site 1799, or it may be secondary storage within the other site 1799. The site may be, for example, a data center. The following section primarily describes the differences between the second embodiment and the first embodiment. Points that are the same as those in the first embodiment may be omitted from the explanation.

[0077] 2-2-1. Overall configuration of the second embodiment (Figure 17) Figure 17 shows the overall configuration 200 of the second embodiment of this disclosure. Note that not all of the functional configurations (and the information handled) shown in Figure 17 are essential. Furthermore, the existence of functional configurations other than those shown in Figure 17 is not prohibited.

[0078] The main difference between the second embodiment shown in Figure 17 and the first embodiment shown in Figure 2 is that in the second embodiment, the location where the computing resources 210 and primary storage 1730 are located is different from the location where the secondary storage 1740 is located. In Figure 17, the location where the secondary storage 1740 is located is referred to as separate location 1799. At site 1799, there are one or more (S in Figure 17) storage servers 1741 that form secondary storage 1740. Additionally, site 1799 has a storage management server 1790 for site 1799. In the second embodiment, the file and object management unit 1743 implemented in the secondary storage 1740 has the same functions as the file and object management unit 243 implemented in the secondary storage 240 in the first embodiment. However, in the second embodiment, as the wide area network 1770 (WAN) is used for mutual communication between the primary storage 1730 and the secondary storage 1740, the file and object management unit 1743 (and the file and object management unit 1800 on the primary storage 1730 side) may have functions that support mutual communication using the wide area network 1770 (WAN). The communication speed of a Wide Area Network 1770 (WAN) is arbitrary, but for example, it can be around 10 gigabits per second.

[0079] Instead of the storage management unit 251 in Figure 2, the storage management server 1750 for the site where the computing resources 210 and primary storage 1730 are located has a hybrid storage management unit 1751, and the storage management server 1790 for another site 1799 has a hybrid storage management unit 1791. By having the hybrid storage management units for different sites cooperate with each other, storage management across the wide area network 1770 (WAN) is realized.

[0080] In the second embodiment, since the location where the secondary storage 1740 is located is different from the location where the compute server 211 and primary storage 1730 are located, the data (file) in which the actual data 920 does not exist in the primary storage 1730 but exists in the secondary storage 1740 (or is expected to exist in the secondary storage 1740) is not necessarily effectively managed by the file and object management unit 1800 implemented in the primary storage 1730 from the outset. To address the above circumstances, in the second embodiment, the file and object management unit 1800 (and its internal functional unit, the virtualization and hierarchical control unit 1732), implemented in the primary storage 1730, is capable of performing processing to newly and effectively manage files corresponding to data at another site 1799 in the file system provided at the local site. Details of the functions (processing) and information of the file and object management unit 1800 implemented in the primary storage 1730 in the second embodiment will be described in "2.2.2. Functional configuration, processing, and information of the second embodiment" below.

[0081] 2-2-2. Functional configuration, processing, and information of the second embodiment The following describes the functional configuration, processing, and information of the management system 101 in the second embodiment. The description in section 2-1-3-1. Job Analysis Unit and Investigation Unit in the first embodiment generally applies to the second embodiment as well, except for the description of the processing of the file and object management unit 1000, which is a functional unit implemented in the primary storage 230 using Figure 10. Therefore, the following section 2-2-2-1. File and Object Management Unit in the Second Embodiment mainly describes the processing of the file and object management unit 1800, which is a functional unit implemented in the primary storage 1730 in the second embodiment, using Figure 18. The description in section 2-1-3-2. Pre-fetch Request Unit and Pre-fetch Management Unit in the first embodiment generally applies to the second embodiment as well. However, in the second embodiment, when transferring data (files) from the secondary storage 1740 to the primary storage 1730 (staging, pre-fetching), the wide area network 1770 (WAN) is used as the transfer path. Therefore, the primary storage 1730 and secondary storage 1740 in the second embodiment will perform processing that corresponds to the wide area network 1770 (WAN). The descriptions in sections "2-1-3-3. Job Assignment Unit and Sorting Unit" and "2-1-3-4. Modified Sorting Unit" in the first embodiment generally apply to the second embodiment as well.

[0082] 2-2-2-1. File and object management unit in the second embodiment (Figure 18) This section describes the functional configuration (processing) realized in the second embodiment by the collaborative efforts of the job analysis unit 300, a functional unit implemented in the compute management server 220; the investigation unit 800, a functional unit implemented in the storage management server 1750; and the file and object management unit 1800, a functional unit implemented in the primary storage 1730. The information used in this functional configuration (processing) is also described. The functional configuration, processing, and information described below provide the same effects as those of the first embodiment. In addition, even if the primary storage 1730 does not initially effectively manage data (files) where the data entity 920 does not exist in the primary storage 1730 but exists in the secondary storage 1740 (or is expected to exist in the secondary storage 1740), the file corresponding to that data can be newly and effectively managed on the file system. Therefore, even in a system configuration where the primary storage 1730 and the secondary storage 1740 are located at different locations, data (files) can be pre-fetched using the same or similar control as in the first embodiment.

[0083] In the second embodiment, when the job analysis unit 300, the investigation unit 800, and the file and object management unit 1800 cooperate, the processing content of the job analysis unit 300 and the investigation unit 800 may be the same as in the first embodiment. In other words, in the second embodiment as well, the job analysis unit 300 may be based on the processing flowchart shown in Figure 3, and the investigation unit 800 may be based on the processing flowchart shown in Figure 8.

[0084] Figure 18 shows a flowchart of the processing of the file and object management unit 1800, which is a functional unit implemented by each of the storage servers 1731 forming the primary storage 1730 in the second embodiment. Each step shown in Figure 18 may be considered to constitute a "file and object management step". Of the processing shown in Figure 18, steps 1801, 1802, and 1807 are the same as in Figure 10 (except that the execution entity is the file and object management unit 1800), so their explanation is generally omitted. Furthermore, among the data (files) for which an inquiry was made regarding the status of the data (files) in the most recent step 1801, there may be some for which the file path 705 for that data (file) is not yet effectively managed in the file system provided by the file and object management unit 1800, as shown in step 1802 of Figure 18.

[0085] After step 1802 in Figure 18, the control transitions to step 1803. In step 1803 of Figure 18, the file and object management unit 1800 selects one of the data (files) that are the subject of the investigation request for which an inquiry about the status of the data (file) was made in the most recent step 1801. In step 1804 of Figure 18, the file and object management unit 1800 determines whether the file path 705 setting for the data (file) selected in the most recent step 1803 is effectively managed in the file system provided by the file and object management unit 1800 (the local file system). The file and object management unit 1800 determines, for example, whether it holds valid management information 910 corresponding to the file path 705 in a configuration file under the file and object management unit 1800. If the result of the determination in step 1804 is positive, (skipping step 1805,) control proceeds to step 1806. If the result of the determination in step 1804 is negative, control proceeds to step 1805. In step 1805 of Figure 18, the file and object management unit 1800 determines that the setting of the file path 705 for the data (file) selected in the most recent step 1803 should be effectively managed in the file system provided by the file and object management unit 1800 (the local file system). For example, if the actual data 920 related to the data (file) selected in the most recent step 1803 exists in the secondary storage 1740 (or is expected to exist in the secondary storage 1740), the file and object management unit 1800 sets the status 900 of the data (file) to "stub state" in the management information 910 for the file path 705, and stores information regarding the location of the actual data 920 in the management information 910. After step 1805, control transitions to step 1806. Furthermore, an example of data (files) whose actual data 920 is scheduled to reside in secondary storage 1740 is big data that could be used as training data for machine learning, which is scheduled to be stored in secondary storage 1740 from its source. In step 1806 of Figure 18, the file and object management unit 1800 determines whether all of the data (files) that were the subject of the investigation request for which the status of the data (files) was queried in the most recent step 1801 have been selected in step 1803. If the result of the determination in step 1806 is positive, control proceeds to step 1807. If the result of the determination in step 1806 is negative, control returns to step 1803, and one of the data (files) that has not yet been selected is newly selected.

[0086] 3. Other (Variations) This disclosure is not limited to the embodiments described above and includes various modifications. Some of the configurations and processes of the embodiments may be replaced with configurations and processes of other conceivable embodiments. Configurations and processes of other conceivable embodiments may be added to the configurations and processes of the embodiments. For example, the following modifications of the embodiments may be made in this disclosure.

[0087] (Variation A) Server Integration In the embodiments described above, each of the compute servers 211 forming the computing resources 210, each of the storage servers 231 (or 1731) forming the primary storage 230 (or 1730), each of the storage servers 241 (or 1741) forming the secondary storage 240 (or 1740), a compute management server 220, and a storage management server 250 (or 1750) were shown. In variation A, some of the above-mentioned servers may be integrated in hardware. For example, the compute management server 220 may not exist as hardware by having one of the compute servers 211 perform the role of the compute management server 220. Alternatively, the storage management server 250 (or 1750) may not exist as hardware by having one of the storage servers 231 (or 1731) perform the role of the storage management server 250 (or 1750). According to Modification A, the system configuration of the embodiment of the present disclosure can be flexibly determined, and hardware costs can also be reduced.

[0088] (Variation B) Variation of data identification by the job analysis unit In the embodiment described above, when the computing resource 210 (compute server) is executing job 102, the job analysis unit 300 analyzes the workflow of job 102 to identify the data (file) to be read. In modified example B, the job analysis unit 300 may use a method other than the method for analyzing the workflow of job 102 (for example, a method based on access patterns in past executions of job 102 (based on historical analysis)) to identify the data (files) that the computing resource 210 (compute server) reads when executing job 102. Modification B also identifies the data (files) that the computing resource 210 (compute server) will read when executing job 102 before the computing resource 210 (compute server) starts executing job 102, and starts pre-fetching the identified data (files). Therefore, compared to the modification B, which starts pre-fetching data (files) after job 102 has started executing, modification B also increases the likelihood that the storage (primary storage) at a tier relatively close to the computing resource 210 (compute server) will hold the data (files) at the time the data (files) are used to execute job 102.

[0089] The technical matters shown in each of the embodiments and modifications of the embodiments described above can be combined as appropriate, as long as no technical inconsistencies arise.

Claims

1. A management system for managing computing resources and storage, A job analysis unit identifies the data that the computing resources will read when executing the job by analyzing the workflow of the job before starting the job execution. A management system having a pre-fetch management unit that controls the system to start pre-fetching from the secondary storage to the primary storage for data identified by the job analysis unit for the aforementioned job, where the actual data does not exist in the primary storage which has relatively high access performance from the computing resources, but the actual data exists in the secondary storage which has relatively low access performance from the computing resources.

2. A management system according to claim 1, The aforementioned job is a management system for performing processing related to models or data concerning artificial intelligence.

3. A management system according to claim 1, The job analysis unit is a management system that performs one of the following in order to identify the data to be read when the computing resources execute the job: analyzing information describing the workflow of the job, referring to the arguments of the command for calling the job, and obtaining information identifying the data to be read as described in the configuration file for the job.

4. A management system according to claim 1, The management system includes an investigation unit that investigates the status of each piece of data that the computing resource reads when executing the job, as identified by the job analysis unit, in the primary storage and the secondary storage. The status of the data investigated by the investigation unit includes the state of the data, which indicates whether the actual data resides in the primary storage or the secondary storage, and the stub file size, which indicates the size of the portion of the actual data that exists in the secondary storage but not in the primary storage. The investigation unit uses the stub file size to calculate the pre-read time required to transfer the portion of the data that exists in the secondary storage but not in the primary storage from the secondary storage to the primary storage. A management system that controls the lookup based on the aforementioned lookup time.

5. A management system according to claim 4, The set of states that the aforementioned data can be in includes at least a non-hierarchical state, a cached state, and a stub state. The aforementioned non-hierarchical state indicates that the actual data exists in the primary storage but not in the secondary storage. The cache state indicates that the actual data exists in both the primary storage and the secondary storage. The stub state is a management system that indicates that the actual data exists in the secondary storage but not in the primary storage, and that management information for the data exists in the primary storage.

6. A management system according to claim 4, The system has job analysis information which includes records for each of the data that the computing resources read when executing the job, A management system in which each of the job analysis information records holds information regarding the state of the data corresponding to the record, the stub file size of the data corresponding to the record, and the look-ahead time required for the data corresponding to the record.

7. A management system according to claim 4, The job analysis unit, the pre-read management unit, the investigation unit, the primary storage, the file and object management unit for the primary storage, and the computing resources for executing the jobs are located at the same site, while the secondary storage is located at a different site. The investigation unit queries the file and object management unit for the primary storage regarding the status of the data. A file and object management unit for the primary storage, if the file path setting for the file relating to the data that was queried is not effectively managed in the file system at the same location, configures the system to effectively manage the file path setting and sets the status of the data to a stub state.

8. A management system according to claim 1, The management system further includes a scheduler unit that allocates the job to computing resources in order to execute the job. The management system includes a scheduler unit which has a pre-fetch request unit which requests the pre-fetch management unit to start pre-fetching data to be read when executing a job for a job that has not yet been allocated to the computing resources.

9. A management system according to claim 8, The scheduler unit follows a scheduling policy when allocating the jobs to the computing resources. A management system in which one of the scheduling policies that can be used in the scheduler is a cached job priority policy which indicates a policy that preferentially allocates the computing resources to jobs in which the stub file size, which is the size of the portion of the data that is present in the secondary storage but not in the primary storage, is relatively small, or jobs in which the pre-fetch time, which is the time required to transfer the data of the stub file size from the secondary storage to the primary storage, is relatively short.

10. A management system according to claim 9, The scheduler unit includes a sorting unit that adjusts the order in which each of the jobs is allocated to the computing resources when following the cached job priority policy. The sorting unit, when following the cached job priority policy, adjusts the priority of assigning jobs to computing resources to increase the priority of assigning those jobs to computing resources if the elapsed time waiting to be allocated to computing resources is greater than or equal to a threshold.

11. A management system according to claim 9, A management system in which, when sorting a job according to the cached job priority policy, the sorting unit adjusts the order in which jobs are assigned to the computing resources such that the look-ahead time for each job is less than the total job execution time, which is the time required to execute a predetermined number of jobs scheduled to run immediately before that job.

12. A management system according to claim 1, The pre-fetch management unit identifies the size of the area of ​​the primary storage that can be used to store data pre-fetched from the secondary storage. The primary storage area that can be used to store the pre-fetched data consists of either or both of the free space in the primary storage and the area in the primary storage that stores the data whose actual data is to be invalidated. The pre-fetch management unit controls the pre-fetching of the actual data to be pre-fetched from the secondary storage to the primary storage if the size of the area of ​​the primary storage that can be used to store the data to be pre-fetched is greater than the sum of the size of the data to be pre-fetched and a predetermined margin.

13. A management system according to claim 12, The pre-fetch management unit controls the pre-fetching of only the portion of the data subject to pre-fetching that can be stored in the primary storage area that can be used to store the data to be pre-fetched, when the size of the primary storage area that can be used to store the data to be pre-fetched is less than or equal to the sum of the size of the data subject to pre-fetching and a predetermined margin, from the secondary storage to the primary storage.

14. A method performed by a management system for managing computing resources and storage, A job analysis step involves analyzing the workflow of the job before starting the execution of the job to identify the data that the computing resources will read when executing the job, A method comprising a prefetch management step that controls the initiation of prefetching from secondary storage to primary storage for data identified by the job analysis step for the aforementioned job, where the actual data does not exist in primary storage which has relatively high access performance from the computing resources, but the actual data exists in secondary storage which has relatively low access performance from the computing resources.

Images