Resource optimization system, and resource optimization method

The resource optimization system addresses inefficiencies in distributed analysis by dynamically estimating and adjusting resource allocation, optimizing task execution time and resource usage on variable platforms.

JP2024169214APending Publication Date: 2024-12-05HITACHI LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023086498
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-25
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Existing technologies for resource optimization in distributed analysis platforms fail to account for variable resource amounts and dynamic data changes, leading to inefficiencies in completion time and cost, and are limited to specific variance analysis applications.

Method used

A resource optimization system that includes an arithmetic device, memory resources, and storage devices to detect target tasks, measure input data, and estimate optimal resource amounts using a resource amount estimation model, dynamically adjusting resource allocation to minimize execution time.

Benefits of technology

The system balances total application execution time and resource usage efficiency by determining optimal resource amounts for each task, reducing processing time and enhancing resource utilization on scalable platforms.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024169214000001_ABST
    Figure 2024169214000001_ABST
Patent Text Reader

Abstract

To provide a resource optimization system capable of instantly determining the optimal amount of resources for each task while balancing the total time to complete application execution and resource utilization efficiency when processing the analysis of variance in applications, during processing, that have multiple task executions, using a distributed analysis infrastructure capable of scaling.SOLUTION: The resource optimization system includes one or more computing units, one or more memory resources, and one or more storage units. The computing units are configured to detect the target task for resource optimization processing from the source code that represents the processing performed by an application, to measure the data amount of input data for the target task, and to estimate the optimal amount of resources to reduce the execution time of the target task based on the measured data amount, and the resource amount estimation model.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a resource optimization system and a resource optimization method. [Background technology]

[0002] Regarding technology for estimating resource amounts for a distributed analysis platform, for example, Patent Document 1 states, "This discloses a method and device for resource scheduling for a distributed system. This device balances the total time and cost for completing tasks in accordance with user requests, improving resource utilization and accelerating task execution. In order to respond to user processing requests for tasks, when a new task is input to the device, the task processing request is evaluated and a priority is assigned, and the task is allocated to a scheduling queue corresponding to that priority, and distributed node resources are allocated starting from the queue with the highest priority in accordance with a preset policy. Each queue calculates an evaluation function from the waiting time and weighting coefficient (user's desired value) of the tasks it holds, and by comparing the evaluation function with other queues, the execution priority order between queues is determined."

[0003] Furthermore, Non-Patent Document 1 states, "This document discloses a method and device for optimizing resource allocation in big data analysis to reduce operational costs in a cloud-oriented system. This device calculates a performance characteristic curve that represents the relationship between resource allocation and performance (processing time) for a task in a stage prior to processing execution, and estimates the optimal resource allocation when the task is executed. Since the change in performance improvement decreases as the resource amount increases, the performance characteristic curve is considered to have an exponential decay, and the performance characteristics of each task are represented by finding an approximation line from the learning data. In order to reduce calculation costs, a unique data expansion technology is used to generate sufficient learning data from a small amount of actual data. Furthermore, in this document, in order to prevent a decrease in processing performance and an increase in waiting time, the allocation of tasks to peak resources (resources with a resource utilization rate close to 100% or close to a specified limit value) is avoided, and the maximum resource amount that optimally trades off resource cost and runtime is set as the optimal resource allocation." [Prior art documents] [Patent documents]

[0004] [Patent Document 1] China Patent Application Publication No. 108345501 [Non-patent literature]

[0005] [Non-Patent Document 1] Anish Pimpley, Shuo Li, Rathijit Sen, Soundararajan Srinivasan, Alekh Jindal, “Towards Optimal Resource Allocation for Big Data Analytics”, OpenProceedings.org, ISBN: 978-3-89318-085-7, Electronic Edition, p.338-350(2022) Summary of the Invention [Problem to be solved by the invention]

[0006] The technology described in Non-Patent Document 1 assumes that a single task is given and that the amount of input data is constant, even though distributed processing involves overhead such as transfer delays associated with data distribution, and the overhead depends on the amount of data to be analyzed. Therefore, it cannot be applied to distributed analysis applications operated in real environments. In addition, it is not possible to properly predict when the amount of data to be analyzed changes. In addition, the amount of data to be analyzed cannot be known until a request is received in the production environment.

[0007] The technology described in Patent Document 1 searches for a scheduling method that minimizes processing time using existing resources, and therefore cannot find an optimal scheduling method for a distributed analysis platform with variable resource amounts.

[0008] When the technology described in Non-Patent Document 1 and the technology described in Patent Document 1 are targeted at an analysis application that performs distributed analysis multiple times during processing, the technology that can optimize resources is limited to a specific distributed analysis.

[0009] The present invention has been made in consideration of the above-mentioned points, and aims to enable an application that has multiple task executions using a scalable distributed analysis platform to balance the total time and cost for completing application execution when performing distributed analysis processing, and to determine the optimal amount of resources for each task in a short period of time. [Means for solving the problem]

[0010] The present application includes a number of means for solving at least some of the above-mentioned problems, examples of which are as follows.

[0011] In order to solve the above-mentioned problems, a resource optimization system according to one embodiment of the present invention includes one or more computing devices, one or more memory resources, and one or more storage devices, and the computing devices detect a target task that is to be subject to a resource optimization process from a source code representing a process executed by an application, measure the amount of input data for the target task, and estimate an optimal resource amount for shortening the execution time of the target task based on the measured data amount and a resource amount estimation model. Effect of the Invention

[0012] According to the present invention, in an application that has multiple task executions using a scalable distributed analysis platform, when performing distributed analysis processing, it is possible to quickly determine for each task an optimal resource amount that achieves a good balance between the total time to complete application execution and the efficiency of resource utilization. This makes it possible to dynamically increase or decrease the resource amount of the distributed analysis platform, thereby shortening the application processing time.

[0013] Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a resource optimization system according to the first embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing an example of the hardware configuration of a computer. [Diagram 3] FIG. 3 is a diagram illustrating an example of source code of an application to which the resource optimization process is applied. [Figure 4] FIG. 4 is a diagram showing an example of the analysis data. [Diagram 5] FIG. 5 is a diagram illustrating an example of a data structure of a resource amount estimation model DB (database). [Figure 6] FIG. 6 is a flowchart illustrating an example of a source code corresponding process by an application in the first embodiment. [Figure 7] FIG. 7 is a flowchart illustrating an example of a dynamic resource optimization process. [Figure 8] FIG. 8 is a diagram illustrating an example of a configuration of a resource optimization system according to the second embodiment of the present invention. [Figure 9] FIG. 9 is a diagram illustrating an example of a data structure of the learning data DB. [Figure 10] FIG. 10 is a flowchart illustrating an example of a source code corresponding process by an application in the second embodiment. [Figure 11] FIG. 11 is a flowchart illustrating an example of a resource amount estimation model update process. [Figure 12] FIG. 12 is a diagram illustrating an example of the configuration of a resource optimization system according to the third embodiment of the present invention. [Figure 13] FIG. 13 is a diagram showing an example of sample analysis data. [Figure 14] FIG. 14 is a flowchart illustrating an example of a resource amount estimation model generation process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In all the drawings for explaining the embodiment, the same members are generally given the same reference numerals, and the repeated description thereof will be omitted. In the following embodiments, the components (including element steps, etc.) are not necessarily essential, except when specifically indicated, or when it is clearly considered essential in principle. In addition, when it is said that "consists of A", "consists of A", "has A", or "includes A", it does not exclude other elements, except when it is specifically indicated that only that element is included. Similarly, in the following embodiments, when referring to the shape, positional relationship, etc. of the components, etc., it includes those that are substantially similar or similar to the shape, etc., except when it is specifically indicated, or when it is clearly considered not to be the case in principle.

[0016] In this specification, a resource refers to a computer having a computational capacity, and a resource amount refers to the number of computers that are resources. The computers are not limited to physical computers, and may be logical computers.

[0017] <Resource optimization system 100-1 according to the first embodiment of the present invention> 1 shows an example of the configuration of a resource optimizing system 100-1 according to a first embodiment of the present invention. The resource optimizing system 100-1 optimizes the amount of resources in a distributed analysis platform and causes the distributed analysis platform to execute tasks.

[0018] The resource optimization system 100-1 includes an application 1 as a target for optimizing resource allocation, a distributed analysis platform 2, and a resource amount estimation model management device 3 and an optimal resource estimation device 4 which constitute a resource optimization unit.

[0019] The application 1, the distribution analysis platform 2, the resource amount estimation model management device 3, and the optimal resource estimation device 4 are connected to a network N and can communicate predetermined information and data with each other. The network N is, for example, a two-way communication network such as a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, or a mobile phone communication network.

[0020] The application 1, the distributed analysis platform 2, the resource amount estimation model management device 3, and the optimal resource estimation device 4 may be appropriately combined and realized by one physical or logical computer. Also, the application 1 and the distributed analysis platform 2 may be realized by two or more physical or logical computers.

[0021] The application 1, the distributed analysis platform 2, the resource amount estimation model management device 3, and the optimal resource estimation device 4 may be distributed. For example, the application 1 may be distributed locally, and the distributed analysis platform 2, the resource amount estimation model management device 3, and the optimal resource estimation device 4 may be distributed on a cloud server.

[0022] The application 1 receives a large amount of analysis data and executes an analysis process of the analysis data using the distributed analysis platform 2. The application 1 calls and uses the distributed analysis platform 2 at least two times.

[0023] The application 1 includes a target task processing unit 11, a non-target task processing unit 12, and an optimal resource estimation library 13.

[0024] The target task processing unit 11 transmits a function (target task) for which the optimal resource amount is to be estimated together with a processing command to the distributed analysis platform 2 to request its execution. The function transmitted by the target task processing unit 11 has a unique annotation (described later) to distinguish it from other functions and the like in the source code of the application 1.

[0025] The non-target task processing unit 12 executes tasks, processes, etc., other than the functions in the source code of the application 1 that the target task processing unit 11 causes the distributed analysis platform 2 to execute. However, the non-target task processing unit 12 may execute tasks and processes using the distributed analysis platform 2.

[0026] The optimal resource estimation library 13 is a group of tools equipped with functions required for estimating the optimal resource amount within an application, and performs functions such as distinguishing between target task processing units 11 and non-target task processing units 12, and controlling the behavior of the application when estimating the optimal resource amount.

[0027] In response to a processing command from an external client (such as the target task processing unit 11 of application 1), the distributed analysis platform 2 uses the computational resources available within the distributed analysis platform 2 at that time, allocates the requested processing to the available computational resources, executes the processing in parallel, and outputs the processing results to the external client.

[0028] The distributed analysis platform 2 includes a distributed analysis processing unit 21, a distributed tracing collection unit 22, and a distributed tracing information DB 23.

[0029] The distributed analysis processing unit 21 is a cluster-type computing resource consisting of one or more physical or virtual computers, etc. The distributed analysis processing unit 21 can increase or decrease the amount of computing resources depending on the processing command from an external client, the resource usage status at that time, etc.

[0030] The distributed tracing collection unit 22 monitors the distributed analysis processing unit 21 and stores monitoring information in a distributed tracing information DB 23. The distributed tracing collection unit 22 also monitors the progress of a transaction of a request when a processing command is passed to the distributed analysis processing unit 21, and observes transactions flowing through each resource in the distributed analysis processing unit 21. The distributed tracing information DB 23 stores monitoring information and transaction information of each resource in the distributed analysis processing unit 21. The monitoring information and transaction information are used to optimize the resources of the distributed analysis processing unit 21.

[0031] The resource amount estimation model management device 3 has a resource amount estimation model DB 31. The resource amount estimation model DB 31 stores in advance, in association with tasks T1 to Tn (n: any natural number representing the number of target tasks) that are included in the source code of the application 1 and request estimation of optimal resources, resource amount estimation models being relational expressions for estimating optimal resource amounts when the tasks T1 to Tn process analysis data of any data amount.

[0032] The optimal resource estimation device 4 estimates an optimal resource amount for minimizing an execution time when data is input to a task Ti (i=1 to n). The optimal resource estimation device 4 includes a model selection unit 41, an input data evaluation unit 42, an optimal resource estimation unit 43, and a resource scale management unit 44.

[0033] The model selection unit 41 uses identifiable information such as the function name of the task Ti to select an appropriate resource amount estimation model from the resource amount estimation model DB 31 and outputs the model to the optimal resource estimation unit 43 .

[0034] The input data evaluation unit 42 measures the amount of analysis data input to task Ti by storing the analysis data input to task Ti in a data set, an array, etc., and outputs the measurement result to the optimal resource estimation unit 43.

[0035] The optimal resource estimation unit 43 estimates the optimal resource amount when processing the task Ti, based on the selected resource amount estimation model and the data amount of the analysis data.

[0036] The resource scale management unit 44 requests scale-in or scale-out, etc., to the distributed analysis processing unit 21 of the distributed analysis platform 2 in accordance with the resource amount estimated by the optimal resource estimation unit 43, and confirms that the request is executed without any problems.

[0037] <Example of hardware configuration of computer 200> FIG. 2 shows an example of the hardware configuration of a computer 200 that realizes the application 1, the distribution analysis platform 2, the resource amount estimation model management device 3, and the optimal resource estimation device 4.

[0038] The computer 200 includes a processor 201 such as a CPU (Central Processing Unit), memory 202 such as a DRAM (Dynamic Random Access Memory), storage 203 such as a HDD (Hard Disk Drive) or SSD (Solid State Drive), input devices 204 such as a keyboard, mouse, or media drive, output devices 205 such as a display, and a communication module 206 such as an Ethernet (trademark) card or a Wi-Fi (trademark) adapter.

[0039] For example, a computer 200 realizing the optimal resource estimation device 4 realizes a model selection unit 41, an input data evaluation unit 42, an optimal resource estimation unit 43, and a resource scale management unit 44 by a processor 201 executing a predetermined program stored in a memory 202.

[0040] The predetermined program executed by the processor 201 may be stored in advance in the memory 202, or may be downloaded from a predetermined server or the like via a removable medium (CD-ROM, flash memory, etc.) or the network N, stored in the storage 203, which is a non-transitory storage medium, and read out from the storage 203 when necessary. For this reason, the computer 200 may have an interface for reading data from a removable medium.

[0041] <Application source code> FIG. 3 shows an example of source code representing the processing executed by the application 1.

[0042] In the example of the figure, the source code is written in Python. In the source code, the distributed analysis platform 2 is assumed to be constructed using Ray (registered trademark), which is a Python-specific distributed parallel processing framework, and the Ray (registered trademark) library is imported in the second line. The optimal resource estimation library 13 that provides functions and the like required for optimal resource estimation by the resource optimization system 100-1 is imported in the third line.

[0043] From the sixth line onwards, processes and functions are defined. From the sixth line to the eighth line, a function to be processed locally is defined. From the tenth line to the thirteenth line, a function to be processed by the variance analysis processing unit 21 is defined by adding the Ray (registered trademark) decorator "@ray.remote()" to the function. Meanwhile, in the functions from the fifteenth line to the eighteenth line and from the twenty-third line to the twenty-third line, an option "optimize_resource=True" is added in addition to the Ray (registered trademark) decorator "@ray.remote()" as in the above-mentioned functions. Therefore, after the variance analysis processing unit 21 is adjusted to the optimal resource when executing the function (task) by the optimal resource estimation device 4, the variance analysis processing unit 21 executes the function (task). From the twenty-fourth line onwards, the source code is written according to the same rules.

[0044] <Example of analysis data> 4 shows an example of the analysis data input to the target task. The content of the analysis data is not important and does not affect the present invention, so a description thereof will be omitted.

[0045] <Data structure of resource amount estimation model DB31> 5 shows an example of the data configuration of the resource amount estimation model DB 31. In the resource amount estimation model DB 31, correlation equations relating to resource amounts and execution times with respect to data amounts, which have been learned in advance using learning data, are recorded as resource amount estimation models in association with identifiers such as function names for each target task.

[0046] <Source code processing by application 1> 6 is a flowchart showing an example of a source code corresponding process by the application 1 of the resource optimizing system 100-1. The source code corresponding process is started in response to a predetermined start operation from the user, for example.

[0047] First, the optimal resource estimation library 13 receives source code indicating the processing content to be executed by the application 1 and analysis data to be processed (step S1). Next, the optimal resource estimation library 13 starts executing the main processing defined in the source code (step S2).

[0048] Next, the optimal resource estimation library 13 reads the source code and determines whether or not a task to be optimized (target task) is detected from the source code (step S3). Specifically, if a function with the decorator "@ray.remote(optimize_resource=True)" is present in the source code, it is determined that the target task is detected. If it is determined that the target task is detected (YES in step S3), the optimal resource estimation library 13 then holds the execution of the process corresponding to the source code (step S4).

[0049] Next, the target task processing unit 11 transmits the identifier of the target task (such as a function name) and analysis data input to the target task to the optimal resource estimation device 4, and requests the execution of dynamic resource optimization processing (step S5). In response to the request, the optimal resource estimation device 4 executes dynamic resource optimization processing for the distributed analysis processing unit 21 of the distributed analysis platform 2, and adjusts resources. The dynamic resource optimization processing will be described later in detail.

[0050] After the resources of the distributed analysis processing unit 21 are adjusted, the optimal resource estimation library 13 then releases the hold on the execution of the process corresponding to the source code in step S4 (step S6). Next, the target task processing unit 11 requests the distributed analysis processing unit 21 of the distributed analysis platform 2 to execute the target task detected in step S3 (step S7). In response to the request, the distributed analysis processing unit 21 in the distributed analysis platform 2 executes the target task. After confirming a response to the request, steps S2 and after are repeatedly executed.

[0051] In addition, in step S3, if the optimal resource estimation library 13 determines that the target task is not detected (NO in step S3), then the non-target task processing unit 12 executes the tasks (asymmetric tasks) remaining in the source code that are not targets of resource optimization (step S8). This concludes the description of the source code corresponding processing by the application 1.

[0052] <Dynamic resource optimization process by optimal resource estimation device 4> FIG. 7 is a flowchart illustrating an example of a dynamic resource optimization process performed by the optimal resource estimation device 4. As shown in FIG.

[0053] The dynamic resource optimization process is started in response to an execution request from the target task processing unit 11 in step S5 of the resource corresponding process (FIG. 6).

[0054] First, the model selection unit 41 of the optimal resource estimation device 4 refers to the resource amount estimation model 31 of the resource amount estimation model management device 3 and selects a resource amount estimation model corresponding to the task ID transmitted from the target task processing unit 11 (step S501).

[0055] Next, the input data evaluation unit 42 measures the amount of analysis data transmitted from the target task processing unit 11 and input to the target task by a method such as storing the data in a data set or an array (step S502).

[0056] Next, the optimal resource estimation unit 43 estimates an optimal resource amount based on the resource amount estimation model selected in step S501 and the amount of analysis data measured in step S502 (step S503).

[0057] The resource scale management unit 44 uses the resource amount estimated in step S503 to request the distributed analysis processing unit 21 of the distributed analysis platform 2 to scale in or scale out resources (step S504), and confirms from the resource status of the distributed analysis processing unit 21 that the requested resource amount has been secured (step S505).

[0058] It should be noted that steps S502 and S503 may be performed in any order. The dynamic resource optimization process performed by the optimal resource estimation device 4 has been described above.

[0059] <Resource optimization system 100-2 according to a second embodiment of the present invention> Next, FIG. 8 shows an example of the configuration of a resource optimizing system 100-2 according to a second embodiment of the present invention.

[0060] The resource optimization system 100-2 is obtained by adding a learning data DB 32 and a model update unit 33 to the resource amount estimation model management device 3 of the resource optimization system 100-1 (FIG. 1). The components of the resource optimization system 100-2 other than the learning data DB 32 and the model update unit 33 are given the same reference numerals as the components of the resource optimization system 100-1 (FIG. 1), and the description thereof will be omitted.

[0061] The resource optimization system 100-2 uses the execution results of the target task after the resource optimization process as learning data, and continuously updates the resource estimation model in the resource amount estimation model DB 31.

[0062] The learning data DB 32 stores execution results (resource amounts, data amounts, execution times, etc.) of target tasks after resource optimization processing as learning data required for calculating a resource amount estimation model. The model update unit 33 re-learns the resource amount estimation model based on the learning data stored in the learning data DB 32, and updates the resource amount estimation model DB31 by overwriting the re-learned resource amount estimation model.

[0063] <Data structure of learning data DB32> 9 shows an example of the data structure of the learning data DB 32. In the learning data DB 32, resource amounts corresponding to various data amounts and their execution times are recorded for each target task. In the example shown in the figure, the resource amounts in the variance analysis processing unit 21, the amount of data input to the task, and the execution time required to execute the task are recorded in association with sequence numbers (seq) assigned to different data amounts. The execution time includes a transmission delay when inputting data to the task.

[0064] <Source code processing by application 1> 10 is a flowchart showing an example of a source code corresponding process by the application 1 of the resource optimizing system 100-2. This source code corresponding process is obtained by adding step S21 after step S7 of the source code corresponding process by the application 1 of the resource optimizing system 100-1 (FIG. 6). Steps S1 to S8 are common, so their explanation will be omitted as appropriate.

[0065] In step S7, when the target task processing unit 11 requests the distributed analysis processing unit 21 of the distributed analysis platform 2 to execute the target task, the distributed analysis processing unit 21 of the distributed analysis platform 2 executes the target task in response to the request. Next, the optimal resource estimation library 13 requests the model update unit 33 of the resource amount estimation model management device 3 to execute a resource amount estimation model update process (step S21). In response to this request, the model update unit 33 executes the resource amount estimation model update process. When a response to the execution request for the target task is confirmed, step S2 and subsequent steps are repeatedly executed.

[0066] <Resource Amount Estimation Model Update Process by Model Update Unit 33> FIG. 11 is a flowchart showing an example of a resource amount estimation model update process performed by the model update unit 33.

[0067] First, the model update unit 33 acquires results (resource amount, data amount, execution time, etc.) when a target task is executed from the distribution analysis platform 2 via the optimal resource estimation device 4, and stores the results as learning data in the learning data DB 32 (step S211). Next, the model update unit 33 re-learns the resource amount estimation model (relational equation) based on the learning data stored in the learning data DB 32 (step S212). Next, the model update unit 33 updates the resource amount estimation model DB 31 by overwriting the re-learned resource amount estimation model (step S213). This concludes the description of the resource amount estimation model update process by the model update unit 33.

[0068] According to the resource optimization system 100-2, the accuracy of the resource optimization process is improved by re-learning the resource amount estimation model (relational expression), so that it is possible to further shorten the time required for source code corresponding processing by the application 1.

[0069] <Resource optimization system 100-3 according to a third embodiment of the present invention> Next, Fig. 12 shows a configuration example of a resource optimization system 100-3 according to a third embodiment of the present invention. The above-mentioned resource optimization system 100-1 (Fig. 1) is based on the premise that a resource amount estimation model generated in advance is stored in the resource amount estimation model DB 31, but the resource optimization system 100-3 is adapted to generate a resource amount estimation model prior to the source code corresponding process (Fig. 6) by the application 1.

[0070] The resource optimization system 100-3 is obtained by deleting the model update unit 33 from the resource amount estimation model management device 3 of the resource optimization system 100-2 (FIG. 8) and adding a sample analysis data DB 34, a learning data collection unit 35, and a model generation unit 36. The components of the resource optimization system 100-3 other than the sample analysis data DB 34, the learning data collection unit 35, and the model generation unit 36 ​​are given the same reference numerals as the components of the resource optimization systems 100-1 (FIG. 1) and 200 (FIG. 8), and descriptions thereof will be omitted.

[0071] The sample analysis data DB 34 stores sample analysis data to be input for the tasks T1 to Tn. The learning data collection unit 35 collects learning data required for generating a resource amount estimation model, which is obtained when the tasks T1 to Tn are executed by the variance analysis platform 2 using the sample analysis data stored in the sample analysis data DB 34. The model generation unit 36 ​​generates a resource amount estimation model based on the learning data stored in the learning data DB 32, and stores the generated model in the resource amount estimation model DB 31.

[0072] <An example of sample analysis data> 13 shows an example of sample analysis data to be input to a target task. Sample analysis data is prepared in association with each target task. Note that the contents of the sample analysis data are not important and do not affect the present invention, so a description thereof will be omitted.

[0073] <Resource Amount Estimation Model Generation Process by Resource Amount Estimation Model Management Device 3> 14 is a flowchart showing an example of a resource amount estimation model generation process by the resource amount estimation model management device 3. The resource amount estimation model generation process is executed prior to the source code corresponding process by the application 1 (FIG. 6).

[0074] First, the resource amount estimation model management device 3 acquires the source code executed by the application 1 (step S31). Next, the learning data collection unit 35 extracts target tasks by detecting a function with the decorator "@ray.remote(optimize_resource=True)" from the source code (step S32). Steps S33 to S36 described below are executed for all target tasks extracted here.

[0075] Next, the learning data collection unit 35 executes the extracted target task while changing the amount of resources while providing the data stored in the sample analysis data DB 34 to the target task (step S33). Note that the amount of sample analysis data provided is also changed successively by cutting out a part of the data. Next, the learning data collection unit 35 measures the execution time for each combination of the data amount and resource amount of the sample analysis data, collects it as learning data, and stores it in the learning data DB 32 (step S34).

[0076] Next, the model generating unit 36 ​​generates a resource amount estimation model based on the learning data stored in the learning data DB 32 (step S35). When generating the resource amount estimation model, as the resource amount and data size increase, the latency also increases, and the amount of change is eliminated by the amount of latency. Therefore, the larger the resource amount, the smaller the amount of change in the execution time, so it is considered that the execution time decreases exponentially as the resource amount increases. Also, as the data amount increases, the amount of change in the execution time increases, so it is considered that the execution time increases exponentially as the data amount increases. Under these conditions, an approximation line is found from the learning data to generate a resource amount estimation model (relational expression) of the resource amount, data amount, and execution time for each target task.

[0077] Next, the model generation unit 36 ​​stores the generated resource amount estimation model (relational expression) in the resource amount estimation model DB 31 (step S36). The above is the description of the resource amount estimation model generation process performed by the resource amount estimation model management device 3.

[0078] According to the resource optimizing system 100-3, a resource amount estimation model that is a prerequisite for executing source code corresponding processing by the application 1 can be generated.

[0079] The present invention is not limited to the above-described embodiment, and various modifications are possible. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to those having all of the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with a configuration of another embodiment, or to add a part of the configuration of another embodiment.

[0080] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in hardware, partly or in whole, by designing them as an integrated circuit, for example. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as a program, table, file, etc. that realizes each function can be placed in a memory, a recording device such as a hard disk or SSD, or a recording medium such as an IC card, an SD card, or a DVD. In addition, the control lines and information lines are shown as being necessary for explanation, and not all control lines and information lines are necessarily shown in the product. In reality, it may be considered that almost all configurations are connected to each other. [Explanation of symbols]

[0081] 1. Application 2. Variance analysis platform 3. Resource amount estimation model management device 4. Optimal resource estimation device 11. Target task processing unit 12. Asymmetric task processor 13. Optimal resource estimation library 21. Variance analysis processing unit 22. Distributed tracing collection 23 Distributed tracing information DB 31 Resource amount estimation model DB 32. Learning data DB 33...Model Update Section 34 Sample analysis data DB 35. Learning data collection section 36 Model generation section 41...Model selection section 42 Input data evaluation unit 43 Optimal resource estimation unit 44 Resource Scale Management Department 100-1~100-3...Resource Optimization System 200...Computer 201 Processor 202 Memory 203 Storage 204 Input Device 205 Output device 206 Communication module

Claims

1. 1. A resource optimization system, comprising: One or more computing devices, one or more memory resources, and one or more storage devices; The computing device includes: Detecting target tasks that are targets of resource optimization processing from source code that represents processing executed by the application; Measure the amount of input data for the target task; Based on the measured data amount and a resource amount estimation model, an optimal resource amount for shortening the execution time of the target task is estimated. A resource optimization system comprising:

2. 2. The resource optimization system of claim 1, The computing device includes: The resource amount of the distributed analysis platform is adjusted according to the estimated resource amount, and the distributed analysis platform is caused to execute the target task. A resource optimization system comprising:

3. 3. The resource optimization system of claim 2, The computing device includes: During the period in which the amount of resources of the distributed analysis platform is being adjusted, a process corresponding to the source code is held. A resource optimization system comprising:

4. 3. The resource optimization system of claim 2, The computing device includes: Monitor the distributed analysis platform and retain the monitoring results. A resource optimization system comprising:

5. 2. The resource optimization system of claim 1, a resource amount estimation model database in which the resource amount estimation models corresponding to one or more of the target tasks are stored. A resource optimization system comprising:

6. 6. A resource optimization system according to claim 5, The computing device includes: selecting, from the resource amount estimation model database, the resource amount estimation model corresponding to the target task detected from the source code, and estimating an optimal resource amount for shortening the execution time of the target task based on the measured data amount and the selected resource amount estimation model. A resource optimization system comprising:

7. 3. The resource optimization system of claim 2, The computing device includes: The resource amount estimation model is retrained using the data amount, the resource amount, and the execution time when the target task is executed by the variance analysis platform as learning data. A resource optimization system comprising:

8. 2. The resource optimization system of claim 1, The computing device includes: The target task is executed by applying a combination of a plurality of sample input data with different data amounts and a plurality of resource amounts, and the resource amount estimation model is generated using the data amount, the resource amount, and the execution time as learning data. A resource optimization system comprising:

9. A resource optimization method by a resource optimization system, comprising: The resource optimization system includes one or more computing devices, one or more memory resources, and one or more storage devices; The resource optimization method includes: a step of the computing device detecting a target task to be a target of a resource optimization process from a source code representing a process executed by an application; The computing device measures an amount of input data for the target task; The computing device estimates an optimal resource amount for shortening an execution time of the target task based on the measured data amount and a resource amount estimation model; A resource optimization method comprising:

Citation Information

Patent Citations

  • Distributed resource scheduling method and system

    CN108345501A