Method for predicting performance of software program

Abstract

To provide a method for predicting performance of a software program, a computer program, a device, and a computer readable storage medium.SOLUTION: A method 100 includes: a step 101 of profiling, by a computer 10, during execution of a software program (a computer program 20) of a computer readable storage medium 15, a bytecode representation of the program to provide an execution property; a step 102 of profiling a runtime on a target device, wherein the profiling is performed while executing at least one test program on the runtime of a target device to provide a runtime property of the target device; and a step 103 of predicting performance of an execution of the software program on the target device based on the execution property of the software program and the runtime property of the target device.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a method for predicting the performance of a software program. Furthermore, the invention relates to a computer program, a device and a storage medium therefor. [Background technology]

[0002] Predicting the performance of software components when they run on different platforms can be difficult due to differences in compiler optimizations, operating system (OS) performance, and computer hardware architectures. This performance uncertainty can be exacerbated by conflicts with other tasks and workloads that cause additional input / output (I / O) on the system, and by memory determinism. Performance predictions are usually in terms of execution time, but can also include energy consumption, memory usage, or network usage. Known solutions to this problem can include worst-case performance measurements, binary analysis, source code analysis, and up-front benchmarking against similar targets, with simultaneous recording of performance events. Moreover, new approaches are investigating techniques based on machine learning, among others, to gather as much information as possible about the platform and the various functions of the software under test. In the above examples, the functions can be statically analyzed by source code or binary code analysis, or metrics about their runtime on the compiled target are measured.

[0003] Bytecode analysis uses static analysis of a program's raw bytecode, i.e., intermediate code format, to predict its execution time. However, this static analysis can be difficult to perform because it can be hard to know which parts of the bytecode will be executed and how often at run time.

[0004] Alternatively, dynamic analysis can be used to capture the stream of bytecode instructions and memory access patterns on the target processor as they are executed. This can be difficult to achieve without significantly impacting program behavior on hardware running in real time, further adversely affecting prediction accuracy. Finally, each analysis of the raw bytecode can be adversely affected by the fact that different hardware platforms may have different bytecode instruction sets, making comparisons between different platforms difficult.

[0005] Source code analysis can use static analysis of a software program's source code to predict its execution time. Source code can be cross-platform in nature, making it very practical for static analysis. However, source code analysis can require a different analysis system for each programming language used, making support for multiple programming languages ​​difficult and time consuming. Source code analysis can also face a common static analysis difficulty of not being able to know which code paths will be taken during execution.

[0006] When profiling performance events, dynamic profiling, in which hardware performance events are observed during execution, can be used to predict performance on different targets. To minimize intrusiveness and not negatively affect program execution, these hardware performance events may not be as comprehensive or intrusive as bytecode analysis. Thus, profiling with performance events may not capture all the features of a program, which limits the potential accuracy of prediction algorithms. Summary of the Invention [Means for solving the problem]

[0007] According to aspects of the present invention there is provided a method with the features of claim 1, a computer program with the features of claim 8, a data processing device with the features of claim 9 and a computer-readable storage medium with the features of claim 10. Further features and details of the invention emerge from the respective dependent claims, the description and the drawings. Features and details mentioned in the context of the method according to the invention also correspond respectively to the computer program according to the invention, the data processing device according to the invention and the computer-readable storage medium according to the invention and vice versa.

[0008] According to one aspect of the present invention, there is provided a method for predicting performance of a software program, comprising the steps of: - profiling a software program, the software program being provided as a bytecode representation, and the profiling being performed during execution of the software program to provide execution properties of the software program; - profiling a runtime on a target device, the profiling being performed while at least one test program is executed on the runtime of the target device to provide runtime properties of the target device; - predicting performance of the execution of the software program on the target device based on execution properties of the software program and run-time properties of the target device; A method is provided, comprising:

[0009] In other words, profiling of a software program can be understood as analyzing the behavior of the software program while it is being executed. Profiling of the runtime on a target device can be understood as analyzing the behavior and technical characteristics of a target device, such as its performance in terms of execution time. The term "target device" can also be called or understood as "target hardware". In general, profiling of the runtime on a software program and a target device can be performed to obtain information about performance and efficiency, function calls, memory usage and access, thread behavior, and / or input / output operations. In brief, according to one example of the present invention, profiling of a software program can indicate which bytecode operations are executed by the software program, and profiling of the runtime on a target device can indicate how much time each such bytecode operation takes on the target device, or how much energy is consumed, or how much memory is used. Thus, the performance prediction may also include energy consumption, memory usage, or network usage. The method according to the invention may be advantageous in cloud computing environments, particularly cloud deployments, due to the fact that it can profile different software programs and different target devices, thereby more accurately predicting performance for different pairings of software programs and target devices. The invention may also be advantageous in situations where the performance of a software program must be predicted as it is moved from one platform to another.

[0010] This method, - compiling the software program into a bytecode representation; - compiling the bytecode representation as a pre-compiler to render the bytecode representation executable on the target device; It may be advantageous when the composition further comprises at least one of:

[0011] Compiling the bytecode representation as a pre-compilation may have the advantages of higher performance, more predictable behavior and performance of the software program, and lower resource usage compared to run-time compilation.

[0012] The bytecode representation includes at least one bytecode instruction, and profiling the software program includes: - executing at least one bytecode instruction at a time to analyze and describe the bytecode representation at least in terms of a frequency of the at least one bytecode instruction and a pattern of memory accesses of the at least one bytecode instruction to provide execution properties of the software program. It is possible for the method to further include:

[0013] The pattern of memory accesses may be, for example, a periodic pattern, a sporadic pattern, or a burst pattern. It may further be contemplated that the frequency of individual memory accesses may be analyzed and described.

[0014] In another example, an analysis and description of the bytecode representation is performed based on a bytecode operation code sequence of bytecode instructions to provide an estimated execution frequency and an estimated memory access pattern for each bytecode operation code sequence. An operation code may be a portion of a machine instruction that indicates an operation that may be performed by a CPU of a computer. A bytecode operation code sequence may refer to a series of such operations, i.e., a sequence of operation codes that the CPU executes in sequence. An operation code sequence may generally result from compiling a high-level program into machine code, or may be a portion of bytecode that a virtual machine or interpreter executes.

[0015] In another example, the step of predicting the performance comprises: - execution time of the software program's execution on the target device; - the energy consumption of the execution of the software program on the target device; - memory usage of the software program execution on the target device; - Network usage of software programs running on target devices and The method includes predicting at least one of the following:

[0016] For each of the above aspects, it is also possible to predict the execution of the software program for different inputs. Predicting some or all of the above aspects, rather than just execution time, can be advantageous in that it provides a more detailed and sophisticated analysis of the software program's execution on the target hardware.

[0017] Profiling the runtime on the target device - acquiring data relating to the execution of at least one test program during the execution of the at least one test program, the data including portions of different instruction patterns of the at least one test program; - analyzing at least a portion of the different instruction patterns with respect to their execution times to provide a runtime property based on the analyzed portion of the obtained data; It is further possible to include:

[0018] An advantage of profiling the runtime with the analyzed portion may be that an understanding of the performance of the target device may be gained, which may then allow for more accurate prediction of the performance of the software program running on the target device. The different instruction patterns may be operation code sequences, in particular bytecode operation code sequences.

[0019] In another example, profiling a software program includes providing a series of at least two different inputs to ensure that at least two different paths of the software program are comprehensively executed. Multiple different inputs may be provided, in which case an increased selection of inputs may advantageously enable more comprehensive profiling of the software program.

[0020] It may be further advantageous when the runtimes of at least two software programs and / or at least two target devices are profiled and performance is predicted based on the execution properties of the at least two software programs and / or the runtime properties of the at least two target devices to provide a performance prediction as a function of the runtimes of the at least two software programs and / or at least two target devices, respectively. The performance prediction as a function may be a matrix that may include execution properties of multiple software programs and runtime properties of multiple target devices, thereby advantageously allowing for an accurate prediction of performance for each pairing of software programs and target devices.

[0021] In another aspect of the invention, a computer program, in particular a computer program product, can be provided which comprises instructions which cause a computer to carry out the method according to the invention when the computer program is executed by a computer. The computer program according to the invention can therefore have the same advantages as those detailed above for the method according to the invention.

[0022] In another aspect of the present invention, a device for data processing can be provided that is configured to perform the method according to the present invention. As such a device, for example, a computer that executes the computer program according to the present invention can be contemplated. The computer can include at least one processor that can be used to execute the computer program. It is also possible to provide a non-volatile data memory that can store the computer program and from which the processor can read and execute the computer program.

[0023] According to another aspect of the invention, a computer-readable storage medium may be provided which includes a computer program according to the invention and / or which includes instructions which, when executed by a computer, cause the computer to carry out the steps of the method according to the invention. The storage medium may be configured as a data memory, such as for example a hard disk, and / or a non-volatile memory, and / or a memory card, and / or a solid-state drive. The storage medium may for example be integrated into the computer.

[0024] Moreover, the methods according to the present invention may be implemented as computer-implemented methods. Further advantages, features and details of the invention will become apparent from the following description, in which exemplary embodiments of the invention are described in detail with reference to the drawings, in which the features mentioned in the claims and in the specification may each individually be an essential part of the invention, either by themselves or in any combination. [Brief description of the drawings]

[0025] [Figure 1] 1 illustrates a method, a computer program, a storage medium, and a device according to an embodiment of the present invention. [Diagram 2] FIG. 2 illustrates a method according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] FIG. 1 shows a schematic diagram of a method 100, a computer program 20, a storage medium 15 and a device 10 according to an embodiment of the invention. A method 100 according to an embodiment of the invention may include a first step 101 in which a software program 1 is profiled, the software program 1 being provided as a bytecode representation 2 to provide execution properties of the software program 1. The profiling may be performed while the software program 1 is executed. In a second step 102, a runtime on a target device 4 may be profiled to provide runtime properties of the target device 4. The profiling may be performed while at least one test program 5 is executed on the runtime of the target device 4. In a third step, based on the execution properties of the software program 1 and the runtime properties of the target device 4, the performance of the execution of the software program 1 on the target device 4 may be predicted.

[0027] FIG. 2 illustrates a method according to an embodiment of the invention in more detail. The present invention addresses, inter alia, the problem of predicting the performance of a software program 1 when it is moved from one platform to another.

[0028] As an alternative to using source code, binary or dynamic profiling measurements to predict performance, it is proposed, inter alia, to profile bytecode programs in an interpreted environment with extensive instrumentation and then to use these measurements to predict how well a compiled version of the software program 1 will perform when provided. In the interpreted environment, a common intermediate instruction set can be used between all programs 1, thereby providing a "narrow waist" for algorithmic interoperability. For example, the choice of a simplified bytecode representation 2 with a reduced instruction set and no garbage collection can make the method according to the invention particularly suitable for runtime prediction. This intermediate instruction set program can then be compiled for provision on the target device 4 by Ahead-of-time (AoT) compilation 8 or Just-in-Time (JIT) compilation 7.

[0029] One realization in the present invention may be that while the performance of large, complex programs is difficult to predict, the performance of fine-grained virtual instruction sequences can be much more easily predicted. Profiling the software program 1 in interpreted mode allows for accurate creation of fine-grained distributions of virtual instruction sequences. Because virtual instructions can be used, profiling can be performed on CPUs that are much faster than the CPU on which the program is provided. In this way, data streams can be processed in real time, although this entails the additional cost of interpreted operations and extensive instrumentation. Finally, the compiled version can run on much slower, resource-limited platforms with similar performance but with much better energy efficiency, more flexible placement, or other favorable system properties.

[0030] The setup may include a software program 1 whose performance has to be predicted, a compiler 6 capable of converting the program into a bytecode or intermediate representation 2, and a "Runtime", i.e. a virtual execution environment that executes the bytecode in an interpretive mode. Note that the generated bytecode may be independent of the underlying machine instruction set architecture. The performance prediction approach according to one embodiment shown in Figure 1 may include five main steps:

[0031] In a first step, the program can be compiled by a compiler 6 into a bytecode representation 2 (e.g., WebAssembly). In a second step, the program can be profiled in interpretive mode (101), inter alia, by executing the bytecode instructions one after the other using a CPU simulated in software. The profiling setup can also ensure that different paths of the program are executed comprehensively by providing a set of inputs. The profiling step 101 can generate a distribution of bytecode instructions and memory access frequencies. In a third step, the runtime on the platform of the target device 4 can be profiled (102) using a number of test programs 5 covering a range of possible programs that may be relevant. In a fourth step, the bytecode can be compiled as an ahead-of-time compilation (AOT) 8 for execution on the target device 4. In a fifth step, the program profile, i.e., characteristics of the execution properties 11, e.g., opcode sequence and frequency, and the runtime properties 41 of the runtime of the target device 4 can be used to predict the performance and provide an expected performance 3 (103).

[0032] The above approach may include the generation of an application bytecode profile, i.e. the execution properties 11 of the software program 1, and a runtime target profile, and may assume that the bytecode will be executed on the target device 4 by ahead-of-time (AOT) compilation 8 or just-in-time (JIT) compilation 7. These steps are described more precisely below.

[0033] The interpretation of the program profiling of step 2 is described in more detail below. A bytecode program can be executed in a software interpreter where the exact instruction patterns and loop executions can be rigorously measured. When this bytecode is then compiled into unique binary data on the target device 4, many of these properties from the interpreted profiling can still be applied even though the underlying instruction set architecture of the device 4 may be significantly different. This can improve the accuracy of the prediction problem since the final target binary file has virtual machine elements that still function similarly to the interpreted version. This step can generate a distribution of bytecode opcodes with estimated execution frequency, along with similar statistics on memory accesses.

[0034] Step 3, Target Runtime Profiling (Virtual Execution Environment), is described in more detail below. A profile of the runtime execution environment may be created independently for a specific purpose, where the objective is to execute a short sequence of bytecode operations and predict their performance on a specific target device 4 after they are compiled into a native binary file. This may require instrumentation of the runtime and subsequent benchmarking of a number of test programs 5 to know how the runtime of this specific target device 4 responds to the short sequence of compiled bytecodes. This process may be automated and repeated for each target device 4, resulting in a target environment distribution of bytecode sequences and predicted performance metrics 3.

[0035] Step 5, performance prediction, is described in more detail below. That is, the final performance prediction 3 can be a function of the application profile distribution and the target performance distribution. This can be achieved in a variety of ways using traditional performance prediction approaches.

[0036] A simple approach may consist in adding the expectation between the application instruction sequence and the closest sequence present in the target distribution. More sophisticated approaches may also be applied. It may be said that performance prediction probably represents a relatively small problem, even when normalized for processor speed and program length. A processor that is effective for a function may probably also be effective for similar functions (e.g., floating-point addition and multiplication). Also, a program that utilizes a function may probably utilize similar functions. Thus, the program runtimes on each device may be organized in a program-device matrix, which may allow the runtime prediction problem to be formulated as a matrix completion problem that is scrutinized. In matrix completion, some of the entries in the matrix may be observed and one may attempt to predict the remaining entries. By grouping all programs and devices together as a single matrix completion problem, potential data efficiency may be maximized. At the same time, the matrix completion formulation may allow for improved predictions for one program or device as data for many other programs or devices is collected, thereby saving the technical effort required to add additional equipment or build more advanced analytical tools that use more data. The matrix completion problem may then be solved using a neural network-based matrix factorization approach.

[0037] This approach can conceptually be applied to any number of bytecode-based virtual execution environments, e.g., Java, WebAssembly, LUA, Python, etc. WebAssembly is sometimes preferred because it supports many different programming languages, as well as interpreted, AoT, and JIT operation modes. The execution semantics of WebAssembly can be formally defined to eliminate non-deterministic execution, making this process even more efficient.

[0038] The above description of the embodiments describes the invention with reference to examples. Naturally, the individual features of the exemplary embodiments can be freely combined with one another, as long as it is technically reasonable, without departing from the scope of the present invention. [Explanation of symbols]

[0039] 1. Software Programs 2 Byte code or intermediate representation 3 Performance Predictions 4. Target Device 5. Test Program 6 Compiler 7 Just-in-time (JIT) compilation 8 Ahead-of-time compilation 10. Computers 11 Execution Properties 15 Computer-readable storage medium 20 Computer Programs 41 Runtime Properties 101 Steps for Profiling a Software Program 102 Steps for profiling the runtime on a target device 103 Steps to predict performance

Claims

1. A method for predicting the performance of a software program (1), comprising the steps of: - a step (101) of profiling said software program (1), said software program (1) being provided as a bytecode representation (2), said profiling being performed during execution of said software program (1) to provide execution properties (11) of said software program (1); - a step (102) of profiling a runtime on a target device (4), said profiling being performed while at least one test program (5) is executed on the runtime on said target device (4) to provide runtime properties (41) of said target device (4); - predicting (103) the performance of the execution of the software program (1) on the target device (4) based on execution properties (11) of the software program (1) and on run-time properties (41) of the target device (4); A method comprising:

2. - compiling said software program (1) into a bytecode representation (2); - compiling said bytecode representation (2) as a pre-compiler (8) to make said bytecode representation (2) executable on a target device (4); The method of claim 1 further comprising at least one of:

3. The bytecode representation (2) comprises at least one bytecode instruction, and profiling the software program (1) comprises: - executing said at least one bytecode instruction at a time to analyze and describe a bytecode representation (2) with respect to at least a frequency of said at least one bytecode instruction and a pattern of memory accesses of said at least one bytecode instruction to provide an execution property (11) of said software program (1); Further comprising: an analysis and description of bytecode representations (2) based on bytecode operation code sequences of said bytecode instructions is performed to provide an estimated execution frequency and an estimated memory access pattern for each bytecode operation code sequence; 3. The method according to claim 2 .

4. The step of predicting performance (103) comprises: - execution time of the software program execution on said target device; - the energy consumption of the execution of a software program on said target device; - memory usage of a software program running on said target device; - network usage of the software program execution on said target device; The method according to any one of claims 1 to 3, characterized in that it comprises predicting at least one of:

5. Runtime profiling on the target device (4) - acquiring data relating to the execution of said at least one test program (5) during said execution, said data comprising portions of different instruction patterns of said at least one test program (5); - analyzing at least a portion of the different instruction patterns with respect to their execution time to provide a runtime property (41) based on the analyzed portion of said acquired data; The method according to any one of claims 1 to 4, characterized in that it comprises:

6. The method according to any one of claims 1 to 5, characterized in that profiling the software program (1) comprises providing at least two different ranges of inputs to ensure that at least two different paths of the software program (1) are executed comprehensively.

7. 7. The method according to claim 1, characterized in that the runtimes of at least two software programs (1) and / or at least two target devices (4) are profiled and the performance is predicted based on execution properties (11) of the at least two software programs (1) and / or runtime properties (41) of the at least two target devices (4) to provide a performance prediction (3) depending on the runtimes of the at least two software programs (1) and / or the at least two target devices (4).

8. A computer program (20) comprising instructions that, when the computer program (20) is executed by a computer (10), cause the computer (10) to carry out a method (100) according to any one of claims 1 to 7.

9. A data processing device (10) comprising means for performing the method (100) according to any one of the preceding claims.

10. A computer readable storage medium (15) comprising instructions which, when executed by a computer (10), cause said computer (10) to perform the steps of the method (100) of any one of claims 1 to 7.

Images