Computer implementation method and computer-readable media

Abstract

The computer implementation method includes the steps of: (I) receiving user program code (UC) having sequences of array instructions, each configured to receive array data structures (A, B, C, D, E, F, G) capable of storing multiple elements of each common data type as input; and generating segmented program code (TCC) from the user program code (UC) that can be executed on a processing unit (PU, CPU, GPU) including several subunits (core #1 to core #4). The segmented program code includes program segments (S1, S2, S3, S4, S5) each containing an array operation, where the array operation corresponds to an array instruction. Each program segment is configured to receive a partially configured runtime instance (A1, B1, C1) of each array data structure as its runtime input, and to partially configure runtime instances (B1, C1, C2) of array data structures (B, C) as its runtime output, and at least partially configure each executable code (KC) containing each array operation, using information from each partially configured runtime instance. The executable code is configured to update the configuration of the runtime instance, which is partially configured as the runtime output of each program segment. The method further includes the steps of: (III) starting execution of the segmented program code on a first subunit (core #1) of the processing unit; (IV) completing the configuration of each executable code when the configuration of the runtime instance received as its respective runtime input is complete and / or updated; and (V) executing each executable code on the processing unit when the respective executable code is only partially configured.

Description

Technical Field

[0001] Embodiments of the present invention relate to computer-implemented methods, and more particularly to a computer-implemented method for executing array instructions on a processing unit including a number of subunits and a corresponding computer-readable medium.

Background Art

[0002] Particularly in the field of numerical calculations and / or data analysis, there is a high demand for efficient utilization of computing resources. This often involves the task of parallelizing array instructions, which is often tedious and cost-intensive and requires important knowledge of low-level computer architectures and related programming techniques. Modern computers may exhibit a heterogeneous architecture as a hierarchy of multiple processing units with varying parallel processing capabilities. For example, a CPU provides multiple individual cores, each of which provides parallel (single instruction, multiple data, SIMD) vector extensions for processing multiple scalar data simultaneously. Furthermore, today, graphics processors and other processing devices are generally available in many common computer setups.

[0003] In order to minimize the execution time and / or energy consumption of a program for obtaining a specific result, the workload processed by the program should be efficiently distributed among the parallel processing resources. One challenge of an automatic parallelization compiler is to identify independent parts of the program workload and, as a result, make them most efficiently processable by parallel processing resources during runtime.

[0004] As described in WO2018 / 197695A1, program segments may be derived from array instructions in user-generated program code written in a subset of common array-based languages, such as ILNumerics, Matlab®, and a subset of NumPy. Segments often replace the original array instructions in the user program. Executing segments asynchronously can help hide the latency of memory copying between individual processing units in a heterogeneous computing system. Thus, various workloads, both small and large, can be handled efficiently. However, executing program segments on a processing unit (PU) with multiple subunits requires more careful resource management, both in terms of memory and thread utilization. Always executing segments asynchronously often results in increased execution time and power consumption, respectively, due to increased latency for handling the additional hardware resources involved.

[0005] Furthermore, constantly managing the resource utilization of segments from host subunits / main threads often leads to high memory usage due to inefficient allocation schemes, limiting the number of segments that can be considered simultaneously. Additionally, any array instruction in user program code that is not supported (by efficient segments), particularly those containing conditional expressions, can result in interrupts in the segment chain, forcing sequential (synchronous) execution on the host / main thread and reducing the efficiency of computational resource utilization. Moreover, existing methods fail to identify all parallel possibilities between subsequent array instructions and their independent parts, and therefore cannot efficiently execute such parts on available parallel computing resources. Each of these, at least some of the possibilities for parallel execution of subsequent segments, has not yet been fully identified or utilized.

[0006] Therefore, there is a need to further improve the typical compiler-based automatic adaptation and execution of user software programs that reference array instructions. [Overview of the project]

[0007] According to an embodiment of the computer implementation method, the method includes the step of receiving user program code, which includes a sequence of array instructions, each of which is configured to receive as input an array data structure capable of storing multiple elements of each common data type. Segmented program code executable on a processing unit, which includes several subunits, is generated from the user program code. The segmented program code includes program segments, each containing an array operation, the array operation corresponding to an array instruction. Each (generated) program segment is configured to receive (at least) a partially configured runtime instance of each array data structure as its respective runtime input. Furthermore, each program segment is configured to use (and / or consider) information from its respective partially configured runtime instance to partially configure a runtime instance of the array data structure as its runtime output, and to at least partially configure each executable code, which includes each array operation. The executable code is configured to update the configuration of the partially configured / configured runtime instance as its runtime output. Execution of the segmented program code is started on a first subunit of the processing unit. When the configuration of the runtime instance received as each runtime input is complete and / or updated, the configuration of each executable code is complete, and when each executable code is partially configured, each executable code is executed on the processing unit.

[0008] Therefore, even with small workloads, automated (i.e., with minimal application at best, typically without manual adaptation) and efficient parallel execution of array-based user programs can be achieved without burdening performance. In particular, the full parallelism potential within the array algorithm of the user program can be automatically identified, and the parallelizable workload can be automatically distributed to subunits. Furthermore, undesirable latencies caused by incomplete data dependencies can often be reduced, hidden, or avoided.

[0009] This can be achieved because program segments, hereafter abbreviated as segments, can be executed synchronously or asynchronously, each depending on the runtime input and the configuration of the runtime instance received as each runtime input (also abbreviated as the configuration of the runtime input). More specifically, at the start of execution, each segment, especially the first segment, may examine its respective runtime input and execute kernel code asynchronously when the runtime input is fully configured (configured / updated). Otherwise (configuration is incomplete and not all input parameters are yet "ready"), the kernel is executed by the (previous) segment that completed the configuration information of the runtime instance as its output. In particular, the runtime kernel of the first segment updates the configuration of the last partially configured input parameters of the first segment that is being executed and updated by the segment, and more specifically, the execution of each runtime kernel of the first segment is executed synchronously by a subunit used by the runtime kernel of the previous segment to complete the output runtime instance element value configuration information. Thus, this method allows segments to be executed on multiple subunits depending on the actual data dependencies and state. Overhead or latency introduced by scheduling runtime kernels and assigning them to subunits or threads, and by switching thread contexts, is prevented. Independent workloads run concurrently while keeping the number of active threads low.

[0010] More generally, the described method is typically a computer implementation for compiling and executing user program code or its substitute, e.g., program code generated from user program code by a trampiler, or an intermediate representation of user program code generated by another compiler from user program code or program code generated by a trampiler. Furthermore, the compilation of this method is typically performed at runtime (at least partially, more typically, at least substantially or even completely) and / or in two stages (phases). In these embodiments, the method may also be called a compilation and execution method. Furthermore, with respect to the proposed two-phase processing, the user program code and its substitute may be considered primary program code, and the segmented program code may be considered secondary program code.

[0011] Note that a runtime instance can be the output of one segment or the input of another segment (or other segments). Therefore, output and input array data structures can define data dependencies between segments. Thus, runtime instances used as outputs and inputs to segments can satisfy data dependencies for each individual call to the segment and can be used to individually represent and / or notify about the state of dependencies for each call and / or each programmatic use of the segment.

[0012] Information from each partially configured runtime instance and / or the configuration of each runtime instance may, in particular, refer to and / or include size information, memory references and / or element values. Thus, each piece of information can be updated individually by individual executable code, and more specifically on different subunits, particularly at different times.

[0013] Furthermore, the information from each partially configured runtime instance and / or the configuration of each runtime instance may further include storage location information, storage layout information and / or element type information.

[0014] Therefore, if at least one of the size information, memory references, storage layout, element values, storage locations, and element type information of a partially configured runtime instance is undefined, uninitialized, or missing, the runtime input may only be partially configured (the configuration is incomplete).

[0015] The configuration of runtime instances as runtime output may be updated, completed, and / or defined by user program code, segmented program code, and / or particularly by executable code. Thus, segmented program code gains great freedom to automatically determine (at runtime and during program segment execution) the best execution strategy, typically a strategy that promises the best ratio between resource management overhead and performance improvements, and in particular a strategy that promises the fastest execution time and / or lowest power consumption by efficiently using as many parallel computing resources as possible.

[0016] Similarly, each part of the configuration of the runtime instance as output may be updated or defined by individual calls to the executable code, and / or each call may execute individual parts of the executable code, for example, on each (separate) subunit. Thus, important configuration information of the runtime data structure is divided into finer information parts, which are individually distributed to later segments and can be received and accessed by later segments to serve as the basis for further execution decisions, without having to wait until the data (the runtime instance of the array data structure) is fully configured.

[0017] Typically, when the configuration of each partially configured runtime instance is updated, typically immediately after the configuration of each partially configured runtime instance is updated, in particular when at least one of the previously undefined, uninitialized, or missing desired configuration information, typically all of them, more specifically when at least one of the respective size information, respective memory references, respective storage layouts, and respective element values ​​is updated, even more specifically when at least one of the respective size information, respective memory references, and respective storage layouts is updated, the configuration of each executable code is completed, and / or the respective executable code is executed on the processing unit. Thus, the resulting executable code is optimized for the particular configuration property of the runtime instance as the respective information becomes available.

[0018] The subunits are at least functionally identical and have access to a (common) shared memory for the processing unit. Therefore, by considering a single processing unit, the costs associated with preparing / copying data between the memories of multiple computing devices / processing units are often negligible. Furthermore, this disclosure supports and improves upon portions of such methods for (automatically) parallelized array-based algorithms for processing on heterogeneous computing resources after selecting processing units (see, for example, WO2018 / 197695A1).

[0019] Furthermore, the processing unit may be a multi-core or many-core CPU, or a CPU or computing grid or cluster with a GPU, or a similar processor.

[0020] Furthermore, each subunit may be a kernel or user thread, management thread, task, fiber, process or compute node, or may be accessed or controlled by or correspond to them.

[0021] The executable code (of each segment) typically serves a similar purpose to the (sequential) execution of the corresponding array instructions. The primary intention is to fully configure the runtime instance as output to the segment, but actual implementations executed in this manner are often optimized in several respects, for example, for efficient low-level hardware resource utilization, efficient memory use, especially to eliminate transient results, and for thread-safe execution. Furthermore, the executable code is often divided into multiple parts, each part often comprising individual information of the runtime instance configuration, and each part is individually executable. The executable code has at least one of several subunits that are (at least functionally) identical, typically each having an executable runtime (computation) kernel. Thus, any part of a segment, in particular any part of the executable code, is executable on any of the subunits. The configuration of the runtime instance can be updated on any of the subunits / threads without requiring the use / return of processing to the main thread to update a particular part of the configuration.

[0022] Typically, the runtime kernel (for each segment) is configured to take array operations and runtime instances as inputs and compute, update, and / or define element values ​​stored in memory referenced by the runtime instances as outputs. In some cases, nested loops are used within the kernel to compute element values ​​in all dimensions of the array result, often using multiple array operations.

[0023] Since the runtime kernel corresponds to a program segment, the runtime kernel is also referred to below as the segment kernel (and simply the kernel). Note that in some embodiments, the runtime kernels described herein may be derived from each program segment, similar to those described in WO2018 / 197695A1 for deriving the runtime kernel within the runtime segment from each program segment.

[0024] Completing the construction of executable code may involve at least one, typically several, or even all of the following: • Generate a hash of the executable code. • Storing or caching executable code on persistent storage (e.g., file system or NAS) • Loading and / or reusing a stored version of the executable code. • Generate at least partially the runtime kernel. • Terminate the runtime kernel Optimizing runtime kernel code with respect to the size, location, and / or storage layout information of at least one runtime instance as input, properties of the execution framework, and / or properties of the processing unit, typically with respect to specific (low-level) instructions (e.g., SIMD or vector register operations), and / or memory properties of the processing unit, such as memory cache properties. - Configure the runtime kernel to receive and / or access at least a portion of runtime instances as input and / or runtime instances as output, - Configure the runtime kernel to run on the processing unit when at least one, typically all, runtime instance element values ​​are fully configured as input.

[0025] Alternatively or additionally, completing the configuration of the executable code may include defining executable instructions or code representations to be executed on a subunit, updating or defining at least a part of the configuration, typically at least the size information of the runtime instance as output, and / or updating or defining at least the respective information of the runtime instance as output when the corresponding information has been updated or defined for the runtime instance as input (for at least one, typically for all runtime instances as input), allocating memory on a processing unit, and updating or defining memory references on the runtime instance as output.

[0026] A code expression (CE) is typically configured to be executed on each subunit used to execute the executable code, and defines or updates the respective information for the runtime instance as input to each segment. Thus, the executable code, particularly the code expression and the runtime kernel, is executed on subunits other than the main subunit / main thread without delaying the processing of (subsequent) segments on the main subunit / thread. Often, the processing of the executable code is also improved (made faster) by avoiding the additional overhead of switching subunits / threads within a sequence of segments having data dependencies on each other.

[0027] Alternatively or additionally, completing the configuration of the executable code may include defining, configuring the information of the runtime instance as input to each segment, typically all runtime instances, particularly size information, and, if available for the executable code at this point, executing the code expression on a first subunit or being followed thereby.

[0028] Alternatively, or further, completing the configuration of the executable code may include, or be followed by, initiating the execution of the runtime kernel on further subunits of the processing unit, provided that all runtime instances as input to each segment are fully configured and available to the runtime kernel at this point. Thus, the typically high workload of computing element values / executing the runtime kernel is delegated to further subunits without delaying the processing of further / later segments by the main subunit / main thread.

[0029] Generating segmented program code may include inspecting and / or analyzing user program code or code derived therefrom in order to extract data flow information and / or analyze data dependencies.

[0030] Depending on this, at least one of the segments may be chained to a previous segment in the sequence of generated segments, and may receive at least one output array data structure of each of the previous segments as an input array data structure.

[0031] Furthermore, configuring the executable code of a second segment in a segment chain may include configuring runtime instances as their respective inputs to the second segment, such that, provided that the output of the first segment is not being used by other segments and all other input array data structures to the second segment are already fully prepared, any modification or update of at least a portion of the configuration, typically at least the size and / or element values, will cause the executable code of the second segment in the segment chain to begin execution synchronously, in particular, on the same subunits, fibers, or threads used by the code that causes the modification or update of the runtime instance's configuration as input, typically by the executable code of each first segment. Thus, subunits / threads are "reused" by continuing to process further dependent segment code without the need to allocate or assign new subunits / threads for their processing.

[0032] The workload of an array instruction or sequence of array instructions is understood as the sum of the basic instructions required to transform input array data into a desired result according to the array instruction, particularly a sequence or set of array instructions. It should be noted that some array instructions may exhibit different characteristics in terms of instruction workload and / or workload for calculating a single element when compared to other array instructions and / or when processed on different processing units.

[0033] In particular, each array instruction may include metadata, hereafter abbreviated as AIMI, which allows for the determination of the configuration information of each input array data structure, and the configuration information of the output of the array instruction, in particular the output array data structure (of each array instruction in the sequence of array instructions).

[0034] Typically, a compiler uses AIMI, and possibly lookup tables according to the array instructions found, to derive and / or generate executable code (bytecode, machine code, code defined in the host language of segmented program code, subject to JIT compilation, or similar). Often, the compiler includes instructions or rules on how to utilize or access the structures of runtime instances and identify specific (size or other) information in each runtime instance storage, so that when the resulting code is executed, the desired configuration information is read from / stored in the runtime instances. The compiler may also use individual array instruction types, having one or more array data structure runtime instances as input and / or producing one or more array data structures (runtime instances) as output. Furthermore, the compiler typically combines or merges the input configuration information, output storage information, and AIMI calculation rules for the output configuration information, in particular calculating output size information for all array operations contained in the segment. Often, the resulting code is appended to the segment in the form of a code representation that is invoked at runtime.

[0035] Typically, the configuration information of the output of an array instruction includes the configuration information of the input array data structure, in particular the size information of the input array data structure.

[0036] Furthermore, AIMI may include additional input data necessary to determine the configuration information of the array instruction's output, particularly information about the source of the additional input data.

[0037] Furthermore, information regarding additional input data may include or refer to the output of each array instruction / segment, and / or be used as input to AIMI, as a reference to input to AIMI, and / or as input to a code expression (CE). Thus, the supported set of array instructions / array instruction types includes array instructions whose configuration information, particularly the size of the output, depends on configuration information generated in other parts of the executable code, and in particular, on the values ​​of elements of at least one input array data structure / runtime instance and / or the size information of the runtime instance as output to the runtime kernel (RK). Consequently, interruptions in the sequence of segments in segmented program code due to the lack of support for a particular array instruction are largely avoided, more segments / array instructions can be considered at once, and more parallelism possibilities for algorithms can be utilized and used for parallel execution. As a result, array instruction level parallelism (AILP) can be achieved or improved.

[0038] Executing segmented program code (TCC) may include receiving information such as AIMI of array instructions corresponding to the segment's array operations, the segment's array operations, configuration information, in particular size information of each runtime instance, runtime instances of each array data structure, and / or information about further input data, to construct an executable instruction and / or code representation, and / or determine and / or update the configuration information, in particular size information of each runtime instance, as the output of the segment.

[0039] Typically, AIMI incorporates, references, receives, and / or determines size information, element type information, and / or layout information for each array instruction. The size information typically includes at least one of the following: the number of dimensions of each runtime instance, the length of at least one of the dimensions of each runtime instance, the position information of each runtime instance, and the stride information of at least one of the dimensions of each runtime instance.

[0040] As used herein, the term “array instruction meta information” (AIMI) is intended to describe a data structure or functional data structure that can programmatically inform about the output size, shape, and potentially other properties of the output of a single array instruction, given information about each of the inputs of the array instruction (at least one of them). Typically, for this method to function, each array instruction is associated with its respective (individual) AIMI, e.g., (code) attribute (referencing the C# attribute class). Alternatively, in another exemplary implementation, each array instruction is associated with a predefined AIMI class, or the compiler matches a known array instruction or an array instruction of a known class with a predefined AIMI.

[0041] Partially configuring runtime instances of an array data structure as runtime outputs for each program segment may include allocating and / or instantiating each runtime instance. Thus, the runtime instance may function as an output runtime instance for a segment. Furthermore, references to the instance may be provided to other segments and used by the executable code of those segments for subsequent updates performed on the runtime instance, enabling (asynchronous) updates to its configuration and / or notifying further segments of the updates.

[0042] Furthermore, the runtime output size information for each runtime instance as the output of each program segment may typically be determined using the AIMI and the runtime size information of the runtime input of each program segment (if the runtime output size is not yet defined and is not determined by other means or later, for example, by different segments). In some cases, especially if a segment contains multiple array operations, multiple AIMIs may be merged or chained into a sequence of individual AIMIs.

[0043] The runtime size information may include at least one of the following: the number of dimensions of each runtime instance, the length of at least one of the dimensions of each runtime instance, the data type of the elements of each runtime instance, the position information of each runtime instance, and the stride information of at least one of the dimensions of each runtime instance.

[0044] Similarly, memory may be allocated (if it has not yet been allocated and will not be allocated in any other way or later) to configure memory reference configuration information and / or to store element values ​​of each runtime output of each program segment.

[0045] User program code is typically user-written code written in an array-based programming language, such as a domain-specific language like ILNumerics, and / or embedded in a host language such as C#, Python, Java®, FORTRAN, or Visual Basic. Alternatively, user program code may be written in a scripting language, numerical language, or scientific language. Furthermore, user-written code may be written using artificial intelligence (AI), particularly AI coding assistance tools such as GitHub Copilot. Examples of user array instructions include sin(A) for calculating the sine of all elements of array A, find(A) for finding the sequential element index of any non-zero element value in A, fft(A) for calculating the Fast Fourier Transform of A, matmul(A,B) for calculating the matrix multiplication of matrices A and B of the same size, sum(A) for reducing the dimensionality of A, and A[B<1] for accessing the element value of A at a (sequential) position where the element value of array B has a value less than 1.

[0046] It should be noted that array-based programming algorithms are popular in the scientific community and many industrial sectors. Two common array-based programming languages ​​today are Matlab (by Mathworks Inc.) and numpy (based on the Python language). Another example is the ILNumerics language, available on the .NET platform.

[0047] In contrast to common algorithms that use various types of scalar data as the basic elements of instructions, array-based languages ​​use array data structures as their basic data elements. Arrays typically exhibit a linear structure. The elements of an array may be arranged along any number of dimensions, and are typically all of the same type, typically numeric data types, such as the "double" or "int" data types in the C programming language.

[0048] A matrix is ​​a typical example of an array having two dimensions. A set of matrices of equal size can be stored as a three-dimensional array. The term "tensor" refers to an abstract mathematical structure of a similar n-dimensional shape. An n-dimensional array can be understood as a tensor stored in computer storage. Due to storage requirements, such storage typically uses one-dimensional computer memory, and therefore, compared to a tensor, more information is desired to fully represent the array. The storage layout is also sometimes called a "stride" and describes how the individual elements of the array are accessed by the one-dimensional memory storage.

[0049] As used herein, the term “stride” is intended to describe information regarding the storage layout of a multidimensional array. Typically, the stride is stored with the array, generally in the header portion of the (input / output) array data structure. The stride value of a dimension generally specifies the storage distance or memory address distance of the elements stored within that dimension. The stride value is often unitless, using a scaling factor of the storage size for a single element of the array element type, or 1 for the byte distance.

[0050] The set of potential array shapes also includes arrays of vector shapes, where vectors may extend along any dimension of the array, often along the first or last dimension. Some languages ​​allow scalar data (0-dimensional arrays) that store only a single element. Some languages ​​define a minimum number of dimensions for arrays, storing scalar data as a matrix or array of at least that minimum number of dimensions, where all dimensions are "singleton dimensions" with length 1. An N-dimensional array may have zero elements if N is greater than 0 and at least one of its dimensions has zero elements. When N is 0, typically by convention, the array is a scalar, i.e., stores exactly one element.

[0051] Furthermore, program segments may be generated as host language code, or as intermediate representations, particularly as bytecode representations of often abstract intermediate machine instructions (e.g., .NET, ECMA CLI intermediate language code, or Java bytecode) or native architecture-specific machine instructions, such as X87 or ARM instructions. For example, segmented program code may be compiled by a compiler, particularly a just-in-time (JIT) compiler, and the executable code and / or runtime kernel may be generated using the same or a different (JIT) compiler.

[0052] However, segmented program code is typically generated using a transpiler, particularly a source-to-source translator that translates program code between programming languages ​​that operate at (almost) the same level of abstraction, for example, between the same level of abstraction and the same language (e.g., C# -> C#).

[0053] More specifically, the sequence of array instructions may be replaced by a sequence of program segments using a transpiler. Therefore, local optimizations can be applied to the region of user program code containing array instructions without needing to analyze the entire user program code.

[0054] Each program segment typically contains an array operation corresponding to an array instruction. However, a program segment may also contain multiple array operations, each typically corresponding to one of the array instructions. Therefore, merging multiple array operations into a single segment runtime eliminates, or at least reduces, the overhead of creating, storing, and reading the intermediate results (often called "temporaries") of individual array operations.

[0055] For each program segment, a device-specific runtime kernel (RK) and code expression (CE) may be generated (using a compiler), which typically perform the respective array operations to calculate the element values ​​and size configuration information of the runtime instance as output to the segment.

[0056] A device-specific runtime kernel may be compiled based on received or determined hardware information and / or configuration information of the runtime instance as input.

[0057] Hardware information may include at least one of the following: the type of processing unit, the number and / or type of compute subunits of the processing unit, subunit properties such as the length of a SIMD processing unit, the size of available cache storage, the length of cache lines, the number and types of similar or equivalent instructions that are executed together, and hardware support for a particular instruction.

[0058] The program flow of user program code may be maintained, at least substantially, by / within the segmented program code. Therefore, local optimization by segments within the segmented program code typically does not alter the semantics of the original algorithm.

[0059] Generating segmented program code typically involves analyzing the primary program code and / or the segmented program code (at its current stage), in particular, the sequence of program segments (generated by the trampiler).

[0060] Furthermore, data dependencies that exist between array instructions determined by the user program code, or between array operations determined for the segmented program code, may be used to generate the segmented program code.

[0061] In particular, dependency graphs for at least some array instructions and array operations may be determined from their respective program code. Thus, segmented program code can be tailored to better match the context of individual array instructions, including decisions regarding hardware resource allocation, especially regarding the synchronous or asynchronous execution of segments, thereby preventing unnecessary overhead in situations where the program lacks parallelism potential and resulting in faster execution time / lower power consumption.

[0062] Furthermore, generating segmented program code may include creating and configuring each program segment for synchronous execution based on a dependency graph, in cases where each program segment does not benefit from parallel execution, particularly when the dependency graph containing array instructions corresponding to array operations within each program segment contains exactly one branch, and executing each program segment synchronously includes executing the executable code, in particular the runtime kernel, on a first subunit (core #1), in particular by a first thread, during or after the configuration of the executable code.

[0063] Furthermore, generating segmented program code may include at least one of the following: including array instructions as corresponding array operations in each program segment; including AIMI in each program segment; including a set of array instructions from a sequence of array instructions in each program segment as corresponding array operations; including reduction array instructions as reduction array operations; including unary array instructions as unary array operations; including array access instructions as array access operations; and including binary array instructions as binary array operations in the first program segment (see also application PCT / EP2022 / 069161, which is incorporated herein by reference in its entirety).

[0064] Therefore, independent workloads may be particularly well distributed and / or multiple subunits of a processing unit (PU) may be efficiently utilized to result in particularly low execution time and / or particularly low energy consumption.

[0065] According to an embodiment of a computer implementation method for executing program code, each configured to be executable on a processing unit, and including program segments, each kernel code and each configuration code, each kernel code includes an array operation corresponding to one array instruction of a sequence of array instructions, and each program segment is configured to receive, as its runtime input, at least partially configured runtime instances of each array data structure capable of storing multiple elements of each common data type, the method includes the steps of: starting execution of the configuration code of one of the program segments on a first subunit of the processing unit to partially configure the runtime instances of the array data structure as the runtime output of each program segment; and executing the kernel code of the program segment on the processing unit when the configuration of the runtime instances received as their runtime inputs is complete and / or updated.

[0066] This method may or may not involve configuring the executable code for each segment that performs each array operation at runtime. This enables efficient parallel execution by leveraging the parallelism of array instructions, and therefore results in better resource utilization efficiency compared to conventional methods that execute adjacent array instructions sequentially, synchronously, and / or on a single thread.

[0067] Therefore, this method is typically a computer implementation method for executing program code derived from user program code. In this embodiment, the method may also be abbreviated as the implementation method.

[0068] The configuration of each runtime instance of this method also typically includes size information, memory references, and / or element values.

[0069] Furthermore, the configuration of each runtime instance may include usage information and / or runtime task information.

[0070] Usage information can be provided, especially at the counter.

[0071] Typically, runtime task information is a reference to a runtime task data structure.

[0072] At entry time, at least one of the size information, memory references, element values, and runtime task information of a partially configured runtime instance is undefined, uninitialized, or missing.

[0073] Executing configuration code may include storing references to runtime instances as runtime inputs to each segment, and / or storing references to runtime instances as runtime outputs to each segment.

[0074] Each reference may be associated with a runtime task data structure that references kernel code and is configured to start execution of the kernel code on a subunit of the processing unit when the runtime instance is configured and / or updated.

[0075] Executing kernel code typically involves using references to runtime instances as runtime inputs and / or runtime instances as runtime outputs to complete the configuration of runtime instances as runtime outputs.

[0076] Partially configuring a runtime instance as runtime output typically involves storing a reference to the runtime task data structure along with the runtime instance.

[0077] Accordingly, completing the configuration of the runtime instance as runtime output typically involves removing the references from the runtime instance to the runtime task data structures.

[0078] Executing configuration code may further include configuring the runtime task data structure to initiate kernel code execution when each runtime instance completes and / or is updated, either as input within the runtime task data structure or as runtime output stored with it.

[0079] The kernel code may also be initiated by kernel code referenced by runtime task data structures, particularly by specific runtime task data structures stored in or associated with the runtime instance when the configuration of the runtime instance received as runtime input is complete and / or updated.

[0080] Usage information may often be updated by configuration code from the main thread.

[0081] In particular, when a reference to a runtime instance as runtime input is stored or associated within a runtime task data structure, the usage counter for each runtime instance as runtime input may be incremented by the configuration code.

[0082] Usage information may also be updated by the kernel code.

[0083] In particular, when the configuration of runtime instances as outputs is complete, and / or when references to runtime instances as runtime inputs are removed from the runtime task data structure, the usage counter for each runtime instance as a runtime input may be decremented by the kernel code.

[0084] Initiating kernel code execution typically involves completing the configuration of a copy of the runtime instance as runtime output, using a copy of the runtime instance as runtime output, based on the usage information of the runtime instance as runtime output, when the usage information indicates that the runtime instance is stored along with further runtime task data structures as runtime inputs, and more specifically, when the value of the zero-based usage information counter is greater than 1.

[0085] Creating a copy of a runtime instance typically involves copying at least a portion of the runtime instance's configuration as runtime output, in particular, copying at least one of the runtime instance's element values ​​as runtime output.

[0086] Program code is typically derived from user-written code, particularly (primary) user program code or its substitute, such as user program code that has been preprocessed by transpilation or other means.

[0087] In these embodiments, the method may include (pre-compilation) using an ahead-of-time (AOT) compilation / compiler.

[0088] In particular, the program code may be pre-generated (e.g., compiled) from user-written code or a substitute thereof. More specifically, in some embodiments, the program code may be generated from user program code, typically as at least substantially or even completely configured program code. In these embodiments, the configured program code may be at least substantially (functionally) similar to or identical to segmented program code (TCC) as described herein.

[0089] In some embodiments, the program code may be identical to the user program code (UC), particularly if the array instructions from the user program code already contain a program segment. In such embodiments, the program segment is typically used and / or referenced (directly or indirectly) from the user program code, similar to the array instructions. Furthermore, in such embodiments, the configuration code, and often even the kernel code, are typically implementations of (almost single) array instructions. Such embodiments are often found in the context of DSLs or interpreted or JIT-compiled programming languages. They may or may not perform method-specific compilation at runtime and / or often use only some of the advantages of the compilation and execution methods described herein. Note that segments in such embodiments often consist of a single array operation corresponding to a single array instruction.

[0090] However, some particularly simple embodiments implement a segment containing a set-up code and / or kernel code having an array instruction, in particular when no adoption of runtime hardware information and / or runtime data is performed for the executable code, in particular when each segment corresponds to a single array operation, when the array operation from the executable code is fully configured before the set-up code of the segment is started, and / or when the array instruction includes a segment containing a set-up code and / or kernel code.

[0091] Furthermore, typically applied to the generation of segmented program code, the generation of program code for a method of executing program code may be performed in particular such that the program flow and / or data flow of the user program code is maintained.

[0092] In particular, user program code may be analyzed using the (generated) abstract syntax tree of the user program code.

[0093] Each program segment typically contains at least one, and typically more, array operations, each array operation typically corresponds to one of the array instructions.

[0094] Typically, a device-specific runtime kernel is generated for each program segment that performs the respective array operation, usually using a compiler, so that each program segment contains the corresponding configuration code and the corresponding kernel code.

[0095] According to the embodiments, a computer program product and a (non-volatile) computer-readable medium each include instructions that, when executed by a computer, cause the computer to perform any of the steps and methods described herein. [Brief explanation of the drawing]

[0096] The components in the drawings are not necessarily to scale; instead, the emphasis is on illustrating the principles of the invention. Furthermore, in the drawings, similar reference numerals indicate corresponding parts. [Figure 1A] A computer implementation method according to the embodiment is shown. [Figure 1B] A computer implementation method according to the embodiment is shown. [Figure 2A] A computer implementation method according to the embodiment is shown. [Figure 2B] Figure 1A shows an exemplary computing system in which the method may be used. [Figure 3A] A computer implementation method according to the embodiment is shown. [Figure 3B] A computer implementation method according to the embodiment is shown. [Figure 4] A computer implementation method according to the embodiment is shown. [Figure 5] A computer implementation method according to the embodiment is shown. [Figure 6A] A computer implementation method according to the embodiment is shown. [Figure 6B] A computer implementation method according to the embodiment is shown. [Modes for carrying out the invention]

[0097] The following detailed description refers to the accompanying drawings, which form part of this specification and illustrate specific embodiments in which the invention may be carried out. In this regard, directional terms such as “top,” “bottom,” “front,” “back,” “first,” “last,” and “next” are used with reference to the orientation of the drawings described. Since the components of the embodiments can be arranged in multiple different orientations, the directional terms are used for illustrative purposes only and are not limiting. It should be understood that other embodiments may be utilized and structural or logical modifications may be made without departing from the scope of the invention. Accordingly, the following detailed description should not be construed as restrictive, and the scope of the invention is defined by the appended claims.

[0098] Various embodiments are referenced in detail here, with one or more examples shown in the drawings. Each example is provided for illustrative purposes only and does not constitute a limitation of the invention. For example, features illustrated or described as part of one embodiment can be used in relation to or in conjunction with other embodiments to bring about further embodiments. The invention is intended to include such modifications and variations. The examples are written using specific languages ​​and should not be construed as limiting the scope of the appended claims. The drawings are not to scale and are for illustrative purposes only. For clarity, the same elements or manufacturing steps are indicated by the same reference numerals in different drawings unless otherwise specified.

[0099] With respect to Figures 1A and 1B, the steps of the computer implementation method 1000 are explained.

[0100] In block I of the illustrated flowchart, which has blocks I to V, each of which can correspond to one or more processes or steps, user program code (UC in Figure 1B) having a sequence of array instructions as shown in Figure 1B is received. Each of the array instructions (find(), sin(), multiplication*, lower than<, array access[]) is configured to receive array data structures A, B, and C as input, which can store multiple elements of their respective common data types. For example, array data structure A may represent medical images such as X-ray images, stock market information, or measurement data.

[0101] In the subsequent Block II, a processing unit containing several subunits, such as the one shown in Figure 2B, generates segmented program code (TCC in Figure 1B) that can be executed on the processing unit, from the user program code (UC).

[0102] Figure 2B shows an exemplary computing system having a processing unit (PU). In the exemplary embodiment, the PU represents a computing resource in a computing system, such as a desktop computer, laptop, or computing node in a computing grid or cloud system. In the exemplary embodiment, the illustrated PU includes a CPU with each subunit, i.e., four exemplary cores (cores #1 to #4) as subunits.

[0103] The following description primarily concerns execution on CPU cores #1 through #4. However, this should not be considered limiting, as the method can also be applied to GPU cores, grid or compute cluster nodes, or any other processing unit having several (separate) subunits.

[0104] The segmented program code TCC of method 1000, as generated in block II of Figure 1A, includes three segments S1 to S3, each having an array operation, where each array operation corresponds to one array instruction of the user program code UC.

[0105] Each of the three exemplary program segments S1, S2, and S3 may be part of a longer sequence of segments, particularly the first part, and is configured to receive runtime instances A1, B1, and C1 of array data structures A, B, and C as its respective runtime input. Each of the runtime instances A1, B1, and C1 may be partially or fully configured, depending on the state of the running PU of the program segments S1, S2, and S3 at runtime.

[0106] Furthermore, each of the three exemplary program segments S1, S2, and S3 is configured to evaluate and use information from their respective partially configured runtime instances A1, B1, and C1 to partially configure runtime instances B1, C1, and C2 of their respective array data structures B and C as runtime outputs of their respective program segments S1, S2, and S3, and to at least partially configure the respective executable code (KC in Figure 1B) that performs the respective array operations, and the executable code KC is configured to update the configurations of the runtime instances B1, C1, and C2 that were partially configured as runtime outputs of their respective program segments S1, S2, and S3 during runtime. As shown by the dashed rectangle in Figure 1B, the executable code KC may be thought of as the (executable) kernel code KC when it is (fully) pre-configured, i.e., in other embodiments, which performs array operations and forms part of each program segment including each configuration code SET, and is configured to partially constitute runtime instances of array data structures as runtime outputs of each program segment, like segmented program code (however, the already configured kernel code KC does not need to be at least partially configured; see Figures 6A, 6B and the corresponding descriptions below for more details).

[0107] In the subsequent Block III, the execution of the segmented program code TCC, more specifically the execution of its first segment S1, begins on the first subunit core #1 of the processing unit (CPU in Figure 2B).

[0108] When the configuration of runtime instance A1 (later B1, C1) received as each runtime input is complete and / or updated, the configuration of each executable code KC is completed in block IV, and if each executable code KC is only partially configured, otherwise block IV is skipped, as indicated by the dashed arrow in Figure 1A.

[0109] Subsequently, each executable code KC may be executed in Block V on a processing unit, for example, on the first subunit core #1. However, this may depend on the configuration of the runtime instance, as will be described in more detail below.

[0110] Figure 1B shows an exemplary execution scenario with a partial configuration of the corresponding runtime instance of the array data structure.

[0111] The example user program code here consists of three lines of ILNumerics code (element type declarations are omitted for brevity). A, B, and C are three non-scalar arrays. A1 (an instance of A) shall have three dimensions, each with a length of 100x-precision floating-point elements.

[0112] In the exemplary embodiment, as already described above, three segments S1 to S3 are created, where each line of the UC gives rise to one segment.

[0113] Upon entering program segment S1, A is already (fully) configured. This means its size, element type, location, storage order, memory references (e.g., buffer pointer or memory address), and element values ​​are fully defined (exemplary assumption). Here, segment S1 runs on thread core #1. This creates a runtime kernel and optimizes (specializes) the kernel to perform the array operation "find(A<0.1)" taking into account the processing unit (PU) and the properties of A1. The kernel is typically stored in a cache or similar storage for fast reuse later.

[0114] Furthermore, segment S1 creates and / or configures a code representation (CE) taking into account the properties of the array operation, runtime instance A1, and processing unit (CPU or GPU). Since the array operation "find(A<0.1)" produces an output array whose size depends on the value of A and the output of the array operation, the segment's TCC is configured to use the result of performing the array operation (runtime kernel RK) to construct the code representation.

[0115] As shown in Figure 1B, the executable code KC of a segment may consist of, or at least include, its respective runtime kernel RK and its respective (corresponding) code representation CE.

[0116] The code representation of segment S1 is configured by the TCC to react to changes in the configuration of the input array instance A1 of the segment / runtime kernel. Here, we describe an implementation that maintains the size, memory references, and element values ​​of the output array. The element values ​​B1 of the output of segment 1 are computed by the runtime kernel when all input array instances are fully configured. The code representation is executed, and the size of the output array is computed typically after all input array instances are fully (sufficiently) configured and the runtime kernel has finished execution.

[0117] Typically, the resulting output runtime instances to a segment are constructed in parallel, i.e., simultaneously, at least partially in parallel, and / or on multiple subunits for multiple segments. This is largely true for the calculation of element values ​​of the resulting instances. It may also be true for other (partial) configuration information, such as size configuration information. However, in this example, size information is relatively easy to calculate. Therefore, size calculation is performed synchronously, typically by the same subunit / thread becoming aware of the completeness of the size information for all / last input array runtime instances to the segment and / or completing the size information. Note that even if the size is calculated synchronously, such calculation may still be performed in parallel with the main thread / subunit processing that is or has processed the TCC.

[0118] Figures 4 and 5, described in more detail below, illustrate exemplary implementations for determining how and when to execute specific parts of the executable code (KC). Thus, Figures 4 and 5 show the process (III) for initiating the execution of the segmented program code (TCC), the process (IV) for completing the configuration of each executable code (KC), and the process (V) for executing each executable code (KC), as described above for Method 1000.

[0119] Referring again to Figure 1B, for this introductory example, we assume the following situation or configuration: If, while processing the TCC, all inputs to the code representation (CE) that calculates the size of the output runtime instance to the executable code, in particular to the segment, are found to have their information (already) configured, then the TCC typically calculates the size information of the (partially configured) output runtime instance to the segment synchronously. In such a case, the executable code corresponding to the size information calculation is called directly by the TCC, and the execution of the executable code is performed on the same subunit used to process the TCC.

[0120] Here, we assume that the input array runtime instance A1 is already configured. However, the coded representation (CE) used to calculate the output size depends on further input data, particularly the results of the runtime kernel (RK) execution of segment S1, which is still missing at this point. Therefore, the coded representation (CE) is only partially configured and prepared for later executions when all input data becomes available.

[0121] It should be noted that in some embodiments, the code representation (CE) that computes the configuration information of a segment's output, particularly the size configuration information of the segment's output, may be integrated or merged with other parts of the executable code (KC). Often, the size of the output is computed by code defined for the runtime kernel (RK), particularly if the output size configuration information depends on at least one element value of the runtime instance as input to the segment. Some embodiments may integrate, merge, or create the size computation code into the runtime kernel (e.g., based on AIMI for array instructions). Some define the code representation as referring to the runtime instance as an output, particularly as referring to the size configuration information of the runtime instance as an output after it has been configured by the runtime kernel (RK).

[0122] Some other implementations may allow the runtime kernel to reference or even invoke a code representation (CE) to calculate certain information, particularly the size of the output at runtime and during the execution of the runtime kernel (RK). In some cases, when the size of the output to a segment depends on the input element values, all of the output configuration information may be calculated and stored by the runtime kernel (RK). Such a kernel may then directly signal / notify the code representation of subsequent segments of changes / updates to the runtime instance configuration information ( / type) before signaling for updates to the element values. In some cases, multiple such signals may be merged into one.

[0123] Each array instruction may exhibit its own semantics / behavior, and when multiple array operations are combined into a segment, a multitude of potential combinations of individual configurations of executable code arise. Those skilled in the art will recognize that the best configuration strategy / efficient configuration of the executable code (KC) generated for each segment may be selected based on and in accordance with each array instruction / operation contained within each array instruction supported by the language and compiler, or, where appropriate, by introducing segment boundaries to separate mismatched instructions.

[0124] For example, some array instructions require updating only one type of information to complete the configuration of their runtime instance as output. One example is the transpose function for changing the orientation of a matrix object. It (re)configures only the size (dimensional order / length and stride) information. All further configuration information is retained, including element values ​​and even memory references, and may be obtained directly from the runtime instance as input to the instruction. Here, the runtime kernel can be configured to perform such a copy, can be configured as a NOP (no operation), or even omitted. Note that this method works best in relation to "lazy copy on write" memory management of runtime instances that use reference counting, for example.

[0125] Alternatively, as another example, the sin() function, which calculates the sine of the element values ​​of an input array runtime instance, requires modifying the element value configuration information, but may inherit or even reuse the size information of the input instance. Here, the code representation for calculating the size of the output instance may be configured to perform such a copy, may be configured as a NOP, or may even be omitted.

[0126] Further input data may indicate information (often additionally) required to compute the desired configuration information, or may point to a source of such information. Often, further input data refers to a temporary output or buffer or memory area used to compute element values ​​without knowing (yet) the number of elements in the resulting output runtime instance, often containing size information of the output resulting from executing the runtime kernel (RK). Which source is referenced by the further input data is specific to the array instruction and the compiler / transpiler's particular implementation / design of the instruction as an array operation. In some cases, further input data indicates the (array or scalar) input parameters of the array instruction, e.g., axis / dimension index specifying the direction of operation of a particular array instruction, or (user / compiler / environment) configuration settings. Often, further input data may be omitted, and / or the corresponding information may be integrated / merged into other parts of the executable code (KC), e.g., as a constant expression, literal, or value.

[0127] In some cases, segment boundaries can be introduced to limit complexity when merging specific array instructions into a common segment introduces too much complexity to handle (by the compiler author or execution framework). The best (fastest or least power-efficient) execution result / most efficient use of hardware resources is often achieved by merging as many array operations as possible into a segment (according to the array instructions), and thus eliminating as many intermediate transient runtime instance results as possible. However, if segment boundaries (alone) can enable support for a particular array instruction, such separation is almost always preferable to forcing sequential synchronous execution of array instructions, especially when the processing is performed on the main thread.

[0128] Correspondingly, if, during the processing of the TCC or the construction of the executable code (KC), all input data to the runtime kernel (RK) is found to fully define their construction, the TCC typically begins executing the executable code (KC), particularly the runtime kernel (RK) for the output runtime instance of the segment, asynchronously. Here, the execution of the executable code, particularly the runtime kernel (RK), corresponding to the element value information computation is initiated by the TCC, and the execution of the runtime kernel (RK) is performed on a subunit separate from the one used to process the TCC.

[0129] Note that runtime kernel startup / scheduling only occurs when all input data / dependencies are already defined, accessible, and available.

[0130] According to Figure 1B, the only input array data structure to segment S1 is A. Here, we assume that A1 is the corresponding input runtime instance, which is already fully configured. Therefore, the runtime kernel (RK) is started asynchronously. The code representation (CE) is configured to begin completing the size calculation / configuration of B1 when the kernel has finished processing. Alternatively, the compiler may omit the code representation (CE) as a dedicated code portion and use the runtime kernel (RK) for the size calculation.

[0131] After the executable code KC has been created, configured, executed, and / or started, the execution of the TCC for segment S1 may be completed, and the execution of the TCC for segment 2 may be started on the same thread.

[0132] Segment S2 receives instance B1 of B created by the TCC of segment S1. At this point, the configuration of B1 is likely still incomplete. Here, we assume that B1 is partially configured and lacks a defined size, a memory reference to an element buffer, and element values.

[0133] The TCC for segment S2 creates and / or constructs the code representation for this segment. It creates runtime kernels for array operations (sin), properties of array input instances, and properties of the PU. Typically, such kernels are cached and reused later. Furthermore, the TCC creates a code representation for determining the size of the output array instance C1. Due to the type of array operation performed by segment S2, the size of C1 does not depend on the element values ​​of any input array. Therefore, the setSize() code representation does not need to consider these element values.

[0134] Knowing the size of the input array instance is sufficient to calculate the size of the output array instance for a segment. The code representation of segment S2 can calculate the size of its output (size C1) even for a partially constructed input array B1, as long as the size of B1 is constructed. Many implementations exploit this fact by reacting to changes in individual categories of configuration for the array instance. Here, the code representation may be executed when the size of input array B1 changes / is defined, but element values ​​are not necessarily required. Compare this to the more complex code representation of segment S1 above, where the code representation setSize() requires execution of a fully constructed input A1. Note that the configuration of inputs and their modifications that trigger the execution of executable code (KC) are properties of the array instruction from which the segment's array operations are derived, and therefore must be known to the compiler, for example, by AIMI (see below).

[0135] The coded representation of segment S2 is configured by the segment's TCC to react to changes in the sizes of input array instances A1 and B1. It is then configured to calculate the size of output array instance C1.

[0136] The runtime kernel for segment S2 is configured by the segment's TCC to react to the completion of the configuration of all input array instances A1 and B1. It computes the element values ​​of the resulting output array instance C1. Often, the kernel also allocates memory for storing the output element values, including configuring the memory reference configuration information for C1. However, some implementations perform the memory allocation for the segment's output after computing / configuring the output size from other parts of the segment's code, particularly from the TCC or code representation (CE). While error handling can be simplified in this way, this method is expected to perform best (in terms of execution time and / or power consumption) by allocating memory as late as possible before storing the results of the runtime kernel.

[0137] After the configuration of the executable code (KC) is complete (assuming the configuration of B1 is not yet complete), the execution of the TCC for segment S2 is complete, and processing may continue to the TCC for segment S3 on the same thread. The code representation of segment S3 is created / reloaded / configured to react to changes in the size configuration of its input array instance C1. It is configured to calculate the size of the output array instance C2 and store the size information in the array instance C2.

[0138] The runtime kernel for segment S3 is created / reloaded / configured to compute the element values ​​of output array instance C2, taking into account (using) the properties of the processing unit, based on the input array instance C1 and the array operations of segment 3. The runtime kernel for segment S3 is configured to react to the completion of the configuration of input array instance C1. Subsequently, the execution of the TCC for segment S3 is complete, stage 1 is complete, and processing may continue on the same thread with further instructions (not shown).

[0139] Please note that the element values ​​have not necessarily been calculated up to this point.

[0140] The description continues regarding processing performed on other threads. Such processing may (generally) be performed in parallel with the TCC processing of the segments, as described above. However, for clarity in this example, we assume that processing on other threads can only begin after the TCC of all three segments has been fully executed. Those skilled in the art will recognize that the implementation of the described method requires all segments, runtime instances and their configurations, as well as associated structures and functions, to be thread-safe and to be prepared for any simultaneous changes occurring at any point during processing.

[0141] A is the only input array to segment S1, and in this example, A1 is already fully configured when processing of segment 1 begins, so the runtime kernel starts processing immediately after the setup of the executable code for segment S1 is completed by the TCC and after the execution of the runtime kernel has been started by the TCC on a separate thread.

[0142] The runtime kernel computes the element values ​​of B1. After the kernel completes, the code representation setSize() is configured to respond to the completion of the kernel operation (and the completion of its input array instance configuration) and begins execution (however, in this embodiment, only instance A1 is already fully configured).

[0143] It should be noted that a memory reference to the element buffer of an output array instance is created by allocating memory for the output array instance on memory accessible by the processing unit (PU). Of course, the size of the memory region must be known for memory allocation. Therefore, the size of the output array instance must be known before allocation becomes possible / feasible. Generally, the executable code (KC) contains expressions / instructions for performing the allocation at runtime. Some implementations construct the memory reference by using code representations in a similar manner to (mostly after) the setSize() part. Other implementations construct the memory reference within the runtime kernel or after the kernel has finished executing. The latter method is particularly useful for array operations where the output size depends on the input element values, especially array operations involving conditional expressions. Other implementations may calculate the values ​​and sizes and perform the allocation, all within the runtime kernel or by code executed before and / or after the runtime kernel.

[0144] In some cases, temporary storage, dynamic memory, or variable-length data structures are used within the kernel before the final size of the output runtime instance is known and before the allocation of memory used as memory references in the output array runtime instance. Other implementations may calculate and allocate the maximum size of the memory region according to the inputs and array instructions / array operations within the runtime kernel, and if the size required for the output is smaller than the size of the allocated memory, they may reference only a portion of the memory region, for example, by constructing the runtime instance size information as the output accordingly.

[0145] The runtime kernel for segment S1 computes the element values ​​of B1 (possibly using a temporary memory buffer or by performing multiple iterations on the input elements), allocates memory for the elements of B1, and stores references to the memory in the configuration of B1. Alternatively, the code representation of segment S1 is executed after the runtime kernel has finished execution. It may compute and construct the size of B1.

[0146] When the size of B1 is stored, the configuration of B1 is changed. It should be noted that such a change triggers the execution of the code representation of segment S2. Therefore, the corresponding code representation is executed. Often, the code representation (CE) for calculating the size of the output is executed on the same thread (synchronously). The following explanation assumes this choice. In some cases, for certain array operations involving expensive size calculations, the execution of the code representation may be triggered on a separate thread. Often, this decision is implemented as a compiler switch.

[0147] The coded representation of segment S2 calculates the size of C1 and stores this information in the configuration of C1. However, a change in this configuration triggers the execution of the coded representation (CE) to calculate the size of the output of segment S3. This calculates the size of the output array runtime instance C2 relative to the TCC of segment S3. Therefore, as soon as the size of B1 is determined, the sizes of all dependent array instances are calculated and configured in a timely manner.

[0148] It should be noted that such "size transfers" are important to ensure that as much information as possible is available to subsequent segments. The executable code (KC) may perform optimizations based on the size information of the input runtime instance. For example, code specialization may be used for special input sizes such as scalar or empty arrays. Alternatively, some array expressions, such as constant arrays, may often depend only on the input size and therefore begin execution as soon as the size becomes available. Furthermore, on heterogeneous computing systems, this method integrates well with the method described in WO2018 / 197695A1, where device selection often depends on execution cost based on the size of the array runtime instance. Moreover, the earlier the configuration of the executable code and / or the preparation of data (e.g., copy operations) is started (e.g., asynchronously or on a worker thread), the more likely it is that the latency or delay associated with the preparation will be hidden or reduced by other operations or tasks running concurrently.

[0149] Once the runtime kernel has finished executing segment S1 and all its dependent (chained) code representations, stage 2 of the execution in this example is completed, as also shown in Figure 2A.

[0150] The runtime kernel for segment S2 is configured to begin execution when all input array instances (not shown) are fully configured. The runtime kernel (RK) for segment S2 receives input array instances A1 and B1. As soon as B1 is fully configured (by the runtime kernel and / or code representation of segment S1), the runtime kernel for segment S2 may begin execution. It computes the element values ​​of output array instance C1 and thus completes the configuration of C1. At this point, stage 3 is complete.

[0151] The runtime kernel instance for segment S3 begins execution when its (sole) input array instance C1 has finished configuring. It computes and stores the element values ​​of the output array instance C2 and completes its configuration. At this point, stage 4 is complete.

[0152] Generally, it should be noted that at least some instructions / tasks executed within the various stages described above may (automatically) run in parallel on a processing unit capable of executing multiple tasks / threads in parallel. This can be supported by configuring the runtime kernel and / or code representation to run asynchronously. Therefore, when a configuration change of an input array instance triggers the execution of a code representation or runtime kernel, the corresponding code is scheduled to run on a different thread / parallel computing resource / different subunit, rather than on the same thread / subunit.

[0153] In some cases, it is sufficient to use works-steal scheduling techniques to distribute the work of the runtime kernel and / or code representation, which are executed asynchronously, across multiple subunits.

[0154] The configuration of memory references for array instances typically involves the allocation of memory regions to store the array instance element values ​​of the output to each segment within the runtime kernel. For optimal performance characteristics, it is crucial to carefully consider when these allocations occur during the processing of segments and the code derived from them.

[0155] In a simple design, memory is allocated to the TCC portion of a segment. This offers the advantage of a simpler implementation for the related parts that support allocation. For example, memory for an array is often managed or allocated from a memory pool. By having all allocations performed from the main thread (core #1) in the TCC portion, the functionality of the memory pool is less restricted. For example, if the main thread is the sole source of memory allocation and deallocation, thread safety requirements are relieved.

[0156] However, when an allocation request cannot be completed (because a sufficiently large memory region cannot be found / is not available), the main thread must wait until memory becomes available from the unprocessed segments. Therefore, the limit on available memory imposes a limit on the number of segments that the main thread can prepare in advance. The larger this number, the more segments are considered and prepared. The more segments are considered (concurrently with and before processing / executing their runtime kernels), the more parallel potential / independent segments can be identified, scheduled on different threads, and more parallel computing resources / subunits can be utilized.

[0157] One particularly efficient way to increase the number of segments prepared (and thus increase the recognition of independent segments, and thus improve parallel execution) is to reduce the “lifetime” time span of the array instance element memory storage being allocated / reserved. In a more efficient design, the allocation is performed immediately before the runtime kernel is about to execute and / or on the same thread on which the runtime kernel is scheduled. Those skilled in the art will recognize the fact that allocating memory may still impose a blocking wait on the corresponding runtime kernel thread. However, due to the way the segments are connected to each other, the present invention ensures that race conditions may not occur by blocking a runtime kernel thread before completion, and that there may not be any other segments / runtime kernels that wait for and depend on the output produced by the blocking runtime kernel and then block the blocked thread (deadlock).

[0158] Typically, memory allocation is performed from the memory pool or memory manager / virtual memory manager as part of executing the executable code (KC), specifically within the runtime kernel code. Note that typically, the runtime kernel (RK) code is executed once all input runtime instance configuration information is complete and available to the executable code. Therefore, all the information necessary to determine the output runtime instance memory region size is also available at this point.

[0159] Figures 3A and 3B typically illustrate Method 2000, which is similar to Method 1000 described above. However, Method 2000 shows exemplary user program code UC having five lines of supported array expressions. In this case as well, each line may be translated by the compiler / transpiler in accordance with this disclosure into one corresponding segment S1-S5 in segmented program code TCC. The right column of Figure 3A shows a dependency graph of these expressions.

[0160] The thick ellipses in the graph represent segment codes. Inputs to a segment are marked by a name consisting of the respective array data structure as input to the corresponding array instruction, and a number indicating the array runtime instance index of this array data structure during the execution of an exemplary segmented program code.

[0161] Runtime instance B1 enters segment S1, which corresponds to the array instruction "D=B-2". As a result, runtime instance D1 is produced as the output of segment S1. D1 is further used as input to segments S3, S4, and S5. Note that segments S1, S3, S4, and S5 build one branch of a common dependency graph, and segment S2 depends on B1 and builds another branch. Furthermore, note that subsequent or previous instructions / segments may exist, and the creation / reception of the results of array instructions is not considered here for clarity.

[0162] Payloads and / or workloads linked by expressions within sibling branches of a dependency graph are independent of each other's results and can therefore be executed in parallel or concurrently. Furthermore, segments 3 and 4 reside within sibling branches, are independent of each other, and can also be executed in parallel.

[0163] The following describes how this disclosure enables the full utilization of the parallelism potential of common array algorithms (as demonstrated) by scheduling independent segments on parallel computing resources / subunits according to their positions in the corresponding dependency graph.

[0164] The following explanation and Figure 3B are intended, in particular, to illustrate the scheduling of individual parts of segment code. The parts of a segment that are executed as part of the segmented program code are named "TCC" with a number representing the segment index appended. Typically, all of TCC1 through TCC5 are executed on the main thread. Similar names are used to identify the executable parts of each segment, with "CE" appended by the segment index representing the code representation (CE) of the corresponding segment. "RK" appended by the segment index represents the (configured) runtime kernel of the corresponding segment. For example, RK4 is intended to indicate the runtime kernel of segment 4.

[0165] When the TCC of segment 1 (TCC1) is executed on core #1 (as the main thread), it is assumed that all input arrays to segment 1 are already fully configured. The array instructions implemented in segment 1 allow the size of the output to be calculated while ignoring element values. Therefore, the code representation CE1 that calculates the size in the executable code (KC) is executed synchronously on the main thread.

[0166] Furthermore, TCC1 constitutes the runtime kernel. In this case as well, since all input arrays are prepared, the kernel is started in a separate thread. Often, a worksteal thread scheduler is used to select a free subunit and enqueue the kernel.

[0167] TCC2 is processed while the kernel is still running. In this case as well, all inputs are prepared (fully configured), and therefore CE2 runs on the main thread, and RK2 is enqueued on another thread (core #3 in this case). Note here how the array instructions for both segments are executed in parallel.

[0168] Continuing with TCC3, we see that the sizes of all inputs have already been configured. Note that the size of D1 has been configured (synchronously) by the main thread (even though its value is not yet ready). Therefore, in this case as well, CE3 runs on the main thread. The kernel for segment 3 is configured. The kernel is not started yet because the value of D1 is not yet ready. It will be started once the configuration of D1 is complete. Until then, RK3 is not enqueued to any thread. It waits in the background (not shown) to be started / scheduled / enqueued. Similarly, TCC4, which is the TCC for segment 4, runs on the main thread, creating, configuring and starting CE4, and creating and configuring RK4 for asynchronous execution once the element values ​​of D1 are fully configured.

[0169] Meanwhile, assume that segment S1 (RK1) has finished execution. RK3 and RK4 are configured to receive a signal upon completion of D1. Generally, as is the case with most concurrent operations, the order in which tasks or runtime kernels are completed on individual subunits is not determined and may vary in any way. Now, assume that the signal arrives at RK3 first. Therefore, kernel RK3 for segment S3 is configured as a continuation of RK1. Thus, it runs on the same thread as RK1.

[0170] Simultaneously with all inputs to segment S4, D1 is also fully configured. RK 4 is a second runtime kernel that receives signals regarding the completion of D1. Thus, RK4 is executed asynchronously, for example, by using a worksteal thread scheduler. Here, it is enqueued to subunit / thread core #4 and executed concurrently with RK3.

[0171] Therefore, RK 3 uses the same thread as RK1, but RK4 is enqueued in a different thread. This assignment is arbitrary and, as is clear, depends only on the order in which completion signals arrive at each runtime kernel.

[0172] Meanwhile, the main thread is processing TCC5 and CE5. In this case, the size of the output of TCC5 is known, but RK5 must wait for the outputs of RK4 and RK3 (F1 and G1, respectively). Therefore, RK5 is configured to wait to be signaled by these runtime kernel outputs after their completion. We assume that RK3 will finish later than RK4 due to its higher workload. However, this is not guaranteed, of course. In a real-world scenario, when asynchronous resources are involved, it is more likely, or at least possible, that one of the kernels will finish earlier. In this example, RK4, having finished earlier than RK3 and RK5, is signaled for the completion of its output runtime instance G1. Thus, the output F1 of segment 3 is the last to complete. RK5 is signaled for the completed configuration of F1, which is the last input to RK5. Furthermore, RK5 is the first signal sink for F1. Therefore, RK5 is enqueued into the subunit on which RK3 was running. See below for a detailed explanation of these scheduling decisions.

[0173] A typical implementation creates and sends a completion signal as part of the runtime kernel (RK). Many embodiments use runtime task data structures (RTDs) to store (reference) runtime instances of input and output data structures and / or executable code (KC, CE, RK), often asynchronously, to manage the execution, signaling, and completion of executable code (KC, CE, RK) for various parts of the configuration of the runtime instance as output to segments. See Method 3000 for details.

[0174] In the examples in Figures 3A and 3B, the sequential array code is parallelized and distributed across four cores / subunits. The majority of the segment (additional) management code runs in parallel with the (optimized) payload portion / runtime kernel. Parallelism between array instructions is identified and utilized by efficiently distributing those payloads / kernels across three worker threads. The described method often achieves a near-optimal rate of increase, depending on the parallelism possibilities (see Amdahl's Law) and the availability of subunits.

[0175] To achieve minimum execution time and / or minimum power consumption, the number of threads / tasks scheduled for parallel execution may not (significantly) exceed the number of segments that can be executed independently at runtime. It will be apparent to those skilled in the art that this number relates to the number of parallel-executable branches in the dependency graph corresponding to the primary program code or segmented program code (UC / TCC). For example, in the exemplary dependency graph in Figure 3A, segments S3 and S4 may be executed concurrently. Segment S2 may be executed in parallel with any of the other segments S1, S3, S4, or S5.

[0176] However, to prevent excessive thread utilization / allocation and further thread overhead, segments within a single branch of the dependency graph may (re)use the same thread for the execution of their runtime kernels (or other asynchronous configuration tasks). Using new threads for scheduling and / or executing executable code (CE), particularly for executing runtime kernels (RK), is only necessary in specific cases / configurations / situations identified and described in more detail below.

[0177] Figure 4 is intended to illustrate a particularly advantageous process for making decisions regarding scheduling executable code (KC) from a segment. Figure 4 shows exemplary steps for choosing whether to execute the executable code created and / or configured by the segmented program code (TCC) of the segment synchronously or asynchronously. Alternatively, depending on the configuration state of the input runtime instance to the segment, the executable code may be configured but not executed (yet).

[0178] The decision process begins with executing the segment's TCC. All configurable information types are iterated over for all runtime instances, assuming that size and element values ​​are configurable. For each information type (size, value), it is determined whether information of that particular type is (already) configured for all input runtime instances received by the segment. If all instances are configured accordingly, the executable code is executed by the segment's TCC.

[0179] Some runtime instance configuration information is relatively efficient or simple to compute. For example, calculating the size of an output to a segment based on the size configuration of all inputs to the segment often requires O(m) effort, where m is the number of inputs configured for the segment. Such types of information are often classified as "not expensive" and are configured and executed synchronously.

[0180] In contrast, elemental value configuration information of the output (often computed by the runtime kernel (RK)) requires at least O(n) effort, where n is the number of elements in the output. This type of information is often classified as "expensive." Therefore, the corresponding executable code (KC) is configured to run asynchronously and execution is initiated. Note that this initiates execution of a "segment chain" or set of segments corresponding to the segments in the branch of the relevant dependency graph, and the newly allocated subunit / thread is typically used to execute further, typically all subsequent segments in this set.

[0181] If the current information type has not yet been configured for at least one input runtime instance, the executable code (for the current information type) is configured by the segment's TCC (see IV in Figure 1A).

[0182] In some cases, this configuration occurs only partially. In this case, the configuration of the executable code is completed when the corresponding information type for the last runtime instance as input to the segment is updated or defined. Note that such completion is typically performed not by the main thread / subunit, but by the thread / subunit that executes the code causing the update. See Figure 5 for an example.

[0183] Substructuring allows for optimizations to the executable code to take into account (use) the updated information. For example, the executable code may be specialized for at least some size information, such as the number of dimensions or stride information.

[0184] After the current iteration for a particular information type has been processed and the corresponding decisions and / or actions have been performed, the TCC implementation proceeds to the next information type or terminates after the last information type.

[0185] Figure 5 is intended to illustrate a particularly advantageous decision tree for scheduling executable code (KC) to be notified / signaled about the completion of the configuration of one of its input runtime instances. In the figure, exemplary runtime instance H3 and segment S17 are used to facilitate reference to individual blocks. It is assumed that the current decision is performed by executable code (KC) that is part of the executable code of the current segment, in particular the type of configuration information being updated for runtime instance H3. Furthermore, it is assumed that the configuration information of H3 has been updated by executable code associated with segment S17 (not shown). Alternative embodiments may perform these decisions elsewhere / by other code without affecting the general functionality of this disclosure.

[0186] The decision begins with updating the configuration information of H3. The executable code of the current segment corresponding to the configuration information type (e.g., a code representation to calculate the size, or a runtime kernel to calculate element values) is invoked by the executable code (KC) of segment S17. If the information type currently being updated is missing or undefined for any other input to the current segment, an update signal is recognized (e.g., by a counter, especially a reference counter), and no further action is required.

[0187] Otherwise, the configuration of the executable code corresponding to the current information type is completed. If the executable code is already fully configured, this step may be skipped.

[0188] Next, a decision is made on how to execute the (fully configured) executable code. If the type of configuration information is classified as "not expensive" (see Figure 4 above), the executable code is often executed synchronously, immediately, and / or by the current thread / subunit.

[0189] Instead, if the configuration information is classified as "expensive," it is determined whether a continuation is configured for the executable code in S17 that updated H3 (hereinafter referred to as the sender code). The continuation is a task or executable code that should be executed after the current task or executable code has finished executing. The continuation may be configured by providing a method or function reference to the task or executable code that is currently executing. For example, an object instance used to signal the current executable code that the H3 configuration information is complete may be configured by the current executable code as a message object to notify the sender (S17) about an executable code reference to be executed (here, a "self" or "this" instance).

[0190] If it is found that no continuation has been configured for the sender code, the current executable code is configured as a continuation to the sender code. Otherwise, the current executable code is configured for asynchronous execution on another subunit / thread and execution begins. Using a continuation (instead of executing the current executable code synchronously and immediately) is recognized by those skilled in the art as changing the order of execution of the current executable code and potentially further executable code (of which further segments also receive updated information). The continuation allows for the signaling of such further segments in a timely manner and before the (expensive) computation of the current executable code is completed.

[0191] Note that in this embodiment, only the current information type is used for each call to the executable code (KC).

[0192] Some systems using this method may utilize multiple main threads to execute multiple versions / instances / parts of segmented program code. Often, programmers utilize parallelization tools or methods provided by runtime systems / operating systems or frameworks used to distribute algorithmic parts across multiple computing cores in a multithreaded environment / system / framework. In this case, individual main threads typically access the same resources (thread pool, memory pool, or memory manager) as a single main thread, and the methods described herein apply similarly to each main thread.

[0193] An exemplary system configured in accordance with this disclosure often utilizes a single thread or subunit of a multicore processor system, the main thread, to execute segmented program code TCC. In favorable circumstances, a sequence of array instructions within the user program code UC is configured to result in a sequence of segments within the segmented program code TCC of considerable length, without interruption by non-segmented code. This situation is particularly promising for achieving efficient resource utilization by this method. This often enables better resource utilization compared to other execution methods.

[0194] While the main thread executes the segmented program code TCC for adjacent segments, a runtime instance as output is partially constructed, and executable code is constructed to complete the construction. The main thread then proceeds to the next segment in the sequence. Thus, generally, progress is made on the main thread when processing a sequence of segments without spending time calculating element values ​​and / or waiting for input array configuration information, in particular without waiting for either or both input array size information and completed configuration information.

[0195] The configuration of runtime instances of array data structures is typically completed on other threads in a multicore processor system. Those skilled in the art will recognize that the overall workload of a processing segment with array instances / data is distributed across parallel computing resources, and both the parallelism potential of the algorithm and the parallelism capabilities of the processing units are used efficiently.

[0196] Regarding Figures 6A and 6B, the computer implementation method 3000 will be explained.

[0197] Method 3000 describes embodiments in which fewer or no adoptions are performed at runtime. Therefore, Method 3000 is expected to exhibit reduced performance for many workloads when compared to the more advanced Methods 1000 and 2000. However, Method 3000 can still achieve efficient parallel execution by leveraging the parallelism and / or pipelining of array instructions, and therefore results in better resource utilization efficiency compared to conventional methods. Implementation of Method 3000 is often easier to implement, particularly when array instructions exceed a certain complexity, when a more optimized implementation of the runtime kernel (RK), or at least an implementation specialized (for hardware and / or runtime data), is not (yet) available or is not available for all processing units or subunits, when only a single subunit is available and configured to execute a segment, and / or when intermediate results are computed.

[0198] In block IIIa of the flowchart shown in Figure 6A, which has blocks IIIa and V, each of which can correspond to one or more processes or steps executed in the context of a single segment, program code having a sequence of array instructions as shown in Figure 6B begins execution on the processing unit (core #1).

[0199] Typically, array instructions are represented in program code, here in user program code (UC), by individual functions and / or operators from a host language, often a domain-specific language (DSL) embedded in a general-purpose language (GPL), as C# from the .NET framework, by the language construct of a particular computer algebra system, or as language elements from a scripting language such as ILNumerics, Matlab®, R, or numpy. Figures 6A and 6B show exemplary program code consisting of two lines of array expressions as user program code (UC) (here from the Matlab® numerical language). The first line takes the first column from (assumed) matrix A, calculates the Fast Fourier Transform (fft) of the resulting complex-valued element vector, and stores the real part of the elements in the first column of (assumed) matrix B. Similarly, the second line calculates the imaginary part of the fft from the second column of A and stores it in the second column of B.

[0200] The selection of these array instructions is, of course, not limited in any way. All common array expressions are supported by the methods described herein. Typically, such expressions, together with further expressions or instructions (often including non-array instructions from the host language when the method is implemented as a DSL), form an algorithm or user program.

[0201] At runtime, array instructions are typically executed on a host processor processing unit (core #1) that includes a hierarchy of parallel processing elements. For example, array instructions can be executed on a CPU exhibiting multiple cores, or on a node in a computer cluster having multiple nodes as processing subunits. Generally, each array instruction from a sequence of array instructions is initiated on the main unit, or by the main thread of the host processor, as part of an executable program containing the array instruction or code derived therefrom (e.g., interpreted code or JIT compiled code). The terms “main thread” and / or “node” are typically intended to describe the entry point to the execution method disclosed herein. Hereafter, the term “main thread” will be used even if the thread / node initiating the execution of the array instructions is not the same as the thread / node used to start the program.

[0202] According to embodiments of Method 3000, array instructions are implemented in segments of executable program code (S1-S5 in Figure 6B), where, for example, each array instruction may correspond to an individual segment implementation. Typically, such segments include a setup code (SET) portion and a runtime kernel (RK) portion, and are referenced directly or indirectly by user program code (UC). Most implementations implement the segments by using the same programming language, technology and / or framework used to implement, define and provide the array instructions. For example, a segment corresponding to the sin(A) function may be implemented simply and exemplary as a function that calculates the sine for all element values ​​of an array data structure referenced by variable A as input to the sin() function (as similarly provided in the ILNumerics, Matlab, or numpy languages), and exposes the same interface (API) as the well-known sin() array instruction (to those skilled in the art of scientific computing / numerical languages).

[0203] Alternatively, the segment components (SET and RK) can be directly implemented in the sin() array function of an existing DSL or numeric language runtime implementation or framework, and thus the sin() function performs the execution method described herein (exemplarily with respect to further segments). Generally, in such implementations, the runtime kernel (RK) may mainly consist of a conventional sin() function implementation, and the configuration code (SET) is added to the sin() function as a further implementation layer. Often, the configuration code (SET) and / or runtime kernel (RK) perform tasks at least substantially similar to the (dynamic) segment implementation in segmented program code (TCC).

[0204] Alternatively, as described for compiling and executing methods 1000 and 2000, program segments can be implemented more dynamically, for example, by using more (runtime) information and / or a compiler or similar technique to replace array instructions from user program code (UC) or code derived therefrom with each segmented program code (TCC) of the corresponding program segment.

[0205] In block IIIa, when the program counter reaches the array read access instruction A(:,1) from the right side of the array expression on the first line of the example user program code (UC) at runtime, the setup code (SET1 in Figure 6B) is started on the main thread / first subunit core #1 of the processing unit (CPU in Figure 2B). Block IIIa of method 3000 may also be considered as block III, in which the execution of program code (derived from user program code (UC)) is started on the first subunit (core #1) of the processing unit.

[0206] Here, the array read access instruction corresponds to segment S1, which is used in place of a normal array read access to initiate the execution of the configuration code SET1 for segment S1. The configuration code SET1 (in this example) initiates the first runtime task data structure RTD1.

[0207] When configuration code SET1 processes a call to segment S1, it receives runtime instance A1. It then receives the scalar index value "1".

[0208] A runtime instance A1 of array data structure A is stored together with RTD1. Note how such storage is performed so that the information of A1 as input is available to the kernel code (KC) when execution starts (asynchronously). The same purpose is achieved when only a reference to A1 is stored. Alternatively, A1 or a reference to it may be stored in a table, database, file, or similar data structure, and the key, pointer, or reference to the stored instance is made available to the kernel code (KC). The same idea is valid for storing data structures as output to segments, for example, storing further arguments to array instructions as scalar values, and storing runtime task data structures (RTDs).

[0209] Typically, runtime instances of array data structures are stored without copying the element values ​​of A1 for performance reasons. Generally, each argument to a segment must be stored and made available to the kernel code unless the kernel code is configured to produce the same result when run without arguments. For example, constant expressions, constant literals, and / or default or missing optional arguments may be omitted and may not be stored with RTD1.

[0210] According to one embodiment, method 3000 stores a utilization counter having A1, which is incremented here by SET1, and tracks the number of segments that runtime instance A1 is scheduled for (asynchronous) utilization.

[0211] In block IIIa (Figure 6A), the further action performed by SET1 (Figure 6B) is to create, store, and return a runtime instance of the array data structure (R1) as the output of segment S1 (this often replaces or performs the array instruction "GetRange" from the UC). Often, this runtime instance of the array data structure is only partially configured to the extent that neither size nor memory references are configured. Therefore, the runtime instance initially lacks size and element values.

[0212] Typically, the runtime instance R1 as output is partially constructed by SET1 by associating a reference to RTD1 with R1. Preferably, dedicated runtime task information, such as fields, attributes, or properties of the runtime instance data structure R1, is used to store or associate a reference to RTD1 with R1. However, other options exist for associating this information with R1. This allows, among other things, subsequent code / sequential segments to be aware of the pending state of the runtime instance. Furthermore, it allows inspecting the processing state of RTD1 and reacting to changes in the state of RTD1, particularly the completion of kernel code (KC) execution.

[0213] The kernel code (KC) for a segment typically contains code that is similar to, or even identical to, the conventional implementation of the corresponding array instruction. For example, the kernel code KC1 for S1 performs an array read access from a given array instance A1 and returns a subarray of A1 according to the provided dimension index / index range. Since kernel code KC1 serves the same purpose, any existing correct implementation may be used to implement S1. However, any appropriate implementation of the kernel code (KC) must allow concurrent execution by multiple threads / nodes or processes (thread safety). Furthermore, all data structures and set code (SET) should also operate in a thread-safe manner.

[0214] One advantage of Method 3000 of the present invention over conventional methods is that the proposed method allows for the reuse of existing array instruction implementations for efficient parallelization as kernel code.

[0215] While it is relatively straightforward to (re)use existing array instructions as kernel code in Method 3000, typically, the only state change associated with such an implementation is completion. For example, the array read access instruction GetRange typically (in conventional implementations) performs all the processing steps necessary to complete the instruction with each call to GetRange(). Therefore, when this method is used as kernel code, runtime tasks cannot notify other segments of the completion of individual parts of the array instruction (e.g., completion of size configuration of the output of GetRange(), completion of output allocation, completion of element value configuration, etc.). Only the completion of the overall configuration of the runtime instance resulting from the array instruction GetRange() is signaled to subsequent segments.

[0216] The performance of Method 3000 can be further improved by implementing finer-grained segment kernel code (KC), more specifically by, for example, calculating the output size individually from element values. After the size configuration is complete, subsequent segments can signal state changes and begin using the modified parts sooner for their processing. For further details, refer to Method 1000 and the explanation in Figures 1A and 1B.

[0217] It is recognized that all methods 1000, 2000, and 3000 can be implemented in the same system, particularly iteratively. For example, a system that (initially) implements method 3000 for a subset of common array instructions can iteratively extend to support more efficient features, including the elimination of transient results (by adding multiple array instructions to a segment), finer-grained pipelining (by dividing the configuration of runtime array data structures into multiple smaller components and processing / signaling the completion of each component individually), and kernel code optimization and specialization (for example, by specializing and optimizing runtime kernel and / or executable code using runtime instance and / or hardware configuration information, and / or by compiling "just in time"). Furthermore, the set of supported array instructions can also be iteratively increased.

[0218] Returning to block IIIa in Figure 6A, the configuration code checks the runtime task information of the runtime instances as input to segment S1. In an exemplary embodiment, we assume that only input A1 is fully configured when SET1 starts execution. Thus, since all runtime instances as input to the segment are configured, the kernel code KC1 for segment S1 is started on a separate thread / another node or process for asynchronous execution (V).

[0219] At this point, the processing of SET1 is complete, and a partially configured runtime instance R1 is returned from SET1 as output / return value.

[0220] Next, the program counter reaches the "fft" array instruction, which is executed as segment S2 in Figure 6B (IIIa in Figure 6A). The setting code SET2 from S2 is called and provides R1 as input.

[0221] SET2 instantiates the runtime task data structure RTD2, stores a reference to R1, and increments the usage counter for R1. Furthermore, since R1 is (still) only partially configured, the runtime task information of R1 is used to configure the kernel code of segment 2, referenced by RTD2 as KC2 in Figure 6B, so that it starts processing (V) when the configuration of R1 is complete (here by KC1 referenced by RTD1).

[0222] Furthermore, a partially configured runtime instance R2 is created (in a thread-safe manner) and stored (associated) with RTD2 as output of segment S2. The runtime task information for R2 is configured to refer to RTD2. R2 is returned from SET2, and processing of SET2 is complete.

[0223] Continuing to describe the actions possible on the main thread, the execution of the configuration code SET for subsequent segments (S3-S5) (IIIa) may occur at this point, when kernel codes KC1 and KC2 have already started processing on a parallel execution system (multithreaded, multiprocessor) (V). Details of (V) are described below.

[0224] Next, the array instruction "real" from the program code is processed. The configuration code SET3 for segment S3 is initiated (IIIa). The actions performed are the same as for SET2: R2 is remembered and marked as used, KC3 is configured to begin processing (V) when the configuration of R2 is complete, and finally, the partially configured runtime instance R3 is returned as the output of segment S3.

[0225] The next array instruction from UC is "SetRange". This function receives runtime instance R3 as input to segment S5. Furthermore, it receives runtime instance B1 as input and output to the segment, and a further argument "1" as a scalar input to the segment. Note how array data structure B as output to the segment exhibits input / output semantics. That is, runtime instance B1 is used by SetRange() to read its configuration and to modify its configuration.

[0226] The configuration code SET5 begins execution on the main thread (IIIa). It receives R3 and B1 as input to the segment. It assumes that R3 is (still) only partially configured and B1 is fully configured. Therefore, KC5 begins execution on an additional thread / node when the configuration of R3 is complete.

[0227] Furthermore, runtime task information for runtime instance B1 as output to the segment is marked in reference RTD4. Runtime instance R3 as input to the segment is stored in or associated with RTD4.

[0228] Before the output is stored / associated with the RTD4 in a segment (i.e., the target of the modification in the kernel code when the kernel code was started), the runtime instance utilization counter is evaluated. It tracks the current number of (scheduled) utilizations of the runtime instance for access by other segments. The utilization information informs of further accesses and / or further segment / runtime task data structures (RTDs) to which the runtime instance may be stored or associated for processing.

[0229] Typically, when the usage counter evaluates to a value indicating that further segment kernel code is accessing / will access the configuration of runtime instance B1 as output to segment S5, a copy of the configuration of runtime output instance B1 is executed, and copy B2 is stored or associated with RTD4 in place of B1. Note that if the source runtime instance of the copy is only partially configured, and the configuration is completed or updated, such a copy may be executed asynchronously.

[0230] However, in this example, we assume that B1 has not been used as input to any previous segment. Therefore, no copy is performed, B1 is marked as used by using the used mark, marked as output by using B1's runtime task information, and stored / associated with RTD4. Then SET5 completes and B1 is returned to the segment as output.

[0231] Copying based on usage marks serves to ensure program accuracy even when the order of write and read access to the runtime instance's configuration may change due to asynchronous execution. Another way to ensure accuracy in this situation is to wait to read the segment / kernel code instead of copying. However, copying may be performed as late as possible, for example as a lazy copy-on-write, or only immediately before accessing the output configuration for writing and after re-evaluating the usage counter, and especially efficiently.

[0232] Therefore, evaluating usage counters often complements the ability to enable efficient copy-on-write allocation schemes (for example, in kernel code (KC)).

[0233] It should be noted that the described method ensures accuracy regarding the order of multiple write accesses to runtime instances, even when the majority of multiple segments are executed asynchronously and concurrently, by utilizing runtime task information of runtime instances of array data structures used as inputs and / or outputs to segments. To enable this guarantee, the order in which kernel code execution for multiple segments begins is determined / controlled by SETs designated to run on the main thread, and therefore follows the original order of array instruction processing as defined in the user program code (UC).

[0234] Therefore, the configuration code (SET) configures the kernel code (KC) to begin execution when all configurations are fully configured, typically immediately after the last partially configured runtime instance as an input or output to the segment. Once all runtime instances as inputs and all runtime instances as outputs are fully configured (assuming pending configurations by further segments), the SET begins execution of the kernel code (KC) for each segment. For questions regarding which subunits handle the kernel code, see Figure 5 and its corresponding explanation.

[0235] The execution of the second line of the array instruction from the exemplary primary user code (UC) in Figure 6B, and the corresponding execution steps in Figure 6A, are the same as the first line of the UC. The new instruction utilizes similar or even the same segments, with one exception: the imag() array instruction is used instead of the real() array instruction, and therefore segments S4, SET4, and KC4 are used instead of S3, SET3, and KC3 from the first part of this example.

[0236] Note that in the exemplary embodiment, RTD5 is instantiated by SET1, which creates a new runtime instance R5 as the output data structure to segment S1. Since only runtime instance A1 as the input data structure to segment S1 is fully configured, kernel code KC1 is started (after further configuration code is executed as described above). Often, KC1 referenced by RTD5 is started on a separate thread or process immediately after or together with KC1 referenced by RTD1.

[0237] When the execution of the SetRange array instruction from the second line of UC in Figure 6B begins with the configuration code SET1, it receives a partially configured runtime instance R6, starts RTD8, stores or associates R6 with RTD8, and references kernel code KC5 (corresponding to SetRange) within RTD8. Before storing / associating B1 with RTD8, the utilization counter of B1 is evaluated.

[0238] In an exemplary embodiment, runtime output instance B1 further functions (is configured) as an input runtime instance to KC5 referenced by RTD4. Often, at this point in execution, kernel code KC5 referenced from RTD4 (from the SetRange array instruction from the first line of UC) is not yet complete. Therefore, runtime instance B1 as input and output to segment S5 is still used by KC5, and the utilization mark corresponding to this use is not yet cleared. Thus, when SET5 processes B1 as a partially configured input / output parameter, the utilization mark is often set, or the value of the zero-based utilization counter becomes greater than 1. This causes SET5 to create a (at least shallow) copy of B1, B2, and to store / associate B2 with RTD8.

[0239] Note that the runtime task information stored / associated in (pending) B1 still refers to RTD4. A copy of B1, B2, is typically performed by copying the element values ​​of B1 and further (pending) configuration information after each piece of information in B1 has completed. Thus, in some embodiments, the runtime task information stored / associated in B1 is used to schedule the copy operation, where the copy is performed when KC5, referenced by RTD4, has completed its execution.

[0240] In some embodiments, asynchronous copy operation can be eliminated (at least substantially) if the copy is triggered only by a previous (pending) write access to the runtime instance. Such implementations may rely solely on utilization marks to track read access to the runtime instance. Furthermore, they may perform the copy operation by copying only a reference to the (pending or complete) source configuration information of the runtime instance, delaying the copying of element values ​​to a later processing point. However, this often requires multiple reference counting layers for runtime data structures and thus results in a more demanding implementation. The kernel code KC5 is configured by SET5 to begin execution when the last partially configured runtime instance as an input or output is fully configured, in this case R6 is fully configured.

[0241] Note that the array instruction / kernel code referenced by RTD1-RTD4 is often not executed in parallel due to data dependencies. RTD1-RTD4 corresponds to a single branch in the UC's (acyclic) dependency graph. The same applies to RTD5-RTD8, which realize other sibling branches from the same dependency graph. The method described allows such sibling branches to be executed simultaneously and in parallel automatically, often on worker threads or worker nodes, while relatively small SET configuration code performs management tasks on the main thread.

[0242] The usage counter described in Method 3000 may be implemented to execute the segmented program code described in Methods 1000 and 2000.

[0243] Typically, the main thread sequentially processes multiple, often numerous, subsequent array instructions / segments from the program code (UC). Generally, the processing of segmented program code and / or configuration code (SET) is independent of (and / or often executed concurrently with) the processing of the runtime kernel (RK) and the kernel code (KC) for each segment. Typically, the main thread stops when user program code (UC) requests access to configuration information, such as element values ​​of a runtime instance, and the runtime instance is (still) only partially configured. Other embodiments limit the number of segments processed concurrently, for example, by using hard limits (a fixed maximum number of concurrent segments, e.g., 1000) or soft limits, where the maximum number is determined based on runtime information such as the utilization rate of processing (sub)units or the size of memory available to the process.

[0244] According to an embodiment of a computer implementation method for executing user program code, which includes a sequence of array instructions implemented as (configured) segmented program code, each including a kernel code and each configuration code, each kernel code includes an array operation corresponding to one of the array instructions, and each program segment is configured to receive, as its respective runtime input, a partially configured runtime instance of a respective array data structure capable of storing multiple elements of a respective common data type, the method typically includes the steps of: starting the execution of a configuration code of one of the program segments on a first subunit of the processing unit; partially configuring a runtime instance of an array data structure as the runtime output of each array instruction; and executing the kernel code of one of the program segments on the processing unit when the configuration of the runtime instance received as its respective runtime input is complete and / or updated.

[0245] According to embodiments that may be combined with other embodiments described herein, a computer implementation includes the step of receiving primary program code, in particular user program code such as user-written code or a substitute thereof. The primary program code includes a sequence of array instructions, each of which is configured to receive as input an array data structure capable of storing multiple elements of each common data type. Segmented program code executable on a processing unit including several subunits is generated from the primary (user) program code. Thus, the segmented program code may also be considered secondary program code. The segmented program code (secondary program code) includes program segments, each containing an array operation, the array operation corresponding to an array instruction. Each (generated) program segment is configured to receive (at least) a partially configured runtime instance of each array data structure as its respective runtime input. Furthermore, each program segment is configured to consider, in particular consider, and / or use information from each partially configured runtime instance to partially configure each executable code containing each array operation in order to partially configure the runtime instance of the array data structure as the runtime output of each program segment. The executable code is configured to update the configuration of the partially configured / configured runtime instance as runtime output for each program segment. Execution of the segmented program code begins on the first subunit of the processing unit. When the configuration of the runtime instance received as each runtime input is complete and / or updated, the configuration of each executable code is complete, and when each executable code is only partially configured, each executable code is executed on the processing unit.

Claims

1. A computer implementation method, A step of receiving user program code that includes a sequence of array instructions, each of which is configured to receive an array data structure as input that can store multiple elements of each common data type. The step is to generate segmented program code executable on a processing unit including several subunits from the user program code, wherein the segmented program code includes program segments each containing an array operation, and the array operation corresponds to the array instruction. Each of the aforementioned program segments is: Each is configured to receive a partially configured runtime instance of its respective array data structure as its runtime input. Each of the aforementioned program segments uses information from its respective partially configured runtime instance, A runtime instance of the array data structure is partially constructed as the runtime output of each program segment. Steps include: configuring each executable code, which includes each array operation, to at least partially constitute the executable code, and configuring the executable code to update the configuration of the runtime instance, which is partially configured as the runtime output of each program segment; The steps include: starting the execution of the segmented program code on the first subunit of the processing unit; When the configuration of the runtime instance received as each runtime input is complete and / or updated, Steps to complete the configuration of each executable code, When each piece of executable code is only partially constructed, The steps include: executing each executable code on the processing unit; A method that includes this.

2. The configuration of each runtime instance includes size information, memory references and / or element values, and the configuration of each runtime instance further includes storage location information, storage layout information and / or element type information, and at least one of the size information, memory references, storage layout, element values, storage location and element type information of the partially configured runtime instance is undefined, uninitialized or missing, and the configuration of the runtime instance as runtime output is updated, completed and / or defined by the user program code, the segmented program code and / or in particular by the executable code, and / or each part of the configuration of the runtime instance as output is updated or defined by individual calls to the executable code, and / or the individual calls execute individual parts of the executable code, for example, on separate subunits, the method according to claim 1.

3. The method according to claim 1, wherein when the configuration of each partially configured runtime instance is updated, in particular when at least one of its size information, memory references, storage layouts, and element values ​​is updated, the configuration of each executable code is completed and / or the respective executable code is executed on the processing unit.

4. The method according to claim 1, wherein the subunits are at least functionally identical, each subunit has access to shared memory for the processing unit, the processing unit is a CPU having a multi-core or many-core CPU or GPU, and each subunit is a kernel or user thread, management thread, task, fiber or process or compute node, or is accessed or controlled by or corresponds to them.

5. The method according to claim 1, wherein each executable code includes at least one of several subunits, typically each executable runtime kernel, and / or the runtime kernel is configured to take the array operation and the runtime instance as inputs to compute, update and / or define element values ​​stored in memory referenced by the runtime instance as outputs.

6. Completing the configuration of the aforementioned executable code means Generating a hash of the executable code, storing or caching the executable code on persistent storage, and loading and / or reusing the stored version of the executable code, To generate at least a partial runtime kernel, Terminating the aforementioned configuration of the runtime kernel, Optimizing the runtime kernel code with respect to the size, location, and / or layout information of at least one runtime instance as input. Configuring the runtime kernel to receive and / or access at least a portion of the runtime instance as input and / or at least a portion of the runtime instance as output, The runtime kernel is configured such that when the element values ​​of the runtime instance are fully configured as input, the execution of the runtime kernel is started on the processing unit. The method according to claim 1, comprising at least one of the following.

7. The method according to claim 1, wherein completing the configuration of the executable code includes defining executable instructions or code representations to be executed on the subunit, updating or defining at least a portion of the configuration, typically updating or defining at least size information of the runtime instance as output, and / or updating or defining information for at least each of the runtime instance as output when the corresponding information is updated or defined for at least one runtime instance as input, typically for all runtime instances, allocating memory on the processing unit, and updating or defining memory references on the runtime instance as output.

8. The method according to claim 7, wherein the code representation is configured to run on each subunit used to execute the executable code, and causes each of the runtime instances to define or update information as input to each program segment.

9. The method according to claim 1, wherein configuring each executable code at least partially or completing the configuration of each executable code includes, or is subsequently executed on the first subunit, if the information of the runtime instance as input to each program segment, in particular size information, is defined, configured and available to the executable code at this point, and / or completing the configuration of the executable code includes, or is subsequently executed on a further subunit of the processing unit, if the runtime instance as input to each program segment is fully configured and available to the runtime kernel at this point.

10. The method according to claim 1, wherein generating the segmented program code includes inspecting and / or analyzing the user program code or code derived therefrom for the purpose of extracting data flow information and / or analyzing data dependencies, wherein at least one of the program segments is chained to a previous segment in the sequence of generated program segments and / or receives at least one output array data structure of the previous segment as an input array data structure, and configuring the executable code of a second program segment in the program segment chain includes configuring runtime instances as inputs to the second program segment such that any modification or update of at least a portion of its configuration, typically a portion including at least the size and / or element values, causes the executable code of the second program segment in the program segment chain to begin execution, and / or such execution is performed motive-driven.

11. The array instruction includes metadata (AIMI) and enables the determination of configuration information, typically size information, for each input array data structure, at the output of the array instruction. The AIMI includes information about further input data required to determine the configuration information of the output of the array instruction, the information about the further input data typically includes the source of the further input data, and the information about the further input data includes or refers to the output of each array instruction and / or is used as input to the AIMI to execute the segmented program code. This includes receiving information, at least partially constructing an executable instruction or code representation using the AIMI of the array instruction corresponding to the array operation of the program segment, the array operation of the program segment, the configuration information, in particular the size information of each runtime instance, the runtime instance, and information regarding further input data as input, and determining and / or updating the configuration information, in particular the size information of each runtime instance, as output of the program segment. The method according to claim 1, wherein the AIMI typically includes size information, element type information and / or layout information for each array instruction, the size information typically includes or refers to at least one of the number of dimensions of each runtime instance, the length of at least one of the dimensions of each runtime instance, the position information of each runtime instance, and the stride information of at least one of the dimensions of each runtime instance.

12. Partially configuring the runtime instance of the array data structure as the runtime output of each program segment means Allocating and / or instantiating each runtime instance, Typically, the runtime output size information for each program segment's runtime output is determined using AIMI and the runtime size information for each program segment's runtime input, and Allocating memory for each runtime output of each program segment. The method according to claim 1, comprising at least one of the following.

13. Generating the segmented program code is, To maintain, at least substantially, the program flow of the user program code. Typically, this involves using a transpiler to replace the sequence of array instructions with a sequence of program segments, each of which includes at least one, typically multiple, array operations, each of which typically corresponds to one of the array instructions. Typically, a compiler is used to generate a device-specific runtime kernel for each program segment that performs each array operation. Analyzing the user program code and / or the segmented program code, particularly the sequence of the program segments, and / or generating an abstract syntax tree for each of the program codes, To analyze data dependencies that exist between array instructions determined by the user program code or between array operations determined for a sequence of program segments. To generate dependency graphs for at least some of the array instructions and / or array operations from each program code. Based on the dependency graph, if each program segment does not benefit from parallel execution, in particular if the dependency graph, including the array instructions corresponding to the array operations within each program segment, contains exactly one branch, then creating and configuring each program segment for synchronous execution, and executing each program segment synchronously, means executing the executable code, in particular the runtime kernel, on the first subunit, in particular within the first thread. The method according to claim 1, comprising at least one of the following.

14. A computer implementation method for executing program code, each configured to be executable on a processing unit, and including a program segment containing its respective kernel code and its respective configuration code, wherein each kernel code includes an array operation corresponding to one array instruction of a sequence of array instructions, and each of the program segments is configured to receive, as its respective runtime input, at least partially configured runtime instances of its respective array data structure capable of storing multiple elements of its respective common data type, The steps include: starting the execution of the configuration code of the program segments on the first subunit of the processing unit to partially configure a runtime instance of the array data structure as the runtime output of each program segment; When the configuration of the runtime instance received as each runtime input is complete and / or updated, the processing unit executes the kernel code of the program segment. A method that includes this.

15. The method according to claim 14, wherein the configuration of each runtime instance includes size information, memory references and / or element values, and the configuration of each runtime instance further includes usage information and / or runtime task information, and at least one of the size information, memory references, element values ​​and runtime task information of the partially configured runtime instance is undefined, uninitialized or missing, the usage information is a counter, and / or the runtime task information is a reference to a runtime task data structure.

16. The method according to claim 15, wherein executing the configuration code includes storing a reference to the runtime instance as runtime input to each segment and / or storing a reference to the runtime instance as runtime output to each segment, the reference being associated with a runtime task data structure that references the kernel code, the runtime task data structure being configured to start execution of the kernel code on the processing unit when the configuration of the runtime instance is complete and / or updated, and the kernel code is executed on a second subunit of the processing unit.

17. The method according to claim 14, wherein executing the kernel code includes using a reference to the runtime instance as runtime input and / or the reference to the runtime instance as runtime output to complete the configuration of the runtime instance as runtime output.

18. The method according to claim 14, wherein partially configuring the runtime instance as runtime output includes storing a reference to a runtime task data structure with the runtime instance, and / or completing the configuration of the runtime instance as runtime output includes removing the reference to the runtime task data structure from the runtime instance.

19. The method of claim 18, wherein executing the configuration code includes configuring the runtime task data structure to start executing the kernel code when each runtime instance, as a runtime output stored in the runtime task data structure stored together with the runtime instance as an input, has completed and / or been updated.

20. The method according to claim 14, wherein the usage information is updated by the configuration code, and in particular when a reference to the runtime instance as runtime input is stored in the runtime task data structure, the zero-based usage information counter typically increases, and / or the usage information is updated by the kernel code, and in particular when the configuration of the runtime instance as output is completed, and / or when the reference to the runtime instance as runtime input is removed from the runtime task data structure, the usage information counter decreases.

21. The method according to claim 20, wherein initiating the execution of the kernel code includes, in particular, when the utilization information indicates that the runtime instance is stored with further runtime task data structures as runtime inputs for further segments, and more specifically, when the value of the zero-based utilization information counter is greater than 1, completing the configuration of the copy of the runtime instance as runtime output using the copy of the runtime instance as runtime output based on the utilization information of the runtime instance as runtime output, and / or creating the copy includes, in particular, copying at least a portion of the configuration of the runtime instance as runtime output by copying the element values ​​of the runtime instance as runtime output.

22. The method according to any one of claims 1 to 21, wherein the user program code is user-generated code or a substitute thereof, the program code is derived from the user-generated code or a substitute thereof, the user-generated code is typically written in a domain-specific language and / or embedded in a host language such as C#, each program segment is generated as host language code or as an intermediate representation, particularly as a bytecode representation, the segmented program code is generated at runtime, the program code is generated in advance using a pre-compiler and / or before the program starts execution, and / or at least one of the segmented program code, each executable code and each runtime kernel is generated at runtime and / or at least partially by a JIT compiler.

23. A computer program product and / or computer-readable medium that, when executed by a computer, includes instructions causing the computer to perform the method described in any one of claims 1 to 21.

Images